This particular customer used its internal PKI servers to create employee smartcards. These smartcards were used to not only log on to computers but to physically access company buildings and other infrastructure.
The customer's root CA (certification authority) server was a virtual instance sitting, disabled, on a VMware host server. The bad guys had found it, copied it offsite, cracked the local (weak) administrator password, and generated their own trusted subordinate CA. They used this CA to issue themselves PKI access to everything they could.
What surprised me most was the video my client showed me of two unknown men posing as employees. Using the fake smartcards they created, they had parked their cars inside the secured company parking lot, walked into the building through the employee entrance, and onto a floor that stored highly sensitive data.
My customer couldn't tell me what happened after that or what was taken, but I knew they were not happy. There was a very serious mood in the room. I was invited to help them create a new PKI and to migrate the company into the better-secured PKI environment.
Lesson: Protect your PKI CA servers. Offline CAs should be just that: offline! They should not be disabled or sitting on the network with their network cards disabled, but off the network, stored in a safe, and not so easy to compromise. CA private keys should be protected by a Hardware Storage Module appliance, and all related passwords should be very long (15 characters or more) and complex. Plus, it can't hurt to look for and monitor if other unauthorized CAs get added as trusted CAs.
APT war story No. 5: Don't forget the accounts you're not supposed to touch
As mentioned above, most APT recovery events involve resetting passwords. If you're going to reset passwords, reset all accounts — though it's easier said than done. All my customers start out doe-eyed, ready to reset all passwords, but when they discover how much it will disrupt the business, they quickly scale back their goals. It's far easier to get fired for causing a significant business interruption than it is for not getting all the hackers out.
This particular customer was ready and incredibly thorough. The plan was not only to reset all user and service accounts, but computer accounts as well. Almost no companies do this, especially when it comes to resetting service and computer accounts. Heck, I'm giddy if they reset all elevated user accounts, because it's hard to get that little bit done thoroughly. Laugh only if you haven't been through this drill.
Password reset day came and went. There were significant service disruptions, some of which were painful enough that we had to tell the CEO. By the end of the week, however, we had reset all the passwords.