Within a few days, the APT owned everything again, picking up all email, controlling all the elevated accounts, including IT security accounts. It was like the password reset never happened. We were perplexed. As best we knew, we had removed the easy holes, educated employees, and couldn't see any evidence of Trojan backdoors.
Alas, there's a built-in Windows account called krbtgt that is used for Kerberos authentication. You shouldn't touch it, remove it, or as far as we previously knew, change its password. It really shouldn't be a user account that shows up in user account management tools, and this APT team knew it.
As I've learned on successive engagements, krbtgt is a go-to technique. After an APT crew compromises an environment, they add the krbtgt account to other elevated groups. Because customers usually leave it alone, even during a password reset, it can be exploited as a go-to backdoor account. Great idea -- if you're a malicious hacker.
My customer reset the passwords of its krbtgt accounts and everything else (again). As far as I know, it has not had another detected problem. Be aware that resetting krbtgt accounts will absolutely cause authentication problems. It's a pain. But if you have to do this, you too will get through it.
Lesson: if you're going to reset all accounts, make sure you know what "all" means.
APT war story No. 6: Information overload is spurring APT innovation, too
My last story isn't about a single client, and it shows the evolution of APT over the years. Early APT practitioners would immediately collect everything they could as soon as they broke in. They would siphon out all old emails and install bots to get every new email sent. Many times they would install Trojans to monitor the network and databases, and if new content was created, they would copy it.
In other words, many companies have online backup services they aren't paying for.
Those were the old days. In the world where terabyte databases are no longer even close to surprising, APT has a problem. When they get complete access to a network and learn where all the information is stored, they have to be more selective. Whereas they used to grab everything, what we see now are very discrete selections. The more advanced APTs these days build their own search engines, sometimes with their own APIs or borrowing the APIs of other well-known search engines, to search for specific data. They may still only leave with gigabytes of data a day, but what they have is highly selective.
Lesson: APT has the same issues finding and managing data just like you do. Don't let them index your data better than you do.