This security consulting firm is well known for fighting APTs and even sells detection software to help. Frequently, on engagements, it succeeds in selling its software and getting it installed on every computer in the environment. I was very used to seeing its service running in Windows Task Manager.
In this particular story, the security consulting firm arrived first, saved the day, and moved on. It also succeeded in installing its software throughout the organization and hadn't been onsite in nearly a year. As far as anyone knew, the customer had been APT-free since the initial remedy. At least no one had detected any signs.
I'm a big fan of honeypots. A honeypot is software or a device that exists simply to be attacked. It can be an unused computer, router, or server. Honeypots can mimic anything, and they are great for detecting previously undetectable adversaries, so I recommend them often. This can be a decommissioned computer to which no person or service should be connecting. When a hacker or malware does connect, the honeypot sends an alert that can trigger an immediate incident response.
In this instance, I spent a few days helping the client deploy some honeypots. Most customers ask me how we are going attract hackers to the honeypots. I always laugh and answer the same way: "Don't worry, they will come." Indeed, every honeypot I've ever set up has detected nefarious activity within a day or two. These new honeypots were no different.
We detected network logon attempts coming from multiple workstations, none of which had a legitimate reason to be logging on. We pulled a few of these workstations and forensically examined their hard drives. We found that the APT had placed a remote-access Trojan on each of them. The Trojan's name? The same as the anti-APT detection software. The bad guys had someone replace the legitimate anti-APT software with a Trojan, and it turns out they did it on nearly every computer.
This explained a few things, like why no APT had been detected. But the bigger question was how did it get installed in the first place. It turned out the customer's "gold build" had been compromised in its build environment, and this Trojan was part of the build.
Lessons: First, verify the integrity of your builds; prevent unauthorized modification or invent some way to detect it. Second, honeypots are a great way to detect malicious activity. Third, always look for and investigate strange network connections from unexpected places.
APT war story No. 4: All your PKI base belong to us
APT attacks on Public Key Infrastructure (PKI) servers used to be somewhat rare. In fact, until two years ago, I never personally ran across an APT where PKI servers had been involved. Now, it's fairly common. But the most relevant story is the one where the PKI turned into physical access in a sensitive area.