It's the first group that matters most, because those people are in organizations that use (or soon will use) Windows 7 as the primary OS, with XP the minority OS for those who can't move. That duality creates all sorts of headaches for both users and IT. But all a company can do is mimimize the use of XP to where there really is no good alternative, and isolate such systems where possible to prevent accidental IE and driver updates that would make them not work properly any longer.
These specialized uses aren't going away -- there's no place for them to migrate, a big failing on Microsoft's part. Yes, it created XP Mode and MED-V and other complex approaches for Windows 7, but they are complex and often do not work. They simply don't scale, and thus don't address the fundamental migration issue specialized users have. Likewise, the individuals who use XP because it's all they need and all they want to know aren't disappearing, either. As a result, XP will be an active OS for years to come, despite its formal retirement.