Responding to an Incident When the Wrong Person is Out

The military would say that no plan survives first contact with the enemy.

Mike Tyson eloquently quipped that everyone has a plan until they get punched in the face.

Cybersecurity incident responders likewise have similarly colorful language rooted in a career’s worth of wisdom derived from the lived experience trying to identify and mitigate risks. With another major breach of US federal government systems fresh in our minds, it is natural to start thinking about cybersecurity, incident response, and planning. Many cybersecurity professionals make the mistake of a myopic view of the specific technical vulnerabilities, the zero day, and the potential for state actor involvement. But there’s another vulnerability, one that cyber hammers overlook because it doesn’t look like a cyber nail. It’s Fred.

We all have a Fred, or maybe you are Fred. Fred has been at the organization for decades. He’s a little crotchety and he eats the same lunch every day, but overall, he’s a good guy. He works in the bowels of the IT departments, and he’s been maintaining the firewall and security system since you procured it. He’s a Splunk wizard and he trained many of the people in the SOC. He isn’t much for policy, but he begrudgingly helped us update our Acceptable Use Policy last year. If anyone has any issues, he’s your guy. He’s on the other side of that [email protected] email address; the one you occasionally forward spam-y looking emails to. You might assume those go nowhere, but that’s gold to Fred. Fred updates the firewall and fine tunes your Microsoft Entra. He has all the devices registered and he never misses closing an account when someone departs.

Why?

Because Fred cares.

To Fred, IT security isn’t his job. To Fred, IT security is good guys versus bad guys. He’s a one man Justice League and though he grumbles about no one knowing how important he is (and you don’t), he doesn’t want the recognition. Fred is a pain in the ass sometimes. But he’s yours. And he’s the guy you want when it all goes down.

The problem is not with Fred; it’s with the universe. After decades of little or no malicious cyber activity, Fred decides to finally take his wife on that vacation they’d been dreaming about; Gatlinburg, Tennessee. They want to see Dollywood, drink moonshine, and see the black bears in Great Smoky Mountains National Park (Fred’s wife was terribly upset by the loss of Dolly Parton). Fred sets his out of office reply and cashes in his long-saved PTO. He leaves his laptop and cell phone at home at the behest of his wife and he’s in the air. And then the universe strikes.

Cyber incidents aren’t something we talk about in the abstract. They aren’t something that might happen. They are a reality of our world and planning for them is critical no matter whether you are a small startup or the FBI.

Many organizations from governments to small businesses do at least some level of planning around cyber incidents. Maybe planning is confined to settings in your O365 or Google Workspace environment. Maybe planning is robust and well socialized. Few are unaware of the need to plan for a cyber event, but equally few are preparing for it with resilience. Think about Fred. Fred is a vital member of the team; so vital that an incident response without him is doomed. Having a Fred is great, but having a Fred might be creating cyber vulnerabilities that you aren’t tracking. What happens to your plan when one or more critical personnel are out? Can your plan survive that punch in the face?

Leaders are often charged with hiring the best people. Succeeding in that charge often means improved sales, product developments, or security. Hiring good people will never disappear from the leader’s book of business. However, hiring that one good person is not enough, as most leaders know. You need to hire good people that work together as a team. People that cover for each other, support each other, and make one another better. In cybersecurity planning, the same is true. You need good people in your IT department, but that’s not enough. Your biggest cybersecurity vulnerability may not be in your system or your IT department. Your biggest cybersecurity vulnerability may be in a lack of resilience in the event those personnel are not available on the big day.

The best plan in the world is no plan at all if it hinges on one person. Whether that person is on vacation like Fred or detained responding to another element of the attack, your cybersecurity plan must survive. For it to survive, you need more than one person that can execute it. Leaders should assume that during a cyber event, the fog of war will set in. People will be pulled in multiple directions, someone might be offsite, comms systems may be down or unreliable, and people may focus on the wrong thing. All of this means that a real cybersecurity incident response plan must include cross training.

No one really knows how they will respond to a crisis until they’ve lived one. IT leaders spend careers trying to avoid cyber issues through defensive measures, but the day may come when your organization’s bottom line, reputation, and data are under threat from an attack. When that happens, you cannot have that be the first time anyone on the response team has faced a crisis. You also cannot have that be the moment when you find out that the plan falls apart because Fred is at Dollywood.

Leaders from governments to corporations need to focus not just on the presence of a cybersecurity incident response plan, but of its resilience. Can the plan survive if one person is out? Can the plan survive if the teams are pulled in multiple directions? Those answers are more critical to your cybersecurity preparedness than the contents of your plan. Here are some practical steps leaders can take to test and fine tune their incident response plans:

1.        Conduct tabletop exercises to identify gaps and improve plans.

2.        Cross train leaders and responders to take on different roles in the moment of a crisis.

3.        Focus on the execution of the incident response plan, not the existence of the plan.

Fred deserves his vacation. We all do. Those vulnerable moments are the wrong time to figure out that you had a plan that depended on one person. Use your non-crisis time wisely by making sure your response team knows the plan and can execute it regardless of who is on vacation. Give them a safe space to fail and learn by exercising the plans. Refine the plans based on what you learn. Fred might complain, but this is what resilience looks like.