BLUF: Having gone through what I hope is the worst parts of an acute organizational crisis (cybersecurity incident) at work, I'm looking for thoughts and resources about how to do better in the next one.  What experience do others have to offer and what resources are good for preparing?  And would Mark & Mike consider a 'cast series on the topic?

The gist is that my employer got hacked and I work in IT.  We had three very intense weeks to get through the worst of clean-up and getting back on-line.  I have some thoughts on what I did reasonably well with and where I need to improve -- with a particular issue on communication.  I've got a start on my own retrospective and will do one with my team.

Thanks in advance. 

bffranklin's picture
Training Badge

Your question isn't entirely specific on if you're looking for responses geared towards technical, policy and procedure based issues of working through incident response or on the management side of the activity.  I've skewed my answers towards the latter, but feel free to poke me about more of the former.

I've got to say, nothing gets the blood pressure up quite like an intrusion.  As a bit of background, I was formerly employed as an intrusion analyst at a Managed Service Provider, so I've been through 8 of these at this point.  I've seen these handled poorly, and I've seen them handled well.  A few thoughts on the matter:

- Managing through an IR effort is similar to managing through any business error.  Remain calm. Identify the scope of the issue.  Make a plan to fix it.  Communicate the problem and plan to your customers.  Implement the plan.  Lessons learned.

- First things first, breathe.  You need to act, but you need to resist the urge to speculate as a manager.  Constantly calculate potential scope of the problem in your head, and the potential responses for each scale of issue, but never speak a word about hypotheticals.  The response manager needs to be oozing "this is under control."

- Clear your plate.  Delegate as much as you can to trusted directs.  You were just tossed a huge ball.

- If you are running the response, status updates to you need to include confidence levels with every bit of information.  You need to know what the best and worst case scenario scopes are at any given moment in the response.  If your security staff is experienced, their guts can be valuable but you need to know what is gut and what is forensically confirmed fact.

- If your security staff isn't experienced, get help.  Much like an infection, you don't want this returning.

- Briefings to stakeholders should be limited to confirmed fact.  I cannot stress this enough.  Let them know that you will tell them when you have the full picture.  This is particularly important so PR staff can communicate to customers once the full story is known.

- Briefings to stakeholders should be short and frequent until a remediation plan is in place, at which time you can move to an update schedule appropriate to the plan.  5 minute updates every hour or every other hour is not out of the question.  Send daily summary emails at EOB.  Intrusions trigger panic, and even small updates make people feel like things are under control.

- Establish who is actively working on the issue, and who is just supporting.  Anyone working on remediation should be in the same location, or at least on a conference call together.  These folks need to constantly be communicating.