Why an Incident Commander is crucial to ITOps

by BigPanda | Mar 16, 2023

It may be counterintuitive to tackle a problem without knowing exactly what the problem is, but an incident commander often does just that. In fact Rob Schnepp—founding partner at Blackrock 3, an Alameda, California-based incident management consulting group—says identifying the root cause of an incident is typically secondary to addressing the symptoms.

“Many people get hyper focused on the root cause of the incident, but when the path is not readily apparent, it’s good to look at the incident from the other direction: what’s not wrong. [The problem] may not be in the network, it may not be in the app . . . let’s figure out what’s not wrong. That’s as valid a method of problem solving as knowing the path,” Schnepp says.

That’s one of many key takeaways discussed during Blackrock 3’s recent webinar at Pandapalooza.

Minimize disruptions

In order to reduce disruptions, an incident commander needs to protect the integrity of the resolution effort by keeping people from asking the wrong questions at the wrong time. Having too many people around can be equally problematic.

“More is not better. It takes a while to get your head back in it if someone’s standing around asking questions,” Schnepp says. “Avoid putting people in who do not absolutely have to be there. If I’ve got someone asking questions who is not relevant to the incident, [I] take that person off the bridge.”

Disruptions can pop up in many forms, too—it’s not always limited to poorly-timed questions. It could be something as minor as a subject matter expert (SME) with spotty internet access.

Remember: you will fail

While you may hope for a 100% success rate, Schnepp explains that’s not realistic.

“The expectation is that we will all show up, do a great job, and we won’t turn a 40-minute problem into a five-hour catastrophe by our own hand. But sometimes, it happens.”

The most important thing about failures is to learn from them—it’s a better teacher than success. After the incident is resolved, use the experience to teach the team, evaluate response mechanisms and think about how the team can better work together.

Watch the full BlackRock session about incident commander and the difference it can make for your organization.