Sam Kendall’s noisy alert problem

By |2018-04-17T18:15:30+00:00February 23rd, 2016|Blog|
  1. Technology serves customers. Not the other way around.
  2. Great tools are no substitute for great process.

Like most BigPanda customers, Sam’s NOC uses more monitoring tools than they should, but they’re too dependent on them to consolidate and too invested in years of configuration to know where to start if it was a priority. They monitor the network with SolarWinds, servers with Nagios, AWS workloads with CloudWatch, uptime with Pingdom, synthetic transactions with Catchpoint, logs with Splunk, and performance with New Relic.

As a result, routine tasks like upgrading the firmware on a router often generate hundreds of redundant alerts. Sam tried everything to suppress noise without missing real issues. He failed. Rosary beads didn’t work. Rabbit feet gave his kids nightmares. Years into his cage match with 2:00 AM wake-up calls he called us.

With BigPanda, he sees one incident per actual issue with a live timeline that provides details about each related alert, including when it occurred and why. His team integrated seven monitoring systems in about a day, and now all event correlation happens in the cloud while he sleeps.

When asked how he responded to a recent attempt to replace BigPanda, he went all Heston with a heartfelt “…they’ll need to pry it from my cold, dead hands.” Before BigPanda, everything in Sam’s world was structured and predictable…except his noisy alerts. Now the chaos is gone. His wife and kids say he’s less grumpy and his team says he has more time for them.

Site visits like this are the best part of my job. They’re a reminder that what we do is bigger than event correlation or monitoring or even technology. Sam, thanks for an excellent day. No need to thank us…just send paella (and hold the clams).

About the Author:

Dan Turchin was BigPanda's VP of Product. Follow Dan on Twitter: @dturchin. Connect with Dan on LinkedIn.