I hear it all the time when talking to future BigPanda customers; “I’m not sure BigPanda can really help me correlate all these alerts together because our CMDB is very immature.” Or sometimes, they don’t even have a CMDB, and incorrectly assume this disqualifies them from meaningful noise reduction and alert correlation.
I’m happy to tell you the same thing I tell the folks who are looking at BigPanda for the first time. “No CMDB? No problem!”.
While it’s true that many other tools in the Event Management space (or the Correlation and Automation space, if you prefer) require a mature and robust CMDB to be effective, BigPanda is different, in that the correlation patterns created by our Open Box Machine Learning can effectively identify ways to correlate alerts and reduce noise even without a CMDB. In fact, when we engage with customers, even before we bring anything from a CMDB, we routinely see compression rates in the 70% range.
Of course, the more information and context we can provide for an alert, the better. Providing additional context and information to alerts within BigPanda is known as enrichment, and there are several ways other than traditional CMDBs that BigPanda can enrich alerts.
Enrichment – a simple example
One of my favorite examples of enrichment is using infrastructure information from virtual machine provisioning tools such as VMWare. Imagine that you’re an operations engineer watching for incoming alerts when all of a sudden, a flood of warnings come in for 20 different hosts, all complaining about connectivity issues. What’s going on? Is it a network issue affecting all 20 hosts? Once you’ve brought in enrichment data from your VM provisioning tools, BigPanda will enrich those 20 incoming alerts and provide additional context. In this case, it turns out that even though it looks like 20 different hosts are experiencing problems, all 20 hosts actually reside on the same physical hardware, and because BigPanda updated the alerts with the physical host, it was then able to correlate all 20 alerts into one incident and point the engineer to the probable root cause. And guess what? By turning 20 alerts into 1 incident, BigPanda reduced alert noise by 95% in this one instance, and it didn’t rely on a CMDB to do it!
Enrichment – a more powerful example
A more powerful example of enrichment is around using topology data from application monitoring tools. Many application monitoring tools in the market offer the ability to automatically discover information about how processes map to applications, and how applications map to hosts. This data can then be ingested by BigPanda as enrichment data. Once we’ve done that, BigPanda can enrich any subsequent alerts that come in, to show which applications are potentially impacted by an issue with a particular host. In addition to the scenario described here, BigPanda is also fully capable of handling complex mappings such as multiple applications per host, multiple hosts per application, or nested dependencies.
Other common sources of enrichment that we can handle with BigPanda include runbooks, knowledge base articles, and even spreadsheets that customers typically create over the years to assist in manual mapping of applications and infrastructure. Because BigPanda takes a best-of-breed approach to our integrations, we allow you to pull enrichment information from any tool or location regardless of the tool vendor.
Here’s a diagram that illustrates how BigPanda uses enrichment information to enrich incoming alerts.
And yes, BigPanda can even help improve your CMDB
Another area where BigPanda delights our customers is using BigPanda analytics to improve the quality of enrichment sources.
For both current and future customers of BigPanda, right after they finish telling us how immature their CMDBs are, they say something along the lines of “but we want to improve our CMDB and we have plans to work on it.” The question I like to ask then is “where do you plan to start?” Creating and enhancing a CMDB can be a daunting exercise. With so many hosts in the enterprise, how do you prioritize where to spend your efforts on improving your CMDB?
Fortunately, BigPanda gives you access to rich analytics that show you, among other things, the percentage of alerts being enriched, stats on which hosts generate the most (un)enriched alerts, stats on applications and tools generating the most (un)enriched alerts, etc. This then becomes the prioritized to-do list for your CMDB – or other enrichment source – improvement project. I’d much rather spend my time enhancing enrichment information on a host that had a thousand alerts last month than on a host that only had fifteen alerts. Analytics like the ones from BigPanda can also help to identify the gaps when new hosts or entities are added to the environment without the accompanying enrichment information.
So other than improving the quality of your correlation, why else should you care about adding enrichment data to your alerts? What we hear, consistently, from our customers is that in addition to suppressing noise through better correlation, enriching alerts helps improve operational efficiency. For example, one of BigPanda’s largest customers – a global chip maker – reports that after enriching their alerts with runbook and wiki links, they’ve seen a 400% improvement in their L1 resolution rates (aka the number of incidents their L1 engineers are able to resolve without escalating them to L2s, L3s or DevOps engineers). Imagine the time saved here simply by automatically providing the contextual information around how to resolve an alert within the alert itself. What would an improvement like this mean to your company?
If you’ve been putting off exploring modern Event Management platforms (also known as Correlation and Automation platforms, Autonomous Operations platforms or even AIOps today) because your CMDB is not yet “ready for primetime”, I hope this post gives you hope.
To learn how BigPanda can help your IT Ops, NOC and DevOps teams handle more incidents, faster than before and to see a demo tailored for your environment, please drop us a note.