Why you can’t have AIOps without Data Engineering

by Craig Ferrara | Mar 1, 2023

There’s a familiar saying: garbage in, garbage out. For ITOps, this directly applies to data engineering. BigPanda’s Area Vice President of Value and Adoption, Craig Ferrara, says the importance of data hygiene—putting good data in to get good data out—is the core of data engineering, and it requires ITOps to take a look at their data before integrating with an AIOps solution.

In this on-demand session during Pandapalooza, Ferrara and our Senior Product Manager of Integrations, Jon Capo, discuss what data engineering is, what happens when it’s working right, and how BigPanda views it. Learn more about what they revealed here.

What is Data Engineering?

Ferrara said the concept behind data engineering is that the quality of output of your AIOps (or the event processing you put in place to reduce workload) is a function of the quality of the data that you put into the system. Good data engineering is “all about making sure that you’ve got complete alert payloads—that you’ve got a solution in place that helps you find good data—and reducing noise,” Ferrara noted.

Why prioritize Data Engineering?

The benefits of Data Engineering and its effect on AIOps’ success span three categories:

• Reduced noise. “With AIOps, we’re always trying to affect workload,” Ferrara said. With Data Engineering in place to filter out the alerts that ITOps doesn’t need to bother with, the team gets time back and is more productive.
• Consistent, high-quality alert payloads. “A function of noise reduction is making sure we know what’s happening all the time, and we’re making sure that when we’re bringing in signal, we’re providing the highest quality and complete work payloads to provide our operations with as much information as we can,” noted Ferrara.
• Maximized alert content. This last piece means there is topology information associated with incoming data. Solid assignment groups, automated prioritization definitions and correlation all come down to data engineering.

The goals of Data Engineering

Since data engineering takes place at the very beginning of the event processing pipeline, good data is foundational for AIOps, and it has two overall goals:

• To reduce workload. You can reclaim time from your operators if you reduce workload. “They will have more time to spend on things like system stability,” Ferrara said, in addition to special projects or new projects that make work less monotonous. This could result in better employee retention.
• To reduce service degradation. Service degradation can directly impact revenue-affecting systems.

What’s involved in Data Engineering?

Capo pointed out that there are quite a few steps BigPanda takes before AIOps delivers high-quality, actionable incidents, including:

• Correct tool configuration. Set up your tools to start sending data over, and then normalize the payloads to ensure all data coming in aligns with a specific schema or design.
• Duplicate and enrich to provide additional context so you can filter out irrelevant alerts.
• Aggregate these alerts that may have the same properties in order to correlate alerts into a single, actionable incident.

These steps result in filtering out the “garbage” data. To learn about the in-depth use cases Capo offers for organizations that have leveraged Data Engineering, check out the full session here.