FreeWheel

FreeWheel, a Comcast company, provides TV advertising platforms that improve ad placement and tracking for agencies and optimize the use of ad inventory by content publishers.

Challenges

• An average of 15,000 daily alerts created an overwhelming volume of email notifications.
• Extended outages and prolonged incident resolutions lead to growing operational costs.
• Manually sorting, escalating, and triaging events was painful, with an average mean time to respond (MTTR) of 25 hours per incident.
• A large data gap between ticketing and communication tools led to substandard cross-team collaboration.

FreeWheel’s network operations center (NOC) was swimming in email notifications and alerts on any given day. Its cloud, on-prem monitoring, and homegrown production tools sent an average of 15,000 events daily. Their response teams felt like they were drowning to manually assess and label email alerts ranging from benign to critical. The labor-intensive incident response process led to higher rates of service downtime and outages because operations teams wasted precious time and resources pursuing false positives, duplicates, and low-priority alerts instead of focusing on critical alerts across their disjointed monitoring tools and platforms.

Collaboration with L2 and L3 response teams was further exacerbated under the immense pressure of properly escalating incidents from IT noise. FreeWheel’s ticketing and on-call platforms could not integrate and sync bi-directionally, making it especially challenging to update different teams on the incident response status. Prioritizing and collaborating on actionable incidents was inefficient and manual, leaving FreeWheel in a reactive posture. They were struggling to meet SLAs and MTTR goals.

Solution

FreeWheel wanted to scale its ITOps team productivity. They decided on BigPanda’s Alert Intelligence to clean and unify their fragmented observability data. Alert Intelligence enriches email alerts with context, such as location, host, and affected service, to generate actionable alerts. This allowed the NOC to easily and quickly detect problems from a first pane of glass and take action on priority alerts. It also alleviated fragmented communication between teams. “BigPanda Alert Intelligence has given us new visibility and far more tangible data. Ultimately, we were able to reduce email alerts by more than 90%,” says Lorenzo.

Benefits

BigPanda Alert Intelligence helps FreeWheel scale ITOps productivity through technology—not headcount. The ability to reduce alert noise and MTTR delivers more consistent uptime and application performance for key applications, and L2 and L3 response teams can focus again on innovative development projects without being distracted by responding to low-priority alert noise.

• 90% noise reduction eliminates the tsunami of email alerts from 15,000 to 1,500 emails.
• 71% compression rate further reduces low-quality alerts from 1,500 to 435 actionable alerts.
• 78% MTTR reduction from 25 hours to 5.5 hours per incident is achieved by delivering high-quality, actionable incidents to response teams to help them focus on real issues related to incidents and outages.

“BigPanda is now integrated directly with Jira. Anything we escalate is automatically becoming a ticket. This has given us new visibility and far more tangible data and has led us to better cross-team communication, which is helping with event status and tracking,” explains Lorenzo.