Case study: How Transnetyx reduced email alerts by 96%

Transnetyx is an automated genotyping company dedicated to providing biomedical researchers with faster, easier, and more accurate results worldwide.

Challenge

• 3,000 email alert notifications per month surpassed the small NOC team’s manual bandwidth.
• Multiple siloed monitoring tools resulted in limited visibility into the full scope of the IT environment.
• No standardization of alert quality meant critical alerts were not quickly identified.

As the world’s first fully automated genotyping system, Transnetyx is responsible for genotyping more than 30 million samples worldwide for biomedical research. Maintaining a high level of service visibility and uptime to meet the growing needs of their clients is a critical performance requirement for Transnetyx.

“Part of my role is to evaluate opportunities to become more efficient within our operations infrastructure,” explains Antonio Taylor, Vice President of Infrastructure, Security, and Services at Transnetyx. “During this process, we discovered just how quickly we were getting alert overload from email notifications that were coming in across all servers. It was impossible for our small team to appropriately address all our email notifications in a timely or prioritized manner.”

With such a high influx of notifications flooding their system, parsing through alerts to accurately identify what required their immediate attention was a challenge. Many critical alerts ended up lost in the mix, given the lack of prioritization of predominantly benign or low-impact alerts.

“One simple example is when we had a server that was running out of space. We were receiving numerous warning emails as the server approached its threshold for space, but due to the high inbound volume of monthly email alerts, approximately 3,000, it was increasingly difficult to promptly identify and prioritize specific alerts that were progressively becoming more critical or leading towards a potential outage before it was too late.”

Solution

Transnetyx needed AIOps to reduce the volume of IT noise, consolidate alerts into a first pane of glass, and automatically correlate those alerts into actionable incidents for priority resolution.

“We quickly configured six standard monitoring integrations to BigPanda to achieve a first pane of glass, which provided rapid ROI. We now have a single consolidated source we can utilize to view the performance across our systems. It is truly phenomenal how much quality detail I see at-a-glance.”

Event Deduplication
Using the BigPanda Alert Intelligence feature, Transnetyx defined event deduplication logic for their six standard integrations and configured alert filtering to stop duplicate, low-relevancy events from being correlated into incidents. By stopping alert noise before it reaches the incident feed, the NOC could focus solely on critical incidents.

Event Normalization
Transnetyx’s NOC defined the standards and logic for 14 normalized tags and normalized alert payloads. Normalizing heterogeneous data from different monitoring tools into a single, consistent taxonomy format using general-purpose key-value pairs, called tags, allowed alerts streaming in from several monitoring sources to be enriched with operational and topology data, creating a clear understanding of alert impact and context for incident resolution.

Event Enrichment
By incorporating context and business logic into incidents using the BigPanda Incident Intelligence feature, the response time of the NOC has been greatly improved. Transnetyx has set up fundamental correlation patterns, which facilitate the identification of high-priority alerts. They have also implemented Priority Tags and automated the definition of P1 incidents. Additionally, Transnetyx has configured outbound sharing and automated sharing to MS Teams, streamlining communication and collaboration processes.

“The level of support that BigPanda provided in the implementation process was very impressive. I’ve never worked with a company that’s been this hands-on with the success of the implementation, which is important for a small team like mine,” says Taylor.

Early Benefits

Transnetyx’s newfound visibility through the reduction of alert noise and preliminary alert enrichment strategies allowed the NOC to expedite the reduction of inbound email notifications from 3,000 per month down to roughly 120. This created a manageable space for the NOC to focus on the incoming alerts that require critical attention. “We’ve changed our workflow. Now, our workflow starts with BigPanda. First thing in the morning, my team looks at the BigPanda platform to view priority P1 tasks.”

• 96% fewer email notifications — from 3,000 per month down to roughly 120.
• Rapid implementation facilitates team expansion of the BigPanda offering ahead of schedule.
• Visibility across the IT stack within one consolidated dashboard uncovers data for improving alert quality.

“We have reduced the number of email notifications by an average of 96%,” says Taylor, “That results in a dramatic reduction of tickets because these low-quality alerts are being addressed and resolved automatically before it reaches our P1 team,” says Taylor.

These reductions directly improved team bandwidth. Taylor’s team is now more readily available to re-focus their efforts on innovative projects, such as cloud migration.

Just the Beginning

The rapid results experienced with BigPanda have also prepared Transnetyx to expand their use of AIOps for their own team and to additional teams far ahead of schedule.

“I wasn’t originally anticipating being ready to roll this out to other areas within the year. I was planning to run BigPanda for our small team for six months to see the value, but we’re way ahead of schedule, so we are now going to be able to expand BigPanda’s offering to other groups and let them also see the value of 96% fewer email notifications.”

“Within my own team, we are also working on enhancing our alert quality standards further. We’re going back to our payloads to see where we can improve the data quality attached to each alert, creating best practices to ensure the only alerts that come through to the NOC are of the highest quality possible.”

On a reporting level, Taylor looks ahead to applying additional data analytics for enhanced business intelligence to provide deep visibility into KPIs, metrics, and trends for continuous optimization. “We will be utilizing Unified Analytics to provide the high-level data I need for executive meetings. Being able to share data insights around improved meantime to resolution, meantime to assign, etc., shows leadership the value and ROI of BigPanda. It also supports our goal of becoming more proactive rather than reactive. BigPanda is helping us to achieve that.”