- OverOps helped BT reduce MTTR for errors and establish a “shift left” culture
- BT release efficiency improved by 20%, moving from 4 to 5 releases per year and saving more than £2m per year
- Better testing resulted in fewer maintenance releases saving £1m per year
Business to Consumer IT Director
BT’s inefficient software quality processes combined with a lack of proper tooling was causing critical errors in production. Poor software quality forced the team to release 30% more versions, costing the BT over £5m a year in wastage. The telecom company needed to establish a “shift left” approach to quality and gain deeper visibility into their code so they could identify and prevent critical issues from being released to production.
Prior to OverOps, BT’s consumer group faced a number of code quality struggles.
Organizationally, heavy reliance on offshore resources for critical application development and resolution made it difficult to gain visibility into the quality of the code they were given. Additionally, inefficient processes for knowledge sharing created isolated pockets of intelligence that left major gaps when critical team members departed the organization.
From a tooling and workflow perspective, existing quality gates and checks failed to catch 100% of issues, leaving errors and bugs to creep through various stages of the delivery lifecycle and reach production. This resulted in app outages, impacting customers, as well as lost revenue and developer toil. Planned maintenance releases required heavy lifting to address the multiple critical issues that occurred in every major release, often throwing off timelines for development and delivery of critical functionality.
New releases were failing to deliver nearly 30% of the intended payload. With each release costing £10 million and four releases a year, this meant that poor software quality was ultimately costing BT £12 million a year in wastage alone.
BT chose OverOps’ Continuous Reliability solution to serve as a core pillar of their shift left software quality strategy. By integrating OverOps code quality gates – such as new errors, critical exceptions and total error volume – directly within BT’s CI/CD pipeline, the team was able to reduce the number of customer impacting production errors and ensure a high level of confidence
in the stability of releases.
“OverOps’ integration with Jenkins allows us to drive ‘go-no-go’ decisions, blocking a release if it falls below a quality threshold.”
OverOps is able to identify critical errors missed by other testing methods, automatically block unstable releases from being deployed, and provide a complete feedback loop to engineers with rich error context, enabling them to reproduce and resolve issues in minutes. This has allowed the team to significantly reduce MTTR and cut down on resource utilization and maintenance releases.
With the help of OverOps, BT was able to reduce wastage, and the team’s pre-production success has since served as a starting point to launch a similar “Shift Right” initiative to ensure fast identification and resolution of any errors that do make it to production.
How are you integrating OverOps with your daily workflow?
We integrated OverOps directly into our existing Jenkins pipeline to analyze the quality of our code before it gets promoted to the next stage. Through the OverOps Jenkins plugin, poor quality releases are automatically stopped and critical errors are routed back to the right developer through our Jira system.
OverOps is able to feed code-level data directly into a number of critical tools in our software delivery pipeline, and we are already exploring additional integrations like the OverOps SonarQube plugin to further enhance our software quality workflow