Understanding true costs of a bug

Bugs are a major issue in the software development business. They create tension within the company, delay releases, damage brand, etc. They decrease potential revenue while increasing the total cost of the software more than you think. However, they are an unavoidable part of software development, more so since the massive expansion of agile and continuous delivery, the best described by Facebook’s “move fast and break things” motto.
This article will deal with understanding the true cost of a bug. Mitigation options will be left for another time.

Four factors affect the cost of the bug:

• The bug severity
• Time from discovery to resolution
• Cost of resolution
• Availability of information

Bug severity affects the income side of the equation. There were many cases of services being unavailable, thus causing a lot of revenue lost. Time from discovery to resolution is very important as it affects the perception of quality and the perception of care for the software. Cost of resolution affects the available capacity for other work, developer distraction, overtime work (and overtime induced issues), etc. The availability of information affects both sides of the spectrum, as insufficient information can delay resolution and inflict additional cost to discover the source of the issue.

Discovery

A bug can be discovered:

• during development – by unit and component tests
• during regression/alpha testing – functional tests (manual or automatic)
• during acceptance/beta testing (testing by the client) – by the client
• in production (by the client while using the software)

Depending on the point where the bug is discovered, the cost of a bug follows an exponential curve. Bugs found during the development and regression are much easier to reproduce. A lot more information can be extracted as well. Having a client involved makes things a lot more difficult (pressure, communication difficulties, the privacy of information).

In practice

Let’s end with some examples, from my experience as a software developer:

• I discovered bugs through unit/component tests many times. They are trivial to reproduce and time between discovery and fix is under 1 hour. No one else is involved
• Through running other people’s automated regression tests (thousands of them), reproduction is similar to my tests. It’s a bit more difficult to debug, but it’s still under a day. Other than me, test owners are consulted.
• When a QA engineer discovers a bug, it took me some time to discuss the issue, decide on reproduction steps, obtain logs, provide patches, etc. Time to fix is usually 3 days to two weeks, with nearly full focus on the issue. Fixing involves people who discovered the bug, people who triaged, and me.
• When a client discovers a bug during acceptance/beta testing, it involves time zones differences, additional communication overhead (project managers, client reps, etc being involved), invalid logs or reproductions, etc. Time to fix is never under a week, it goes up to a month for the majority of bugs. During that time, I was involved with 20-50% of my capacity. Fixing involves all the people from above plus at least one person from the client’s team
• When a bug is discovered in production, on top of everything above, it gets extremely difficult to get proper reproduction steps and data. Explanations like: “it just stopped working”. These bugs go through various levels of support. Usual time to fix is between a month to six months. It often needs multiple tries to get the actual fix. It involves a lot of different people. Developers, meanwhile, are usually idle.

Summary

The purpose of this article was to demonstrate how bugs affect the cost and the revenue of a software product. We ended with the examples from my own experience, describing how time-to-fix can grow from one hour to over a year, just by not discovering the issue soon enough.

What I hope you will do after reading this article, is to discover when and how the majority of the bugs are found, calculate the direct cost of solving and the indirect cost of revenue loss/brand damage. Once you know how much bugs costs you, discuss ways of reducing those costs.

And for a software developer reading this, if you feel your management pushes you to cut corners and deliver “hacks”, try defending your position through explaining the end results for the business, like mentioned here.