The reason for test automation usually boils down to the fact we all want to get more testing done, faster. But when it comes to proving the return on investment, or the ROI, of your automation efforts, things can become a bit blurry.
Whether you’re trying to convince management, your teammates, or whoever’s in charge of your budget, prescribing an exact value can be difficult. But this doesn’t mean you should give up on the dream of implementing automation across your organization.
When we talk about software testing and development, we want to address three major elements: speed, quality, and cost. As we evolve our processes to encompass Agile, CI/CD, and DevOps in our workflows, this becomes especially true.
We want to create and test more features in less time, rather than releasing on a monthly, quarterly, or yearly basis. We want those integrations to be innovative and functional at the same time, rather than releasing updates that cause the software to break. We also want to do all of this in a way that’s cost-efficient.
Often we get stuck and are only able to focus on one or two out of these three priorities. But the true return on investment of adopting automation and doing it successfully is addressing all at once.
So how does automation help answer speed, cost, and quality all at the same time?
We know that automation is supposed to save us time, but how does it do that? Especially when it takes additional time to write the code or create the test, how can we be sure the investment of time is worth the time we save?
Where automation really come into play is with repetitive tasks and repeatable scripts. Say you have a test case that takes about an hour total. Realistically, this isn’t that much time — if you’re only doing it once.
If your team is integrating new code twice a week and you identify 5 configurations that are necessary to run as part of your regression suite, this means this test is taking 10 hours of your work week.
And that’s only one regression test. If you have 5 of these, that’s 50 hours just to check that everything is still working as intentioned; it doesn’t include testing any new features or areas that should be explored.
Of course, you could skip regression testing completely. But there’s always the possibility that when the new integration is pushed live, it will break something that will cost the entire team even more time to fix.
Automated testing takes care of time-consuming tasks so you can focus on testing new areas and building new tests, rather than manually executing the same tests over and over.
In this example, if you were able to automate those 5 tests, you would be saving the time it takes to repeat them in every browser and during every regression. While the initial time investment is there, the return on that investment is greater because you can reuse the scripts instead of manually repeating them.
Not only does this save time, gives you more time to perform new tests and create more automated tests. It also decreases the time it takes to get the test results back to the development team and thus helps to speed up the entire SDLC.
The best way to prove the ROI of automated testing for speed if it’s part of your current strategy is to add up the time it takes to do manual regression testing, then compare it to the time it takes to automate those regression suites.
It can be hard to determine the true cost-benefit of automation, but there are a few ways you can understand ROI when it comes to budget.
A good place to start by thinking about cost in relation to time — as they say, time is money. Quite simply, if you’re paying one or more software testers, you’re getting more for your money with an automation strategy in place to alleviate time-consuming work by replacing it with new tests of value, as we previously discussed.
However, another way to consider cost is to look at how much money you’re saving every time you find a bug before it reaches your customers.
The earlier on you find each bug in the software development life cycle (SDLC), the better according to IBM research, which found that cost to fix a bug increases as it moves through the SDLC.
In fact, the cost to fix an error found after release was about 15 times higher than if it’s uncovered during the design phase. It’s up to 100 times more expensive if it’s identified in the maintenance phase.
For example, if a bug costs $100 to fix in Gathering Requirements phase, it would be $1,500 in the testing phase and $10,000 once in Production.
The idea here is that it’s harder to fix an issue and more expensive as it approaches the end of the development cycle. Once it reaches the customer, it’s the most costly — not to mention intangible costs to your reputation and trust as a company when customers find bugs that you missed.
By getting more testing done sooner and verifying that old and new code is functioning, you avoid the cost that comes with finding a bug as you move down the SDLC.
Improved application quality comes with finding more bugs before new integrations reach customers as we just discussed, but it also accounts for the fact that time allows for more exploratory testing ideally being conducted during each development cycle.
The job of the tester isn’t simply to find bugs. An often overlooked element of the tester’s role is to provide feedback and insight when going through the application to make suggestions and affect changes for future changes and additions to the application. It’s impossible to do this if the tester is spending all their time checking whether the code is stable and functional.
By making more time exploratory testing and leaving automated scripts to check regressions, testers can focus on suggesting improvements, collaborating with programmers, and helping the software evolve, rather than spending the bulk of their time manually going through the same actions on different browsers every time they need to check core functionalities.
Additionally, automation takes out the factor of human error. While manual testing is essential to make observations about the application, tests that are merely for conducting the same steps over and over is left up to risk when done by a person. Automation is the answer to ensuring that your results are consistent and will provide a “Pass” or “Fail” when run.
Automation should never replace manual testing but support it in regression checking to allow for more time for manual tests that require human observation. Not only will this go towards confirming the quality of the current application, it will also aid in making changes to support improved quality in the future.
Additionally, by allowing more time for these manual tests and regression checks, automation helps address broader code coverage, meaning that the application that reaches the customer is more likely to work the way it should.
How to Prove ROI
You can see where speed, cost, and quality come together to help create a more productive development workflow and a better application.
But, if you’re still confused on how to prove ROI, here’s a few actionable items you can take:
- Relate it Back to Your Business – It’s impossible to give one blanket statement for ROI that works for every company in terms of effectiveness. If you’re finding a challenge in convincing members of your team that automation is the right option, do your best to add up the manual hours that could be solved with automation. Identify which tests could make a good case for automation and where automation could help you could save time in your day-to-day.
- Offer a Success Story – There’s a reason why everyone’s talking about test automation. Plenty of individuals, teams, and organizations have found success in adopting automation practices. Take America’s Test Kitchen. Their team found that they weren’t able to cover all the tests they wanted on all the configurations they needed to and keep up with the continuous iterations of the development team. By adopting a Selenium strategy and bringing on an automation engineer, they were able to find a solution to some of these pain points.
- Leverage Parallel Testing – Parallel testing allows you to run the same test in multiple browsers at once, which means it’s extremely easy to prove the ROI. With every test you run in parallel, you’re increasing your test coverage that many times. For example, if you run two tests in parallel, you’re testing twice as much in the same amount of time. If you up it to 10 tests in parallel, you’re testing 10 times faster than running them on after another. Those are numbers that anyone can agree to.
- Do Your Homework – Take the time to learn tools and automation practices on your own time. Understand what will make automation successful and what skills you can learn to make the strategy effective, so once you’re given the green light you can start to see the benefits. Learning any test automation tool does take onboarding and time to learn, but staying ahead of the curve means you’ll be more ready to jump in and start making tests. If you’re already automating to an extent, share the results and insights with your team. Record the amount of time it takes to create tests and how much time it saves, so you can have tangible proof of the ROI.
- Look at the Browser Gap – A major benefit of automation is that it takes out the manual work of repeating a test in different browsers. If you’re not currently testing more than a few browsers, it’s worth it to investigate and see where you could be compromising quality. Check out which are the most popular browsers and devices your customers are using — you may have a gap that automation could help fill.
- Do the Math – If you’re looking for hard numbers, this guide on measuring the ROI of automated testing may be a good place to start. By calculating costs that relate to automation of new tests, automation of prior tests, coverage across environments, reduction of defects, test redundancy and reusability, you can back up your argument for test automation with real data.
The ROI of Test Automation
Yes, automation does take some investments to get started. It’s easy to see why teams can be hesitant to hop on board with automated testing when looking at these factors.
The value comes with the ROI. Why it may not be immediate, you will see the ways that automation aids with time, costs, and quality over time.
As the team becomes more well-acquainted with the ways automation can help move the business forward, it will continue to be more of an asset. In fact, you might begin to question how you ever managed without the help of test automation.