Recently, Bas Dijsktra of On Test Automation joined the CrossBrowserTesting team in the webinar “Lighten Up: Methods for Making Your Automation Suite Leaner ”.

Bas’s presentation was jam-packed with important lessons and actionable takeaways. To get the most out of it, you should take the time to watch the full webinar at the bottom of this post.

But we’re also rounding up a few of the most important lessons we learned from Bas. Here are our top five:

1. There’s such thing as too much automation – With the introduction of tools like Selenium that allow us to automate at the UI level across different browsers, it can be tempting to automate anything we know how to write a test for. However, automation isn’t always the answer, and if you’re automating too much, it may be more of a problem than a solution when your automation suite becomes bloated. This can mean slow tests, shallow feedback, and high maintenance costs that reverse the benefits of automation. Additionally, when your suite is too heavy with automation, it no longer helps to shorten your feedback loop, and will likely add time instead. In order to get the most value out of your automation suite, Bas says that it’s important to understand where you should be testing at the UI level, when you should be testing at a lower level, and when to skip automation altogether.
2. We have some unrealistic expectations around automation – Automation should never be the end goal of our testing. Bas advocates that we stop trying to automate everything and start focusing on automating the right things. Prioritize the business value rather than automating just to automate. Understand that not everything testers do can be automated, and not all test should be automated. Know the limitations of automation and your testing tool, and familiarize yourself with the tests that could be taken out of your suite to lighten your load to improve your feedback loop.
3. Automation requires resources – One of the reasons why it may not be worth it to automate is return on investment, or ROI. Automation requires resources to write, maintain, run, and analyze tests. Bas says that this is too often forgotten when teams strive for “complete automation”. You have to decide whether it’s worth it to expend this energy and resources on automation, or if it would be best spent somewhere else. For example, maybe it would be more valuable to do an exploratory test of your application and do a deeper dive into areas you can’t assess with automation. Ask, “Are we comfortable not having the information provided by this test?” If you are, it might be a reason to redirect your resources away from automation.
4. Not everything should be tested from the end user perspective – UI testing is great to test the end user perspective, but it’s not always the best method. You should automate as close to the implementation as possible and only do user interface-driven end-to-end automation when verifying end-user journeys. Ask, “where is the logic that we’re verifying actually implemented?” Often times you can run a test in the UI, but it won’t give you the feedback you’re looking for. In his example, Bas shows there are instances where testing at a lower level will give you more accurate test results.
5. You might not need to run your automation suite as often as you think – UI tests can take a long time to run, and with multiple browsers and operating systems, it can take even longer. While tools like CrossBrowserTesting allow you access to a seemingly unlimited amount of browsers, that doesn’t mean you have to test in them all every time. Bas encourages you to decide which browsers you need to test and how often by looking at which your users are on and which are most problematic. If you think there’s value in testing in a browser or operating system but not adding it to your regression suite, CrossBrowserTesting lets you look at those same configurations in on-demand so you can test manually, as well. Bas also suggests moving tests out of the build pipeline and into an overnight job, so you’re not spending as much time running your suite.

Conclusion

As teams become better acquainted with automation, they have certain choices to make about what to automate, how to automate it, and why it should be automated. Unfortunately, sometimes we don’t ask these questions first and end up with a bloated test automation suite. Bas Dijkstra gives valuable advice for considering will actually set you up for success and give you more valuable test results.

To watch “Lighten Up: Methods for Making Your Automation Suite Leaner ” play the video below.