Though automation is a great method for executing more frequent tests, teams that try to automate 100 percent of their tests will quickly find that it’s a herculean task. GUI tests should be carefully prioritized with regards to business critical processes, revenue generating flows, and common test cases.
As testers new to automation gain more experience, they should be able to prioritize tests, saving organizations time and money.
When Not to Automate
Usability Testing
User experience design (UX) is a human driven evaluation of the usability of a product, application, or feature. This means the UX can’t be evaluated by automation very accurately because that would take out the aspect of human observation. Using our Screenshot Engine, you can visually verify the UX across different devices and browsers, but we recommend serious Usability Testing be done manually.
One-time tests
This one is pretty self-explanatory, but if there’s a feature you only plan to test once or twice, there’s no reason to try and automate it. Automation is primarily used for regression and repeated tests, and best saved for more stable code bases. For example, an ad-hoc test may just be used to check something without the need of extensive documentation and a formal test structure.
Brand new features
Smoke testing and exploratory testing are better for testing a new feature than writing full end-to-end tests for it. Automation is great for reducing testing time, but it doesn’t act as an end-user the way a manual tester is trained to do. Automation scripts won’t seek out new bugs; see below. Once the UI of that feature is stable, it’s in a better state to be tested across browsers and when small code changes are made.
Exploratory testing
Automated tests follow a script, they don’t explore the software to experiment with an application and break features. Unfortunately, since users aren’t robotic they won’t always know to use software the way it’s intended. By exploratory testing, manual testers can go in and consider more creative possibilities of how a user will handle it and execute negative testing.
Tests to Prioritize Automating
Regression tests
Again, the most basic use of automation is for regression tests. Are there certain test scripts that you know you always execute with every deployment? Did you just get to a stable point in your integration tests of a new feature and want to speed testing. That’s where automation comes into greatly reduce the time it takes, thus effectively assisting the ability for quality deployments in a CI/CD environment. Regression testing allows you to build a test suite that ensures quality with every iteration and change to the code, pinpointing where possible new bugs may be when integration systems or deploying new features.
Unit tests
As the base of Mike Cohn’s Test Pyramid, unit tests are great for automation because they’re the smallest kind of test (Hint: you should have a lot of them). This also encompasses the motto of testing “early and often” that’s a foundation of any good testing strategy.
High traffic pages
You want to have a good reputation in the eyes of your users, and a bug or broken link on your homepage can be embarrassing. If your analytics show that your traffic is converting through your webinar section then you should prioritize that page before your blog, making sure customers are able to sign up and join them.
Impact on customers
This is similar but slightly different than targeting highly trafficked pages. Will your customers be more likely to leave your site if they don’t have a good experience during check out compared to browsing inventory pages? Consider pages with high bounces rate, or pages that have a high ROI – like your free trial or contact us page.
Data-driven tests
Manual testing is great for UI, but it’s not a very data-driven approach, more so relying on human intuition and curiosity. If you are looking for specific test inputs to tell you exactly where bugs lie automation will give you a more accurate portrayal.
Cross-browser testing
You’ve tested your web app, but if you only tested in one browser, you’re missing the mark for broader coverage. This is a simple way to make sure that your web application looks acceptable to a wide array of users on different devices, operating systems, and browsers, which is cross-browser testing. However, unless you want this to take hours or days of your valuable time, your best option is to automate parallel tests.
Conclusion
Deciding between manual and automation can be pretty straight forward. Manual testing is most often used for more detailed human observation, while automated tests are a faster and more data-driven approach for repeated tests.
In addition, sometimes the line can blur between which tests you should execute first in both situations. Our best advice is to always consider the end-user first. Think about which tests will best ensure usability, functionality, and performance for them specific to the stage of development your software is in.
By making both manual and automated testing part of your ongoing QA strategy, you can establish that your software continuously is known for optimal quality among your user base.
How do you decide what tests should be automated first? Tell us how you prioritize in the comments!