You’ve heard it a million times before — you can’t automate everything. But if your organization is just starting into the swing of automation, they’re probably thinking about all the ways they can automate. So how do you make sure you maintain the proper balance of manual testing and automation?
Whether you have an entire QA team whose roles are split between automated and manual testing, or there is one tester whose jobs is to do both, striking the right balance will be the difference between a testing strategy that provides value to your team and one that gives you more frustration than feedback.
By understanding how to best support your manual strategy with automation, you can ensure you’re getting the best of both world.
Know the Limitations of Automation
For example, CAPTCHAs are often very popular with ecommerce sites. However, they’re inherently impossible to automate. This means that any test case that includes a CAPTCHA can’t be automated, and manual testing has to step in.
This, of course, isn’t the only case where you won’t be able to use automation. There are plenty of instances that it will be possible to automate, but it will be so complex to script, that you’re better off doing it manually.
Assess the limitations and challenges of test automation so you know where to step in with manual testing.
Solidify Your Regression Suite
The bottom line for automation is that you want to automate tests that are repetitive and time-consuming, so think about which tests you find yourself going through the most often. Solidifying your regression suite will help you determine which tests you want to cover after every integration or change to code.
Keep in mind that this will change over time as your application changes, but it’s helpful to have a suite that ensures basic functionality.
Additionally, deciding which environments you want to test on will be key to understanding which browsers, devices, and operating systems will be included in your regression suite.
Don’t Sweat the Small Stuff
Small test cases are best for automation and give you tests that are reusable and maintainable. The results of your automation are dependent on the state of your application, which means the smaller and more specific the test is, the more resistant it will be to changes in the UI. This is why unit and integration tests are ideal for automation.
But longer, more complex test cases and entire user journeys are often better left for manual testing. This gives you more room to explore the application and requires less time for scripting. Moreover, it means that if the UI of the application changes, you won’t have to go back into your script and figure out where it needs to be adjusted.
What’s the Risk?
When you’re thinking about whether or not to automate, risk should be a leading factor for consideration. However, while there are many methods for evaluating which tests are the most at risk, identifying these doesn’t mean you want to leave all other test cases that aren’t high risk out completely. In fact, these are perfect for manual testing.
When you considering risk, there are usually three tiers that you sort test cases into — high, medium, and low. As we discussed, most high-risk test cases will probably be included in your automation suite.
But, if you’re finding medium and low-risk cases that you still find valuable to test in order to provide feedback, then they might be something you want to run through manually.
Is It Providing Value?
Automation is a great way to speed up testing, but it also takes time for coding and maintenance, and that may not be worth it if you implement a test that isn’t adding value.
Say you run a test suite every day, and it passes 100 percent of the time. Is it worth it to keep automating if your results never change?
Just because a test was once automated doesn’t mean it should stay that way. Automation requires making changes, updates, and edits as the application code changes.
If a test is no longer providing value through automation, it might be better to take it back to manual testing to see if there are new parts to explore.
Communicate With Developers
Developers can’t tell you what to automate and what not to automate, but they can give you insight into the state of the application.
Keep open lines of communication with developers and other team members so that you are clear on what’s being changed in the application during development cycles. Depending on whether integrations are minor changes to the code, completely new features, or a revamped UI will make a difference in whether the test that follows will be part of your automation suite or should be tested manually.
To get additional insight, it may be helpful to pair with developers when going through the application to get a more in-depth look at what has changed and what areas or functions need to be tested.
Exploratory testing is crucial to visually and functionally verifying new features and changes to the UI. Automation frees you up to test parts that you wouldn’t normally get to, so take it as a challenge to find a new bug or test for a different persona.
Human observation, curiosity, and creativity are paramount to being a tester. Automation can only take you so far when it comes to finding new bugs. That’s why manual testing, and especially exploratory testing, will always have a place in the QA process.
Functional automated testing is not the only thing you have to worry about. Oftentimes, a functional test will pass because your application is working correctly, but it could still be visually inconsistent across browsers. Not only could this affect your layout and design, it could also affect the way your users are able to interact with your application.
In addition to functional testing with Selenium, you should incorporate automated screenshot testing into your workflow. This will help you account for visual cross-compatibility issues that you may have missed with functional testing by allowing you to compare screenshots of your web application side-by-side in different browsers.
Increasing Test Coverage
Automation will take care of repeated sequences, but in order to increase test coverage, you want to use manual testing to look at areas that have not yet been tested.
In this way, automation can be used as a tool to gradually help you increase test coverage over time. Test high-level functionality with automation, then fill in the gaps with manual testing. Look at edge cases and try negative testing to go beyond pass and fail results to gain deeper insight into your application.
This is also a good time to figure out what new tests you would like to add into your automation suite, so you can continue to increase coverage. Evaluate manual test cases to determine whether they would be valuable in regression, and assess which tests may be missing from the suite already.
At the end of the day, it’s best to consider return on investment when deciding whether or not to automate. Automating takes time to analyze, code, and maintain.
If the time it takes to automate isn’t worth the results that you’d get from the test, it may be something you want to do manually, or not at all.