CrossBrowserTesting.com

A Design, Development, Testing Blog

  • Back Home
  • Free Trial
  • Design
  • Development
  • Manual Testing
  • Test Automation
  • Visual Testing

Deciding What to Automate When You Can’t Test Everything

May 25, 2017 By Alex McPeak Leave a Comment

 

prioritizing when you can't automate everything

Though automation is a great method for executing more frequent tests, teams that try to automate everything will quickly find that it’s a herculean task. GUI tests should be carefully prioritized with business-critical critical processes, revenue-generating flows, and common test cases.

As testers new to automation gain more experience, they should be able to prioritize tests, saving organizations time and money.

When Not to Automate

Usability Testing

User experience design (UX) is a human-driven evaluation of the usability of a product, application, or feature. This means the UX can’t be evaluated by automation very accurately because that would take out the aspect of human observation. Using our Screenshot Engine, you can visually verify the UX across different devices and browsers, but we recommend serious Usability Testing be done manually.

One-time tests

This one is pretty self-explanatory, but if there’s a feature you only plan to test once or twice, there’s no reason to try and automate it. Automation is primarily used for regression and repeated tests, and best saved for more stable code bases. For example, an ad-hoc test may just be used to check something without the need for extensive documentation and a formal test structure.

Brand new features

Smoke testing and exploratory testing are better for testing a new feature than writing full end-to-end tests for it. Automation is great for reducing testing time, but it doesn’t act as an end-user the way a manual tester is trained to do. Automation scripts won’t seek out new bugs; see below. Once the UI of that feature is stable, it’s in a better state to be tested across browsers and when small code changes are made.

Exploratory testing

Automated tests follow a script, they don’t explore the software to experiment with an application and break features. Unfortunately, since users aren’t robotic they won’t always know to use software the way it’s intended. By exploratory testing, manual testers can go in and consider more creative possibilities of how a user will handle it and execute negative testing.

What Test Cases to Automate

Regression tests

Again, the most basic use of automation is for regression tests. Are there certain test scripts that you know you always execute with every deployment? Did you just get to a stable point in your integration tests of a new feature and want to speed testing. That’s where automation comes into greatly reduce the time it takes, thus effectively assisting the ability for quality deployments in a CI/CD environment. Regression testing allows you to build a test suite that ensures quality with every iteration and change to the code, pinpointing where possible new bugs may be when integration systems or deploying new features.

Unit tests

As the base of Mike Cohn’s Test Pyramid, unit tests are great for automation because they’re the smallest kind of test (Hint: you should have a lot of them). This also encompasses the motto of testing “early and often” that’s a foundation of any good testing strategy.

High traffic pages

You want to have a good reputation in the eyes of your users, and a bug or broken link on your homepage can be embarrassing. If your analytics show that your traffic is converting through your webinar section then you should prioritize that page before your blog, making sure customers are able to sign up and join them.

Impact on customers

This is similar but slightly different than targeting highly trafficked pages. Will your customers be more likely to leave your site if they don’t have a good experience during check out compared to browsing inventory pages? Consider pages with high bounces rate, or pages that have a high ROI – like your free trial or contact us page.

Data-driven tests

Manual testing is great for UI, but it’s not a very data-driven approach, more so relying on human intuition and curiosity. If you are looking for specific test inputs to tell you exactly where bugs lie automation will give you a more accurate portrayal.

Cross-browser testing

You’ve tested your web app, but if you only tested in one browser, you’re missing the mark for broader coverage. This is a simple way to make sure that your web application looks acceptable to a wide array of users on different devices, operating systems, and browsers, which is cross-browser testing. However, unless you want this to take hours or days of your valuable time, your best option is to automate parallel tests. To learn more cutting down test time, read our guide to parallel testing.

Stop Trying to Automate Everything

Deciding between manual and automation can be pretty straightforward. Manual testing is most often used for more detailed human observation, while automated tests are a faster and more data-driven approach for repeated tests.

In addition, sometimes the line can blur between which tests you should execute first in both situations. Our best advice is to always consider the end-user first. Think about which tests will best ensure usability, functionality, and performance for them specific to the stage of development your software is in.

By making both manual and automated testing part of your ongoing QA strategy, you can establish that your software continuously is known for optimal quality among your user base.

How do you decide what tests should be automated first? Tell us how you prioritize in the comments!

Filed Under: Test Automation Tagged With: automated testing, manual testing, test management

About Alex McPeak

Alex McPeak is a Content Marketing Specialist for CrossBrowserTesting and is always looking to provide insights in testing, development, and design for the software community, appearing in outlets such as Abstracta, DZone, and Ministry of Testing. She's especially interested in writing about the latest innovations in technology and is forever #TeamiPhone.

Leave a Reply Cancel reply

Your email address will not be published. Required fields are marked *

Try CrossBrowserTesting

Everything you need for testing on the web. Browsers & Devices included.


  • Grid In The Cloud
  • Simple CI Integrations
  • Native Debugging Tools
  • Real iOS and Android
  • 1500+ Browsers
  • Intuitive REST API

Start Testing Today

Top Posts

  • Selenium 101: How To Automate Your Login Process
  • What’s the Best Programming Language to Learn First? It Depends
  • Selenium vs. Cypress: Is WebDriver on Its Way Out?
  • How To Use Google Analytics To Find Out What Devices Your Customers Use
  • Selenium 101: Managing Multiple Tabs

Want Testing Tips?

Want the latest tips & strategies from industry experts right to your inbox? Sign up below.

Join Over 200,000 Testers & Developers And Start Testing Today

Learn more Free Trial

Features

  • Live Testing
  • Selenium Automation
  • Automated Screenshots
  • Screenshot Comparison
  • Local Testing
  • Real Devices

Solutions

  • Automated Testing
  • Visual Testing
  • Manual Testing
  • Enterprise
  • Internet Explorer

Resources

  • Browsers & Devices
  • Blog
  • Webinars
  • Integrations
  • ROI Calculator

Company

  • About Us
  • Careers
  • Plans
  • Terms of use
  • Security

Support

  • Help Center
  • API Docs
  • Schedule A Demo
  • Contact Us
  • Write For Us