Test automation is great, but even with the ability to cover more parts of your application in less time, testing everything is impossible. This is why you have to perform a risk analysis of your web application in order to decide when certain test cases get performed and how often.
There are many levels to creating a risk analysis, and deciding which tests go to the top level depends on largely on your business needs and the state of your application. However, these five considerations will give you a good place to start when trying to decide your most important tests.
- Most Popular and Problematic Browsers – Take a look at Google Analytics or Mixpanel to see which browsers, operating systems, and devices are being used most often to access your web page. The number of configurations you include in your testing will depend on how fragmented your user base is. You want to ensure basic functionality for at least 80 percent of your users, so if 80 percent are between five browsers, you can get away with including those five in your first tier of testing. However, if it’s split between ten configurations, you’ll want to allow more time for cross-browser testing. If you do find browser use is highly fragmented, parallel testing is a good way to check off your boxes in multiple configurations at once. But we’re talking about risk here, so you also want to think about which configurations cause the most problems for you. IE 8 always breaking? Make sure to include it in your first tier.
- Most Critical Unit and Integration Tests – Which tests will have the biggest impact on your bottom line if broken? For example, if you’re running an e-commerce website, the ability to enter payment details and submit them for processing is clearly important for allowing users to make a purchase. If they aren’t able to make a purchase and the application fails when they try to submit this information, they won’t be giving you their business and will probably go to a competitor. On the other hand, this will be different if you’re working in healthcare, finance, or government, for example. Identify the unit and integration tests that make up the foundation of your business as well as your application and associated risk on a scale from minimal to critical in order to determine which place at the top of your risk analysis.
- Most Common Landing Pages – Ask any marketer — a big part of the success of any web application is a user’s first interaction with it. This means that you should be looking at the pages with the heaviest traffic to determine which should be tested most often in regression. Choose five to ten pages that users land on most often. Whether that includes the homepage, pricing page, or the “about us” page, you want to make sure these are consistently up to par with a healthy mix of functional testing and visual testing.
- Security and Privacy – Ensuring security in any instances where personal information is being requested from your users should be of top importance in your risk analysis. Take note of any times where payment is being received, passwords are being entered, contact information is input, or confidential information is requested. In addition, any part of the application that may have an impact on the safety or financial status of your users should always be a priority to ensure that their data is uncompromised. Just ask the companies that have been victim to some of the worst software bugs in history — once you lose user trust, it’s hard to get it back.
- Extended Downtime – Which functionalities will take the longest for your team to get up and running again? While your users should usually be your first priority, sometimes risk goes beyond their inconveniences when it puts the entire team on a backlog. If you already know it’s not going to be an easy fix that could affect the application downtime for days and team productivity for weeks, it should probably be high on your list when thinking of risk. Where have you had issues in the past? History repeats itself in life and your application. Log and report areas that have a high probability of failure so you can prepare your team and avoid excessive repair time.
You have to be familiar with the risk associated with different functions and features in an application to know what should be tested first, what should be tested again, and what you can deprioritize.
While you can never guarantee that code will be deployed with zero bugs, you can develop a strategy to ensure that any issues that do slip through the cracks won’t be detrimental to your application.