Testing an application solely on the machine it was developed on won't do much to tell you whether or not it’s cross-compatible. But once you decide between using a Selenium grid or a cloud tool like CrossBrowserTesting, there still comes the issue of deciding which browsers to test. After all, you can’t test everything.
The following are some of the best methods for picking which browsers, operating systems, and devices are going to be most beneficial to your team in testing. By choosing configurations to account for as many users as possible, you can form an intelligent desktop and mobile testing strategy.
Browsers and Device Selection Through Analytics
Using Google Analytics is a precise method of finding online user data because it looks at who’s actually coming to your website and how.
To get a list of the devices your users on on with Google Analytics, you’ll want to go to “Audience” on the left side, click down to “Mobile”, and then click “Devices”.
This will show you how many people are visiting your application on what kind of mobile device, and you can get more information by setting a secondary dimension to view additional data such as service provider, operating system, mobile brand, and screen resolutions.
Additionally, you can go to “Overview” under “Mobile” and set the Secondary Dimension to “Browser” to see find out what browsers are being used on desktops, mobile devices, and tablets.
You can also find out what browsers your customers are using by going to “Technology” under “Audience” then clicking “Technology” and “Browser & OS”.
If you select “Operating System” as the Primary Dimension and set the Secondary Dimension to “Browser,” Google Analytics will give you the most common configurations. You can also set the Secondary Dimension to “Browser Version” when on “Browser” as the Primary Dimension for more specific browser data that will tell you how what version users have.
As far as understanding your users goes, this information gives you the most accurate glimpse of the devices your users are already accessing your web application on. After looking at this data, make a list of some of the top browser, operating system, and device combinations to prioritize during testing.
You can also use other analytics engines, such as Mixpanel, to get an understanding of which environments your users are on.
Most Popular Browsers and Devices
While Google Analytics is a great way to understand the current users that come to your website, it’s not very predictive nor does it account for your potential users. As your organization grows, you should expect your traffic and the diversification of browsers and devices to expand as well.
Because of this, it’s a good idea to include a few of the most popular browsers and devices even if they aren’t seen on your analytics list. If a high number of people use them, it’s logical that you want your web application to support them as well.
Look at resources such as W3Schools, W3Counter, NetMarketShare, StatCounter, and Statista for some of this data. You'll likely find that it's a good idea to test the most recent versions of Chrome, Safari, Firefox, and even some older Internet Explorer versions. Additionally, Testing Windows, Max, and popular mobile Android and iOS operating systems will prove important.
You also may want to think about popularity by regions. If you your organization is international, you will probably want to prioritize Android phones when thinking about testing mobile devices since they lead in worldwide popularity. However, if your customer base is mainly located in the United States, you’d want to focus on testing iOS.
This data will continually change from country to country and might even change month over month, so you will want to continuously check the market share for your target market.
Testing the Old and the New
If you really want to touch all the bases, it’s important to include a few legacy browsers. Older browsers and devices are going to cause more issues for your application, so while it’s good to check popular models, you also want to test on ones that are less popular but still capture a percent of your users.
You also have to consider browser and devices that may be new to the market. While the number of machines you have now may seem like it could get the job done, new browsers and devices come out on the time. If your users have them, you want them too.
Statistics and analytics might not always reflect newer options, so it's your job to stay on top of what's new on the market.
When thinking about testing our applications, we want to target pages that get high traffic and business crucial functions, but we also want to test areas that we know are problematic in order to avoid expected downtime. The same is true with browsers.
While you may find that your users are saturated on the most recent versions of Chrome and Safari but you also acknowledge that your application has difficulties in another browser, it’s probably a good idea to add that configuration to your mix.
Internet Explorer, for example, may not take up a large portion of your user base, but it’s known to be one of the most problematic browsers.
Even if you aren’t running tests on 100 browsers every regression, it’s a good idea to do some exploratory testing and visual testing on a wide array of browsers and operating systems in order to get a better understanding of which ones are most likely to create issues for your team.
Logging and documenting this information so you can that browser is included in regression testing will also help ensure reoccurring bugs don't slip through to your users.