Over 5,000 professionals in Software Development, QA, and Testing responded for the SmartBear 2017 State of Testing Survey to provide insightful feedback into how they organize their testing strategy and execute daily responsibilities.

Of special interest to the CrossBrowserTesting team were findings among browser testing, mobile device testing, parallel testing, and cloud testing, which we’re highlighting here.

Browser Testing

According to the State of Testing report, four out of five teams test on multiple browsers, but half only test on the latest versions.

It’s evident that the majority of testers know the importance of running tests on more than one browser. However, the fact that many don’t find the need to test on previous browser versions can hinder an otherwise thoughtful cross-browser testing strategy.

Older browsers are often the most problematic when it comes to rendering code. If browsers are not updated automatically and users don’t take the time to update, their browser is unaccounted for in testing, even if they’re just using an earlier version of Chrome.

Bugs can pop-up even with smallest browser update, sometimes in the decimal level versions. Of course, this becomes a bigger issue the older the browser is, which is why it’s in the best interest of testers to add more than one browser version to their browser testing arsenal.

Additionally, the largest percentage of respondents (27%) test on only three browsers. Again, although teams are seeing the value in testing more than one browser, it’s not quite enough. Considering that teams are only testing a few browsers in the latest version, this leaves a significant gap in a lot of organizations’ web testing approach.

Mobile device testing

Two-thirds of survey participants reported test on mobile devices, and 80 percent of respondents reported testing on more than one mobile device. Unsurprisingly, the most common mobile devices tested are Androids at 92 percent and iPhones at 85 percent.

The majority of testers — 60 percent — are only testing on two different devices. Fortunately, 64 percent of mobile UI testers do not exclusively test on the latest configuration of mobile devices, which means they are incorporating some good practices when thinking about web and browser testing.

Still, 36 percent of testers only focus on the most recent device iteration, which means they are often missing large segments of their user base. Though organizations are making a noble effort to incorporate mobile testing for an increasingly on-the-go consumer, there is plenty of room to expand mobile device testing.

Incorporating older iPhones, for example, accounts for users who have not upgraded to the most recent version. As we’ve pointed out in the past, the array of options on the Android market is ideal for consumers who like to tailor their choices, but it complicates the job at hand for developers, requiring testers to target more diverse platforms.

The issue of fragmentation causes more drastic differences in screen resolution and performance due to varying operating systems, models, and brands, which means that the more Android devices included in a mobile testing strategy, the broader the coverage will be.

Parallel and Cloud Testing

The survey showed that the majority of software teams are only running a few tests in parallel, or none at all.

Seventy-seven percent (77%) of respondents are doing some parallel testing by running at least two UI tests in parallel, with the highest percentage of respondents (35%) saying they run two to five UI tests in parallel. The second highest response, however, was running no parallel tests at 23 percent. Though the majority of teams are parallel testing, they are evidently not running enough tests in parallel.

Those running parallel tests are also more likely to use a cloud service for testing than those who don’t do any parallel testing. Fifty-six percent (56%) of respondents who reported doing some parallel testing also reported running some tests in the cloud, while respondents who don’t do any parallel testing also don’t run any tests in the cloud.

Only 30 percent of organizations utilize a cloud service for testing, while 34 percent reported not running any tests in the cloud. This suggests that the total number of organizations utilizing both parallel and cloud testing hovers around 43 percent.

Where parallel testing allows test scripts to run concurrently on multiple browsers and devices through one environment, cloud services offer a great compliment, allowing testers on demand testing environments to run these parallel tests. Though it’s promising to see teams leveraging both parallel and cloud testing, it’s obvious the value of these methods have not yet been fully recognized, and there is ample opportunity to incorporate more of it in order to speed up testing times and increase quality.

Other Takeaways

In addition to the preceding topics, the State of Testing Survey covers trends in other areas including API testing, automation, and development, among others.

A few other takeaways we found interesting include:

• Web applications are the most common type of application being tested at 85 percent, followed by APIs, desktop apps, and mobile web apps.
• Agile is the dominant model for software development, but most teams that describe themselves as Agile are only releasing on a monthly basis. This is not ideally part of Agile best practices, which encourages frequent releases, and this response in the survey calls into question how teams define Agile.
• Test automation is growing as a practice, but still, less than half of tests (44%) are automated on a daily basis.
• The top challenges testers face with UI test automation include test stability, object identification and management, and test maintenance.

The trends reflected in the SmartBear State of Testing Survey reveal not only the processes, strategies, struggles of software teams today, but also imply the future of these trends and suggest ways to improve current procedures.

By understanding the full potential of software teams and tools among industry shifts, organizations can better prepare to meet the needs of a faster development and delivery cycle.

To explore the extent of SmartBear’s 2017 findings, you can read the full State of Testing report here.