With the conception of responsive web design, it’s become easier to build and enjoy the mobile web on any screen or device. As more and more users carry higher expectations from their mobile experiences on an increasing number of smartphones, understanding the principles of designing responsively is no longer a unique skill, but an essential one.
However, as developers and designers start thinking mobile-first, somebody has to make sure that it actually performs accordingly. Any experienced mobile testing strategist knows that it takes more than one test on one device to provide broad mobile coverage. In fact, mobile testing takes a comprehensive strategy and is becoming more complicated with these changing user behaviors.
That’s why we’ve prepared this guide that includes the four steps to lead you to a successful mobile test strategy. By better understanding the necessary parts of mobile testing, you can be more prepared to create an exceptional mobile experience for every user.
1. Consider usability from a mobile perspective – You want to have a great design for your mobile application, but at the end of the day, customers are going to either stay on your app or leave it based on usability. The UI should be customized for mobile users, which means it’s going to be different than the desktop experience and has to be tested that way.For example, form filling proves to be much more difficult for a mobile user. If a form isn’t optimized correctly, users will often encounter issues where forms are hard to fill out with their touch and keyboard, whereas they don’t have issues on a laptop or desktop computer. Considering how people view and use content on a small screen is key to derive insight for testing. Additionally, interrupt conditions (i.e. what happens when a notification or other actionable pop-up takes you out of your current task) don’t affect desktop users, but they need to be tested on mobile applications to evaluate functionality and performance.
For example, form filling proves to be much more difficult for a mobile user who doesn’t have the luxury of a keyboard or mouse. If a form isn’t optimized to fit the screen, users will often encounter issues where forms are hard to fill out with their touch and keyboard. Considering how people view and use content on a small screen is key to derive insight for testing. Additionally, interrupt conditions (what happens when a notification or other actionable pop-up takes you out of your current task) don’t affect desktop users, but they need to be tested on mobile applications to evaluate functionality and performance. These are the components you have to consider that you may not think about when you’re using a laptop computer if you’re testing for mobile users.
Not to mention, testing for UX will become more difficult as mobile devices become more diversified. Device fragmentation is already an obstacle for most testers. In a perfect world, it would be possible to test on every device, browser, operating system, and even screen resolution. In reality, there are thousands of configurations and that just isn’t plausible. Today many phones have similar designs and screen sizes, which means that most people can get an adequate account of their cross-browser compatibility with a few phones. However, with new releases such as the iPhone X and Google Pixel XL, it looks like mobile devices are trending towards bigger screens and narrower bezels, making you wonder what the future of digital media consumption will look like. As designers have to start planning for usability concerns like the iPhone’s notch, testers will be tasked with diversifying their device labs.
2. Utilize your toolkit and testing knowledge – You don’t have to go mobile alone. In fact, there are a plethora of tool specifically to make your life easier when testing. When thinking about open source options, Appium is the way to go for most testers who want to automate. This means that you don’t have to perform the same manual actions over and over again on every new device, but it does take some skills, such as coding. Learn more about getting started with automation as a manual tester here. Some testers also use other free tools like Robotium, Selendroid, MonkeyTalk, Calabash, and Frank.
There are also paid options that can help make your testing strategy even better (and faster). TestComplete allows you to create and run repeatable and robust UI tests across iOS and Android apps. CrossBrowserTesting, on the other hand, lets you run automated parallel tests across multiple browsers, devices, and OS at once. But you can’t forget about manual testing either. With over 1,500 browser configurations available for live testing, you can go through and debug on different devices from the comfort of your preferred desktop browser. Additionally, our full-page screenshot comparison tool makes it easy to spot inconsistencies and remotely debug across browsers. And, the best part is that the integration of these tools allows you to use them together.
Of course, deciding when you should test manually and when you should be automating is something you have to use your knowledge of testing best practices. It won’t do you any good to show of your test automation skills without running any exploratory tests first. However, if you depend solely on manual testing you probably won’t get much done in your workday, or if you do it won’t be on very many different devices. Either way, it’s not a great strategy. Look over this resource for a refresher on the best times to test manually vs automating.
3. Pick & prioritize configurations – It’s not just enough to have the right testing tools, you also have to have the right devices to test on. This isn’t a time to trust your intuition — collecting data and analytics on user behavior will be the best way to figure out which screens you should be running your application on.
Google Analytics is a popular method to look at the mobile devices your site’s visitors are using the most. You can use Google Analytics to see what percentage of users are accessing your website on mobile, as well as more specific data such as the device brand, model, service provider, operating system, and screen resolution. You can also use secondary dimensions to see what browsers they’re using. Additionally, you want to make sure you’re testing older browsers that may be more problematic because that’s the point of testing — catching bugs wherever you suspect they may be.
Another way to account for current and potential users is to look at the most popular browsers. You can do this either by region — for example just looking at the United States — or by looking at the most popular configurations worldwide. For example, according to NetMarketShare, Chrome 60, Safari 10, Android Browser 4, Chrome 59, and Chrome 61 are the most popular mobile browsers right now. Testing a number of these popular browsers means you can be sure that you’re targeting the majority of users that will likely land on your website instead of just the people who already have. By looking at both your user data and the most popular configurations, you should be able to get a good handle on where you should be testing.
4. Choose between emulators, simulators, and real devices – Emulators, simulators, and real devices all have their advantages and disadvantages, but ultimately, they all have their unique purposes and use cases. For the very best mobile test strategy, you should ideally have a mix of emulators, simulators, and real devices. This way, you can approach speed, accuracy, and scale at once.
Emulators and simulators both mimic the functionality of real devices and can imitate a user’s actions. While emulators are a little closer to the real deal since they replicate the software, hardware, and OS, simulators come pretty close to copying the internal behavior, although they do not emulate hardware or work on the OS. While both emulators and simulators are assets for fast mobile testing with unavailable devices, they don’t always encounter the same environmental factors as a real device and therefore aren’t always as accurate because many factors can affect the validity of the results.
Using real devices, on the other hand, means that you’re testing the same way your customers are using their devices, so accuracy is spot-on. This can be especially important when looking at performance or usability. To achieve the precision of real devices, though, you need a real device lab. Whether that be a physical device lab you build yourself or one you access remotely in a cloud is up to you, but it does generally require investing in more costly resources.
Not just anybody can be a great tester, and similarly, not just anyone can create an effective mobile test strategy. However, if you want the large percentage of users that visit your application on their mobile devices to be able to interact and engage, knowing your stuff when it comes to mobile testing is key.
A mobile test strategy isn’t built overnight, but understanding the different components will help you start from the ground up. By understanding how to test on mobile devices, which tools to use, and which machines to test on, you can release new integrations with confidence that users won’t be subject to any wacky behavior that desktop users don’t have to deal with.