Whether you’re a freelance designer or a full-time QA engineer at a large company, you know the value of browser testing. If you decide to DIY, you might be considering building a device lab to ensure breadth, quality, and functionality across multiple devices. While we’d rather you use a platform like CrossBrowserTesting, and believe there are countless advantages, some companies have budget constraints or simply can’t take their testing to the cloud for security purposes.
But, before you go in headfirst and buy all brand new iPhones and Androids, there are a few questions you need to ask, steps you need to take, and follow-up procedures you need to prepare for.
Device Lab Basics
What is a device lab? – A device lab is an environment where physical devices such as PCs, smartphones, and tablets are held and maintained to support testing and development needs.
Who uses device labs? – Anyone who is concerned with a website or application including web designers, developers, product managers, testers, and other software QA engineers or teams.
Why do I need a device lab? – Users aren’t on all the same high-performing devices that developers know to use. Having real, physical devices is pretty much the best method for conducting testing that represents a wide user base and accounts for responsive design.
When should I use a device lab? – Having a device lab is necessary once you realize you need to be testing on more than one device and is a more accurate alternative to virtual machines.
Once you decide that a device lab is necessary to your testing strategy, you have to evaluate how it will benefit you and your team in order to consider more specific requirements.
Planning and Research
Buying a device lab isn’t cheap. In fact, the average Android cost about $670, while iPhones come in at around $700. Just a few mobile devices will already put you in the thousands, but a more expansive device lab can put quite the dent in a company’s allowance.
Before anything, you need to decide on a budget for the amount of devices you’re going to buy, but it’s crucial that you first learn what devices your actual users are on so you don’t waste your time and money on the wrong devices.
The easiest and most effective way to see the devices, operating systems, and browsers your users are on is with Google Analytics. Google Analytics data shows the preferences of your majority traffic so you can test the exact devices visitors are accessing your web application on without the guesswork.
Furthermore, when testers largely live by the mantra to expect the unexpected, you should throw in a few older device and browser configurations for good measure, since that’s where the bugs are.
Additionally, you have to consider complementary expenses, since unforeseen costs can add up quickly. Maintenance costs, electricity, and Wi-Fi are just a few of the components that should be examined in cost evaluation.
For example, while putting all your devices on a general Wi-Fi network could affect performance and give inaccurate results, it may be best to have a designated device lab Wi-Fi if you have a good number of devices. Meanwhile, having all those devices running and consistently charging will surely rack up the electric bill.
Organizing and Maintaining Devices
Your work isn’t done after the logistics are all sorted; the fun has just begun!
Your device lab will require substantial organization and maintenance to prove useful, especially if it’s being accessed by multiple team members.
First of all, the degree of security your device lab requires depends on how many people will be using it.
For example, if an entire company can access a lab, you want to make sure the Wi-Fi network and devices are password protected so that only authorized users can get into them. You also need to have some sort of documentation or security to record who uses them to prevent theft or abuse.
There are also less severe breaches that will affect your test results if you don’t pay attention to the details. You want to make sure that each device is set so that its operating system doesn’t automatically update as they’re usually programmed to and that device lab users don’t initiate updates.
You also have to make sure the device lab is organized for easy use. Labeling features of devices including make, model, screen resolution, and operating system will ensure they don’t get mixed up. Meanwhile, grouping devices by provider, size, or operating system will make it easier to find what you need when you need it.
Additionally, you need to find a setup that works to hold your devices. Whether that be a custom-made bookshelf like Etsy or a dish rack (an old CrossBrowserTesting technique), all devices should have designated place to stand and charge.
Last but certainly not least, you need to be revisiting that user traffic data on a semi-regular basis. This is because as new devices, operating systems, and browsers are released or gain popularity, you may want to include them in your device lab. Alternatively, it would be surprising if your user traffic data stays completely persistent over a number of months, so you want to make sure you are staying on top of those statistics and updating your devices.
Manual and Automated Testing
As we’ve stated before, there’s a time and place for both manual and automated testing, and your device lab will have to work for both.
A device lab is an easier asset for anyone that primarily does manual testing. Having a selection of physical devices at-hand for UI testing will give you highly accurate insights into how users are interacting with your web application
However, if you want to try your hand at parallel testing to make things a bit speedier, a Selenium grid is the answer for running regression tests on multiple devices simultaneously.
If maintaining a lab becomes too much upkeep, it’s also worth considering a third-party cloud platform to complement or replace browser testing.
Have you ever built your own device lab? Share your challenges and successes in the comments!
