11 Min. Read
The Question of "Build vs Buy" Itself
The question of build it or buy it has been around as long as software has existed. Give up total control of the software project for a finished, out-of-the-box solution, or spend the man hours and learning experiences of building a solution specifically for your team’s needs. But with a testing environment, it’s not only the choice of building your own software but also buying hardware and physical devices that will work with that software. The ability to use the latest devices and browsers to test on is a critical requirement for development teams building consumer facing products. Is your solution going to adapt and scale fast enough to not affect your software’s quality.
The quick answer is no. Teams with even the steepest budgets often fail to take into account the necessary maintenance and programming that goes into a well built testing-grid or device lab. There are certain real costs of building an in-house environment, but there are countless other secondary costs of not going with a cloud provider, like CrossBrowserTesting.
Capital Costs of In-House Testing Environments
First Things First: Mobile Devices
Three things works against all testers and developers when it comes to mobile devices. The physical cost of devices, the breadth of devices, and the accuracy of those devices. Let’s talk about the accuracy level first. If these mobile devices are to mimic user's’ gestures, or come under the strain of network or battery, well then they must be real, physical devices. Not the simulators or emulators most manufactures make accessible, but the actual devices running actual code. These real devices, as you’ll see, have a considerable cost to them when compared to simulators.
The second factor working against all testers, the breadth of devices, is a problem that is continually expounding. Did you know the average year sees over 100 new devices introduced into just the Android ecosystem? Customers are now using mobile devices to consume content more than any other medium and the browsers and devices they are using are not the same. Apple has a massive, growing, & loyal fan base to the iOS platform and they are newly challenging developers and testers with new phone sizes and resolutions. Android, since it’s inception, has always had a fragmentation problem. A robust testing platform will have access to a large portion of these devices, including: Samsung, Nexus, LG, Pixel, Motorola and more. With new devices coming out every month, the simple headache of knowing which devices to buy and when can frustrate developers and testers trying to build quality applications for a diverse set of customers.
The last factor lies in the average cost of these physical, real devices. These phones can average anywhere from $300 to $500, depending on the make and model. While one of each device may seem like a perfectly plausible solution, the operating system of each device throws in a new variable. The Samsung Galaxy S6, one of the world’s most popular phones, runs several different Android operating systems ← You’ll need to test at least 3 to capture 60% of that market.
To have serviceable testing coverage on 10 of the most popular devices, with enough historical versioning for 2 OS’s, would cost about $10,000. And that is only for a snapshot view of the mobile consumer world. These devices will be obsolete in 1 or 2 years, and a new lab will need to be purchased.
A testing platform can take all the pain out of owning and maintaining your own device lab. A good testing platform, like CrossBrowsertesting, will have sufficient coverage on older, outdated devices but will also keep up with new devices as they grow in market share. And the one-device, one-OS problem simply vanishes: use the latest Nexus device with Android 6,7, or 8 in the cloud without worrying about installing or uninstalling images.
Your New Ops Team
Another aspect of building your own in-house device lab is your new ops team member(s) that will be joining your development or testing team. To build & maintain your own testing grid, you’ll need the experience of an IT/Ops individual. Let’s say we put the average cost of that employee at $80,000. As your product grows and your testing suite expands, you’ll need to hire more than one Ops member to scale your in-house grid.
With a testing platform like CrossBrowserTesting, you can scale your testing with releases or size without the hassle of hiring additional Ops employees. We act as a remote Ops team, making sure your device cloud works within your network, with the tools your team is using.
Maintaining VMs and Licenses
When choosing between an in-house device lab or a cloud platform, local VMs can often be overlooked. Because VM’s don’t have the enormous price tag of a real-device lab, they are often mislabeled as a cheap solution for testing different operating systems and browsers. But the truth is that once you factor in the hidden and soft costs of these VMs, you’ll be looking at a very unattractive price tag. The base price for a VMWare virtual machine is $2500 for the year. Depending on the amount of different configurations you’d like to test on, or the speed and scale you’d like your team to achieve, you can be looking at a mix of 5-20 VMs for a small, mid-sized business.
While the cost of virtualized computing can be readily calculated, upkeep and maintenance of your systems and software can be steep for a business of any size. Building your own testing lab requires you to have your VMs updated with the latest and greatest operating systems and browser versions, something that takes a surprising amount of time. Chrome alone updated 18 times in 2016, and Windows 10 has a bit of a mandatory updating problem.
ROI of Measured Associated Costs
Keeping Up With CI
Today we know that the digital environment is more complex than it was a few years back, and testers and developers are updating, changing and overhauling units of code on a daily or even hourly basis rather than a yearly one.
This pattern of Continuous Integration (CI) makes it vital for developers to test as much as they update to avoid bugs and breakpoints.
Furthermore, consumers are requiring more from their user experience. If a website isn’t optimized for their personal use on Firefox, for example, they’re not likely to download the newest version of Google Chrome before they instead abandon the website they’re on to seek out a competitor.
Browser testing proves necessary to ensure bottom line goals as well as a brand’s reputation. While as a developer you may be aware of the fastest and more visually appealing browsers and devices for the most optimized version of a web design, users may not be as familiar or not have the same access to those resources.
Letting testing fall to the wayside with sloppy methods and lacking strategy can be detrimental to business initiatives. While a device lab may seem like a realistic answer to testing in theory, there are infinite components that inevitably prevent the process from being profitable and effective, especially as technology continually advances and sites are constantly optimized.
Having an unreliable browser testing practice defeats the purpose of conducting one in the first place, and using a Selenium grid in a cloud platform is a much more efficient way to conduct testing.
By keeping all devices, browsers and browser versions updated, it ensures that the tests are completed and run smoothly and accurately. On the contrary, not installing regular updates means failed tests and more time wasted.
Additionally, when a device breaks, it’s going to take time and money to replace, delaying testing. While many Apple products often have a longer lifetime, you’ll be fixed to find a Windows PC that lasts for more than a few years, which means this will be a reoccurring problem in the testing process.
Keeping up with these processes is virtually impossible to do as an individual or a small team. With CrossBrowserTesting, you can be assured that there is a higher degree of reliability across a number of devices at one time so you’re not going to run into roadblocks with old hardware or software.
Time is money. While it’s difficult to evaluate how much time it will take to operate a device lab since it’s dependent on the size of the project, it’s evident that the associated costs that go into this process can defer from other priorities.
Figuring an Ops team may have enough physical devices to run a satisfactory number of tests, the amount of time it would take to manually test on all of them would exert an excessive amount of time and energy and still take hours longer than automated testing since tests cannot be run parallel unlike when it’s automated in the cloud, which also promotes Continuous Integration and Delivery as opposed to sequentially testing.
Not to mention that with agile CI, once the first round of manual testing is finished there will likely be a need for another round to include important new updates. Using a cloud platform allows QA to simultaneously perform exhaustive browser testing, while continuing to build out other elements and add features as needed.
Additionally, there has to be time allotted for human error when manual testing. Rather, automated browser testing like CrossBrowserTesting allows a faster process that can be accessed remotely with a more precise accuracy and shareable results.
Taking Testing Above and Beyond
Adding Value with Exclusive Features
While some issues around basic functionality can be simple to fix, more complex issues like layout differences can complicate the testing process. With CrossBrowserTesting, you can take a full-page screenshot across several of the devices at one time. You can even log-in to a secure or "subscribed" areas and have our screenshot system take the perfect snapshot then. You'll see quick bugs and visual errors quicker, across more devices.
After you've taken screenshots, you can compare browsers or historical versions with our Comparison Engine - highlighting differences in the DOM automatically. Small changes or modifications to production code can have unforeseen issues; regression testing can be a life-saving tool that builds a verification system off of something you know works.
CrossBrowserTesting's extensive Selenium and Appium support comes with added functinality like high-definition video recordings and error logs. By allowing developers to test imperative elements through a cloud with over 1,500 real desktop and mobile browser variations for functional and visual testing, CrossBrowserTesting is able to better mimic the user experience.
A Place for Manual Testing
While automated browser testing has become essential to the operation of coherent web development, that’s not to say that there’s not still a place for manual testing.
For ad-hoc and exploratory testing practices, handing devices over to the QA team will do the trick- but with CrossBrowserTesting you'll also get native developer tools on mobile devices. This allows for simple debugging on real devices, right in the cloud.
Additionally, CrossBrowserTesting helps you connect to test environments behind a firewall or across a proxy with full capabilities. Even if the site isn’t ready to go live, you’re afforded all the full-scale benefits of manual testing.
Although building your own in-house device lab and testing grid offers fexibility and access, cloud platform's offer a superior solution. With the reliability, scalability and additional features present in a platform like CrossBrowserTesting software teams have a ready place to turn to for the cloud.