automation Archives | CrossBrowserTesting.com

Why You Need to Stop Relying on Developers to Perform All Testing

February 7, 2018 By Alex McPeak Leave a Comment

As companies decipher why testing matters based on the cost of a bug or a customer pointing out their mistakes, teams that produce and deploy software applications ultimately come to terms with the fact the testing should never be skipped.

However, some people think they can get the job done without paying the price — the price of a software tester, that is. While testing might seem simple at first glance, there are countless factors that go into effectively evaluating an application.

Organizations that rely on developers to do the job will quickly run into roadblocks that prevent on-time delivery and impact customer satisfaction. Here are a few reasons why your software team needs testers:

Testing is a specialized skill set – Developing and testing are two different jobs with two different skill sets. Just like a doctor shouldn’t try to do a nurses job and vice versa, building an application and having an understanding of the techniques to properly test and provide feedback for that application are two different things. While it a developer could maybe perform a high-level test of the application, a tester is truly needed to take a deeper dive and handle more complicated procedures that go beyond a basic click-through.
Testing takes time – Testing is not a quick one-and-done. In fact, most testers find that they don’t have enough hours in the week to get to everything they wanted to test. A big reason behind this is that it’s virtually impossible to test everything — there are limitless possibilities and there will always be some part of the application that’s going to go under the radar, no matter how much time and effort is spent on it. This means that if your developers are doing the testing, they will either be skipping a lot of tests that they need to run due to time split between their responsibilities, or they’ll be doing an acceptable amount of testing that will keep them from doing their main job. Let your developers develop and your testers test so you can stay on track for frequent integrations, fast delivery, and quality software.
Knowing how to code ≠ knowing how to automate – Learning a programming language is the first step to getting started with automated testing, so it might seem natural to have developers learn to automate and perform checks because they already know how to code. However, you’ll find instead that the execution doesn’t go as smoothly as expected. There’s a lot to learn when it comes to automation, and coding is just a very small part of that. You need to know when to automate, how to write comprehensive scripts, how to approach challenges with automation tools, how to manage test cases, how to maintain tests, and much more. This comes back to the idea that testing takes time and a special skill set — it’s more likely you’ll find your developers are frustrated and falling behind than making any strides in automation.
Reporting and documentation are important – A test that’s run but isn’t reported or documented is only so helpful in the long run. Even if your developer is willing to run through their own part of the application, it doesn’t mean they’re making scripts that are maintainable and reusable. And it definitely doesn’t mean they’re going through different parts of the application from other developers to conjure effective integration and end-to-end testing strategy. If all your developers are testing their own code, all you’re going to end up with is fragmented information that probably won’t do much good catching bugs anyway.
You need to be in the right mindset – If you’re testing your own code, you’re not really accounting for new scenarios. That’s to say that if someone writes a feature, they’re only going to test for the use cases they already considered in development. Having a tester come in brings a fresh set of eyes to look at the feature in new ways that the developer may not have thought of. It’s like having someone edit a paper for you — oftentimes they’ll find mistakes or make suggestions that you missed no matter how many times you read it yourself. A tester brings this same value to the table by not only finding errors that were missed but also providing insight.

Having your developers test their own code is a poor use of time, money, and skills. While you may be tempted to resort to using your developers to quickly test a new integration before release, it’s important to understand that there’s a good chance you won’t get the results you want.

Hiring for testings and bringing in a team that is specifically trained to inspect software quality and focused on finding and reporting bugs will not only improve time to market but also increase the capacity of your developers and improve the software you release. Don’t skimp on testers the same way you know not to skip testing.

The Most Common Selenium Challenges

August 3, 2017 By Alex McPeak Leave a Comment

Selenium WebDriver (Selenium) automates the browser to mimic real user actions on the web. As the web gets more complex, using Selenium to test has become increasingly popular. Although it has made web testing far easier for many teams and organizations across the world, it still has its fair share of challenges due it’s open source nature.

Most of the issues that testers experience have fairly straightforward solutions, which is why we outlined the most common Selenium challenges and how to fix them.

Pop-up Windows

When a simple, prompt, or confirmation alert pops-up, it can be difficult to automate it to either accept or close. Additionally, Windows-based alerts are beyond Selenium’s capabilities since they’re part of the operating system instead of the browser. However, since Selenium WebDriver can operate multiple windows, web-based alerts can usually be handled with the switchTo method to control the pop-up while keeping the browser in the background.

Learn more about handling pop-up windows with Selenium.

Dynamic Content

A web page that has dynamically loaded content has elements that may not be at first visible when you visit the site. This can mean that the page is user specific and will display different content based on different users, new content appears on the page after a certain amount of time, or it appears after clicking something on the page. For when a locator cannot identify a new element present at that time, Selenium WebDriver comes with an integrated Explicit Wait where you can specify an amount of time to wait before automating a command. This should help give the page enough time to load and identify the element to interact with.

Learn more about testing dynamic content with Selenium.

Flakiness

Sometimes Selenium will give you flaky tests, which means they’ll come back as positive or negative when they actually are the opposite. Eventually, too many flaky tests may mean testers might ignore the results, though this isn’t a great option either. Oftentimes, unstable builds can be a product of one of many factors: poor test isolation, flaky external services, and timeouts. By examining the elements of your Selenium tests, it’ll be easier to find out why your builds are unstable so you can approach the problem head-on.

Learn more about managing flaky Selenium tests.

Mobile Testing

While Selenium WebDriver can test on any operating system and browser on desktops, it’s limited when it comes to mobile testing in that it cannot run on native operating systems like iOS and Android. So, for example, the version of Facebook you pull up on Safari on your Mac can be tested with Selenium, but not the one on mobile Safari – on your iPhone 7. The Selenium family has a solution. It’s cousin, Appium, is an open source test automation framework that drives iOS and Android native, mobile, and hybrid apps using the WebDriver protocol and can be used for this purpose for testing mobile apps instead of web applications. As users increasingly move to mobile devices, it’s no surprise that developers and testers are taking advantage of the functionalities of Appium for this purpose.

Learn more about automating with Appium.

Limited Reporting

While Selenium will exponentially increase your automated testing capabilities, because it’s an open source tool it is limited and features and does not support much reporting on its own. For this purpose, Selenium is best complimented with a third-party tool like CrossBrowserTesting that can capture browser screenshots and share reports through integrations like Slack and HipChat. However, you can also set up a framework to generate an output folder after a test with report information like errors, execution time, pass/fail count, etc. like TestNG.

Learn more about generating reports.

Multi-tab Testing

Selenium WebDriver has the ability to test in multiple tabs, but it might present as an obstacle if you don’t know the correct commands. For example, if you want to open a new tab to perform a specific action without leaving the original tab, you’ll still want to be able to control both tabs and go back to the parent tab if users are likely to perform these kinds of actions. The best method for this is to store the previous window handle in a variable and then continue in the new window and store that in a second variable. Then, you can use the switchTo method to go between both windows as needed.

Learn more about how to test multiple tabs.

Manual Testing

Many testers will get excited about Selenium’s capabilities and expect that they can automate everything when the reality is that this just isn’t possible. In fact, a lot of automation relies on basic manual testing efforts. Rather, learning to prioritize which tests to automate will be beneficial to any tester. Additionally, as we mentioned, Selenium WebDriver lacks capabilities for visual testing, which is important for testing beyond functionality to assess overall user experience. The most efficient way to do visual testing is by outsourcing to a third-party screenshot tool that integrates with Selenium.

Learn more about the need for manual testing, prioritizing automated tests, and how to take screenshots during a Selenium test.

Scalability

Again, while Selenium WebDriver allows you to test on pretty much any browser or operating system, it’s still limited in how many tests it can run at once and how fast it can run them based on how many hub/node configurations the tester has. And again, as your software scales, so must your testing. Without a Selenium grid, you can only test sequentially. However, with a Selenium grid and a device lab or a third party cloud tool like CrossBrowserTesting, you can test in parallel. This means multiples tests on different configurations can run at the same time, reducing the time it takes to run automated tests and increasing the configurations you can test on.

Learn more about parallel testing.

Conclusion

One of the best things about being a software tester is that there’s always something new to learn. As the industry shifted from manual to automation, testers were tasked with learning new skills such as programming.

Now, as they become more familiar with automation techniques and tools such as Selenium WebDriver, it’s admissible that they’ll encounter more challenges as they explore and familiarize themselves with new testing trends. Fortunately, as software continues to evolve, experts can depend on insights and support from the rest of the testing community to make sure they’re keeping up. Pick your preferred programming language to get started with Selenium testing here.

9 Signs You Should be QA Testing in the Cloud

June 7, 2017 By Alex McPeak Leave a Comment

We’re all familiar with the cloud. As a topic that’s clearly trending and growing in software, our appreciation of the cloud testing stems from its innovation as well as its effectiveness.

The cloud is only projected to become more popular and more common in years to come. In fact, 95 percent of organizations are using a cloud in some way, and the market is supposed to increase 22 percent annually according to SD Times.

In software development, cloud testing provides an easy-access, remote testing environment to produce real user scenarios. As organizations consider the benefits of the cloud testing, here are some signs that tell you it’s time to use it for browser testing.

You’re starting a business – One of the best times to check your web design across browsers are in the early stages a.k.a the exploratory tests. New companies should ensure they establish the UI of their website to look and function responsively across multiple devices when quality can make or break a business reputation.
You’re moving from manual to automation – Cloud testing offers an easy onboarding for organizations whose testers are a little newer to programming and automation. As you continue to recognize the benefits of automating your tests, the cloud is a practical way to make the process pain-free. With on-demand access, it’s easy for your team to test whenever they need and learn as they go. Additionally, browser screenshots, extensions, frameworks, debugging tools, and integrations are included to optimize usability and make testing more enjoyable. Plus, customizable preferences and customer-centric user interface assure an optimal user experience.
You’re realizing there aren’t enough hours a day to test everything – If you find yourself with test cases that are taking more time than there are hours in the day, it might be time to think about adding some more machines to your parallel testing. Parallel testing is a feature exclusive to cloud testing, which will allow you to significantly cut down your testing time by testing different cases on multiple machines at the same time, instead of one after the other. Additionally, the cloud is available all day every day, so you can test whenever you want without having to be on-site. In fact, you can even run automated tests in your sleep. This will make it easier to test all the configurations you need to, while still meeting deadlines.
You have a hard time keeping up with all the tech – Software testing is not what it used to be. With so many new browsers, OS versions, and devices coming out all the time, it can be difficult to keep track of all the configurations you need to test on. With cloud testing, new releases are added for you so you don’t even have to think about it.
You think new smartphones are way overpriced – If your device lab budget isn’t quite allowing all the configurations you need, cloud testing might be just the right answer. When an average Android is about $670 and an iPhone rounds out at around $700, the numbers can add up fast when you try to test on every device your user has — not to mention maintenance.
You recognize the benefits of simulators, emulators, and real devices – While emulators and simulators have their place in testing, real devices are crucial in order to test on the machines that your users are on. To get the most comprehensive breadth of simulators and emulators plus real devices for testing, cloud testing is the best way to access all three for different purposes, and the combination ensures that you always have access to testing.
You’re scaling your team – You had QA all figured out when you had just one (albeit really talented) tester. But, as your company scales, so must your testing team. When everyone’s trying to complete different tasks that all seem to be of top priority at the same time, the cloud gives you more room to accommodate growing pains. Since everyone has access to the same devices at once, you can ensure you’re supplying your growing team with the resources they require.
You have testers in different offices – Similarly, it’s a little bit difficult to share the capabilities of your device lab among a QA team that’s spread out among different cities, countries, and continents. Cloud testing provides one platform that further encourages communication between different departments and remote teams with integrations like Slack sharing.
Your customers are getting demanding – Has your customer support team been extra stressed out lately? Are you getting user reports saying that your website is slow, or there’s a bug, or it’s not working on their browser? Your customers have high expectations of you. They might be using a wider variety of devices than you think, and it’s compromising their interaction with your web application.

What pushed you to make the decision to adopt a cloud testing service? Tell us in the comments!

What is Cross Browser Testing?

March 8, 2017 By Alex McPeak Leave a Comment

Browser testing is a method of quality assurance for web applications across multiple browsers. That is to say, it ensures the quality of your website on different screens. It’s implemented to ensure a website’s functionality and design and includes testing a range of devices and operating systems being used in the market and customer base. On top of that, screen size, screen resolution, OS version, and browser versions all change and contribute to how someone is viewing content, making the practice of cross-browser testing increasingly necessary for understanding diversified user experiences.

While fundamentally self-explanatory, browser testing has an expansive amount of components. Understanding the many factors can exponentially impact your web application and increase customer satisfaction while ignoring its importance can have a negative effect on reputation and bottom line goals.

Why it’s Needed

There used to be only a handful of browsers existing across the internet, and they were mainly situated on desktops. Additionally, with a waterfall model design process where developers were primarily making quarterly or monthly changes to code, manually testing each one was relatively simple.

Times have changed, however, and today’s consumers are on a multitude of devices ranging from desktop computers and laptops to mobile phones and tablets, while there are too many browsers and browser versions to count at this point. Furthermore, to parallel the rise of mobile and desktop internet consumption, design and development have moved to an agile methodology. Developers are no longer making yearly updates but instead instituting continuous integration with updates by the hour or day.

Meanwhile, each browser responds to these code changes differently, and so with every change of code comes a risk that any of those browser/device/OS combinations may not respond according to expectations. Layout, accessibility, connectivity, quality, and performance can vary slightly or exponentially from browser to browser, as each renders CSS, HTML, and JavaScript differently.

Developers inherently know and learn the lowest risk browsers and devices, and in turn, use them every day to view the product of their coding. However, a misconception is that another user will be visiting the same website on the same browsers and devices. “It works on my machine” is not a valid argument.

While about half of users will be on the popular, low-risk browsers that developers use such as Chrome and Firefox, the bugs will mainly be saturated in high-risk browsers like Internet Explorer. You could direct a user who’s experiencing issues on a bugged browser to either update their browser version or download different browser altogether, but it’s more likely that user will be unwilling or unable to switch browsers just to properly view your website. Instead, they’ll just leave your page.

Though it may be unrealistic to test 100 percent of all browser platforms, it’s important to create an acceptable user experience amongst the majority of your user base. The more browsers that are tested and optimized, the more inclusive your site will be for a variety of visitors. This is not to say that each should look exactly the same, but instead that there is an element of responsive design that has been implemented and tested.

Bottom Line Goals

As a developer, your job is to make your website accessible and appealing to the broadest range of users.

When someone on Internet Explorer is able to access and enjoy a website the same way somebody on Google Chrome is, it relays a dedication to the user experience and a thorough design process. This means that consumers are less likely to encounter bugs and off-putting layout elements, and most likely to foster a sense of brand loyalty.

Not only does browser testing encourage a positive reputation, it also impacts bottom line goals when users can easily view content and experience optimized functionality no matter what device, operating system, or browser they access it from.

How it’s Done

Pick Your Devices

The first thing that needs to be done is to decide how many browsers you want to test on and which ones. A popular method to determine where the majority of your audience is saturated is to measure traffic with Google Analytics, although keep in mind this only provides a sample of your visitors as well as potential visitors.

Decide on Execution

After you choose which devices and browsers you’re going to test, you have two main options for execution — building a device lab or testing in the cloud with a tool like CrossBrowserTesting. With both options, you have to decide whether to test on an emulator or simulator or on physical devices.

First and foremost, we recommend real devices because they replicate the actions of the user, while the results received on emulators tend to vary from a true user’s experience. However, it’s also important to include emulators and simulators at different stages of testing, as well.

The cost of a device lab, of course, will vary with how many machines you want to buy and test. We estimate that the minimum cost of purchasing to include all the essential devices and operating systems is around $10,000, but this will only reach about half your user market. This number also doesn’t consider the cost of replacing the devices every few years.

In this case, VMs may seem like an attractive solution to save money at a base of about $2,500 a year if you’re willing to give up some accuracy. However, the hidden costs of licenses, maintenance, and updates can result in unforeseen expenses that will continue to add up over time and put the two at comparable costs.

The other option is to go with a testing platform where the added value is in cost, speed, features, and spectrum of devices. Since cross-browser testing platforms charge a monthly fee to test on hundreds or thousands of devices, you’re likely to have a greater range of configurations to test, while saving quite a chunk of change.

Additionally, parallel testing is exclusive to automated cross-browser testing tools, allowing you to simultaneously run a multitude of tests in a fraction of the time, often cutting hours and even days off of a boring and laborious manual testing regime.

Included features such as browser screenshot comparisons, live testing, and local testing further simplifies the process for testers and developers so they have the time to focus on designing a better product instead of individually testing numerous browsers and devices every time a code is updated.

Begin Testing – Manual and Automation

Where you’re at in your development and design process always has a say in the way you’re going to execute your browser testing.

For early-stage development, one-off or exploratory testing is usually appropriate and can be done through manual testing. Once you’re further along in the process, regression testing and functional test automation is going to provide the breadth and expansion required to understand numerous coding elements and detect bugs.

No matter how you’re testing, you should be considering how your site both looks and performs across numerous browsers. Whether you manually test or automate the process, CrossBrowserTesting has a multitude of options to help save time, cut costs and improve the overall quality of your web application.

An Introduction to Selenium Open Source Automation Tool

March 2, 2017 By Alex McPeak 1 Comment

Selenium is an open source testing tool that allows users to test web applications across different browsers and platforms. Selenium includes a suite of software that developers can use to automate web applications including IDE, RC, WebDriver and Selenium grid, which all serve different purposes.

In this post, we’ll explore the multiple components of the Selenium framework and its evolution as well as how it’s used and why developers and testers depend on it for browser testing.

What’s the Use?

The uses of Selenium increase as you advance in the suite, and each component has advantages and disadvantages. The basic fundamentals of Selenium though revolve around browser testing, as developers and testers alike love the open source framework’s ability to mimic a real user’s actions in the browser. Developers create test scripts that run locally or in the cloud to assess basic functional correctness and verify visual layouts. These test scripts can range from simple JavaScript unit tests around an action like a button click, or fully automated end-to-end functional tests that resemble the full actions of an everyday user, like the checkout or login process.

What makes Selenium so easy to learn is its ability to bind to several popular languages, letting you write in your preferred programming language including Java, Javascript, PHP, Perl, C#, Ruby, .Net, and Python.

A Brief History of Selenium

Selenium has a dynamic history since it came to being in 2004 when Jason Huggins, who was a ThoughtWorks programmer at the time, decided to make a tool for testing web applications. By the end of the year, it had become a wildly popular open source tool.

The first version of Selenium had many limiting factors like its Javascript based automation engine, causing Huggins to seek other contributors. Simon Stewart, another ThoughtWorks employee, stepped up to the plate and they produced Selenium Webdriver, or Selenium 2.0

The combination of Selenium and WebDriver in 2006 became a master web application tool know as Selenium 2.0 and is one of the leading free testing tools for programmers around the world.

While Selenium 3.0 has been released, it’s basically a new and improved version of Selenium 2.0 with the same fundamental purpose and structure.

Breaking Down the Selenium Suite

The Beginning – Selenium IDE

Selenium IDE (Integrated Development Environment) is the Firefox plug-in tool that allows you to record, playback, edit and debug test scripts to export them in a preferred programming language. By providing a test domain specific language, Selenese, IDE lets you test without learning a new scripting language and makes it easier to send commands to the browser and automate testing. However, IDE is limiting in the way that you can only use it to test web applications in Firefox, and therefore it cannot be used to test other browsers.

Selenium IDE is mostly used now for bug reproduction scripts, exploratory testing, learning about Selenese commands and exporting simple test cases to WebDriver. Selenium Builder is also a Firefox extension that can import HTML files recorded in Selenium IDE to run tests locally or on a Selenium server and further assist automation.

The Evolution – Selenium 1 and 2

Selenium RC (Remote Control), or Selenium 1, was a method used to run the test script in different browsers by launching them as a proxy server, but has since been abandoned since the creation of WebDriver, or Selenium 2.0. Selenium 2 is what you would use in testing today, as RC is no longer supported.

This is because WebDriver doesn’t rely on JavaScript for automation like RC does, and instead, directly communicates with the browser from the operating system making it faster and more accurate. WebDriver also supports more browsers and operating environments than RC including the HtmlUnit browser, and has a more concise API.

Selenium 2 is popular among developers and QA teams for regressions tests and cross browser testing, since it allows you to test on a spectrum of browsers and operating systems using the native operating system functionality. This means that commands perform as a human would if they were using the same browser.

Optimization – Selenium Grid

Selenium Grid allows you to multiple run tests at the same time on different machines and browsers in order to speed up testing. For CI motions, testing demands to be done at a rapid pace, and running tests sequentially can be painfully slow. Running in parallel on a grid can improve the speed of your testing greatly. Selenium Grid is comprised of one machine called a hub and multiple nodes for execution. The more nodes you have, the more tests you can run simultaneously.

While you can make your own Selenium grid with a costly device lab or VMs, many have found moving Selenium grid to a third-party cloud is faster and more productive, allowing more tests to run on more OS systems and browsers simultaneously.