automation Archives | CrossBrowserTesting.com

The Most Common Challenges Testers Face with Selenium

August 3, 2017 By Alex McPeak Leave a Comment

Selenium automates the browser to mimic real user actions on the web. As the web gets more complex, using Selenium to test has become increasingly popular. Although it has made web testing far easier for many teams and organizations across the world, it still has its fair share of challenges due it’s open source nature.

Most of the issues that testers experience have fairly straight forward solutions, which is why we outlined the most common Selenium challenges and how to fix them.

Pop-up Windows

When a simple, prompt, or confirmation alert pops-up, it can be difficult to automate it to either accept or close. Additionally, Windows-based alerts are beyond Selenium’s capabilities since they’re part of the operating system instead of the browser. However, since Selenium can operate multiple windows, web-based alerts can usually be handled with the switchTo method to control the pop-up while keeping the browser in the background.

Learn more about handling pop-up windows with Selenium here.

Dynamic Content

A web page that has dynamically loaded content has elements that may not be at first visible when you visit the site. This can mean that the page is user specific and will display different content based on different users, new content appears on the page after a certain amount of time, or it appears after clicking something on the page. For when a locator cannot identify a new element present at that time, Selenium comes with an integrated Explicit Wait where you can specify an amount of time to wait before automating a command. This should help give the page enough time to load and identify the element to interact with.

Learn more about testing dynamic content with Selenium here.

Flakiness

Sometimes Selenium will give you flaky tests, which means they’ll come back as positive or negative when they actually are the opposite. Eventually, too many flaky tests may mean testers might ignore the results, though this isn’t a great option either. Oftentimes, unstable builds can be a product of one of many factors: poor test isolation, flaky external services, and timeouts. By examining the elements of your Selenium tests, it’ll be easier to find out why your builds are unstable so you can approach the problem head-on.

Learn more about managing flaky tests here.

Can’t Test Mobile Apps

While Selenium can test on any operating system and browser on desktops, it’s limited when it comes to mobile testing in that it cannot run on native operating systems like iOS and Android. So, for example, the version of Facebook you pull up on Safari on your Mac can be tested with Selenium, but not the one on mobile Safari – on your iPhone 7. The Selenium family has a solution. It’s cousin, Appium, is an open source test automation framework that drives iOS and Android native, mobile, and hybrid apps using the WebDriver protocol and can be used for this purpose for testing mobile apps instead of web applications. As users increasingly move to mobile devices, it’s no surprise that developers and testers are taking advantage of the functionalities of Appium for this purpose.

Learn more about Appium here.

Limited Reporting and Documentation

While Selenium will exponentially increase your automated testing capabilities, because it’s an open source tool it is limited and features and does not support much reporting on its own. For this purpose, Selenium is best complimented with a third-party tool like CrossBrowserTesting that can capture screenshots and share reports through integrations like Slack and HipChat. However, you can also set up a framework to generate an output folder after a test with report information like errors, execution time, pass/fail count, etc. like TestNG.

Learn more about automated screenshot testing here and generating reports with Test NG here.

Multi-tab Testing

Selenium has the ability to test in multiple tabs, but it might present as an obstacle if you don’t know the correct commands. For example, if you want to open a new tab to perform a specific action without leaving the original tab, you’ll still want to be able to control both tabs and go back to the parent tab if users are likely to perform these kinds of actions. The best method for this is to store the previous window handle in a variable and then continue in the new window and store that in a second variable. Then, you can use the switchTo method to go between both windows as needed.

Learn more about how to work with multiple windows here.

No Manual Testing

Many testers will get excited about Selenium’s capabilities and expect that they can automate everything when the reality is that this just isn’t possible. In fact, a lot of automation relies on basic manual testing efforts. Rather, learning to prioritize which tests to automate will be beneficial to any tester. Additionally, as we mentioned, Selenium lacks capabilities for visual testing, which is important for testing beyond functionality to assess overall user experience. The most efficient way to do visual testing is by outsourcing to a third-party screenshot tool that integrates with Selenium.

Learn more about the need for manual testing here, prioritizing automated tests here, and alternative visual testing with Selenium here.

Scalability

Again, while Selenium allows you to test on pretty much any browser or operating system, it’s still limited in how many tests it can run at once and how fast it can run them based on how many hub/node configurations the tester has. And again, as your software scales, so must your testing. Without a Selenium grid, you can only test sequentially. However, with a Selenium grid and a device lab or a third party cloud tool like CrossBrowserTesting, you can test in parallel. This means multiples tests on different configurations can run at the same time, reducing the time it takes to run automated tests and increasing the configurations you can test on.

Learn more about parallel testing here.

Conclusion

One of the best things about being a software tester is that there’s always something new to learn. As the industry shifted from manual to automation, testers were tasked with learning new skills such as programming.

Now, as they become more familiar with automation techniques and tools such as Selenium, it’s admissible that they’ll encounter more challenges as they explore and familiarize themselves with new testing trends. Fortunately, as software continues to evolve, experts can depend on insights and support from the rest of the testing community to make sure they’re keeping up. Pick your preferred programming language to get started with Selenium testing here.

9 Telltale Signs That It’s Time to Move Your Software Testing To The Cloud

June 7, 2017 By Alex McPeak Leave a Comment

We’re all familiar with the cloud. As a topic that’s clearly trending and growing in software, our appreciation of the cloud stems from its innovation as well as its effectiveness.

The cloud is only projected to become more popular and more common in years to come. In fact, 95 percent of organizations are using a cloud in some way, and the market is supposed to increase 22 percent annually according to SD Times.

In software development, the cloud provides an easy-access, remote testing environment to produce real user scenarios. As organizations consider the benefits of the cloud, here are some signs that tell you it’s time to use it for browser testing.

You’re starting a business – One of the best times to check your web design across browsers are in the early stages a.k.a the exploratory tests. New companies should ensure they establish the UI of their website to look and function responsively across multiple devices when quality can make or break a business reputation.
You’re moving from manual to automation – The cloud offers an easy onboarding for organizations whose testers are a little newer to programming and automation. As you continue to recognize the benefits of automating your tests, the cloud is a practical way to make the process pain-free. With on-demand access, it’s easy for your team to test whenever they need and learn as they go. Additionally, screenshots, extensions, frameworks, debugging tools, and integrations are included to optimize usability and make testing more enjoyable. Plus, customizable preferences and customer-centric user interface assure an optimal user experience.
You’re realizing there aren’t enough hours a day to test everything – If you find yourself with test cases that are taking more time than there are hours in the day, it might be time to think about adding some more machines to your parallel testing. Parallel testing is a feature exclusive to the cloud, which will allow you to significantly cut down your testing time by testing different cases on multiple machines at the same time, instead of one after the other. Additionally, the cloud is available all day every day, so you can test whenever you want without having to be on-site. In fact, you can even run automated tests in your sleep. This will make it easier to test all the configurations you need to, while still meeting deadlines.
You have a hard time keeping up with all the tech – Software testing is not what it used to be. With so many new browsers, OS versions, and devices coming out all the time, it can be difficult to keep track of all the configurations you need to test on. In the cloud, new releases are added for you so you don’t even have to think about it.
You think new smartphones are way overpriced – If your device lab budget isn’t quite allowing all the configurations you need, a cloud might be just the right answer. When an average Android is about $670 and an iPhone rounds out at around $700, the numbers can add up fast when you try to test on every device your user has — not to mention maintenance.
You recognize the benefits of simulators, emulators, and real devices – While emulators and simulators have their place in testing, real devices are crucial in order to test on the machines that your users are on. To get the most comprehensive breadth of simulators, emulators, and real devices for testing, a cloud is the best way to access all three for different purposes, and the combination ensures that you always have access to testing.
You’re scaling your team – You had QA all figured out when you had just one (albeit really talented) tester. But, as your company scales, so must your testing team. When everyone’s trying to complete different tasks that all seem to be of top priority at the same time, the cloud gives you more room to accommodate growing pains. Since everyone has access to the same devices at once, you can ensure you’re supplying your growing team with the resources they require.
You have testers in different offices – Similarly, it’s a little bit difficult to share the capabilities of your device lab among a QA team that’s spread out among different cities, countries, and continents. Testing in a cloud provides one platform that further encourages communication through different departments and remote teams with integrations like Slack sharing.
Your customers are getting demanding – Has your customer support team been extra stressed out lately? Are you getting user reports saying that your website is slow, or there’s a bug, or it’s not working on their browser? Your customers have high expectations of you. They might be using a wider variety of devices than you think, and it’s compromising their interaction with your web application.

What pushed you to make the decision to adopt a cloud testing service? Tell us in the comments!

What is Browser Testing?

March 8, 2017 By Alex McPeak Leave a Comment

Do you have a cross-browser testing strategy?

Browser testing is a method of quality assurance for web applications across multiple browsers. That is to say, it ensures the quality of your website on different screens. It’s implemented to ensure a website’s functionality and design and includes testing a range of devices and operating systems being used in the market and customer base. On top of that, screen size, screen resolution, OS version, and browser versions all change and contribute to how someone is viewing content, making the practice of browser testing increasingly necessary for diversified user experiences.

While fundamentally self-explanatory, browser testing has an expansive amount of components. Understanding the many factors can exponentially impact your web application and increase customer satisfaction while ignoring its importance can have a negative effect on reputation and bottom line goals.

Why it’s Needed

There used to be only a handful of browsers existing across the internet, and they were mainly situated on desktops. Additionally, with a waterfall model design process where developers were primarily making quarterly or monthly changes to code, manually testing each one was relatively simple.

Times have changed, however, and today’s consumers are on a multitude of devices ranging from desktop computers and laptops to mobile phones and tablets, while there are too many browsers and browser versions to count at this point. Furthermore, to parallel the rise of mobile and desktop internet consumption, design and development have moved to an agile methodology. Developers are no longer making yearly updates but instead instituting continuous integration with updates by the hour or day.

Meanwhile, each browser responds to these code changes differently, and so with every change of code comes a risk that any of those browser/device/OS combinations may not respond according to expectations. Layout, accessibility, connectivity, quality and performance can vary slightly or exponentially from browser to browser, as each renders CSS, HTML, and JavaScript differently.

Developers inherently know and learn the lowest risk browsers and devices, and in turn, use them every day to view the product of their coding. However, a misconception is that another user will be visiting the same website on the same browsers and devices. “It works on my machine” is not a valid argument.

While about half of users will be on the popular, low-risk browsers that developers use such as Chrome and Firefox, the bugs will mainly be saturated in high-risk browsers like Internet Explorer. You could direct a user who’s experiencing issues on a bugged browser to either update their browser version or download different browser altogether, but it’s more likely that user will be unwilling or unable to switch browsers just to properly view your website. Instead, they’ll just leave your page.

Though it may be unrealistic to test 100 percent of all browser platforms, it’s important to create an acceptable user experience amongst the majority of your user base. The more browsers that are tested and optimized, the more inclusive your site will be for a variety of visitors. This is not to say that each should look exactly the same, but instead that there is an element of responsive design that has been implemented and tested.

Bottom Line Goals

As a developer, your job is to make your website accessible and appealing to the broadest range of users.

When someone on Internet Explorer is able to access and enjoy a website the same way somebody on Google Chrome is, it relays a dedication to the user experience and a thorough design process. This means that consumers are less likely to encounter bugs and off-putting layout elements, and most likely to foster a sense of brand loyalty.

Not only does browser testing encourage a positive reputation, it also impacts bottom line goals when users can easily view content and experience optimized functionality no matter what device, operating system, or browser they access it from.

How it’s Done

Pick Your Devices

The first thing that needs to be done is to decide how many browsers you want to test on and which ones. A popular method to determine where the majority of your audience is saturated is to measure traffic with Google Analytics, although keep in mind this only provides a screenshot of your visitors as well as potential visitors.

Decide on Execution

After you choose which devices and browsers you’re going to test, you have two main options for execution — building a device lab or testing in the cloud with a tool like CrossBrowserTesting. With both options, you have to decide whether to test on an emulator or simulator or on physical devices.

First and foremost, we recommend real devices because they replicate the actions of the user, while the results received on emulators tend to vary from a true user’s experience. However, it’s also important to include emulators and simulators at different stages in testing, as well.

The cost of a device lab, of course, will vary with how many machines you want to buy and test. We estimate that the minimum cost of purchasing to include all the essential devices and operating systems is around $10,000, but this will only reach about half your user market. This number also doesn’t consider the cost of replacing the devices every few years.

In this case, VMs may seem like an attractive solution to save money at a base of about $2,500 a year if you’re willing to give up some accuracy. However, the hidden costs of licenses, maintenance, and updates can result in unforeseen expenses that will continue to add up over time and put the two at comparable costs.

The other option is to go with a testing platform where the added value is in cost, speed, features, and spectrum of devices. Since cross-browser testing platforms charge a monthly fee to test on hundreds or thousands of devices, you’re likely to have a greater range of configurations to test, while saving quite a chunk of change.

Additionally, parallel testing is exclusive to automated cross-browser testing tools, allowing you to simultaneously run a multitude of tests in a fraction of the time, often cutting hours and even days off of a boring and laborious manual testing regime.

Included features such as screenshot comparisons, live testing, and local testing further simplifies the process for testers and developers so they have the time to focus on designing a better product instead of individually testing numerous browsers and devices every time a code is updated.

Begin Testing – Manual and Automation

Where you’re at in your development and design process always has a say in the way you’re going to execute your browser testing.

For early stage development, one-off or exploratory testing is usually appropriate and can be done through manual testing. Once you’re further along in the process, regression testing and functional test automation is going to provide the breadth and expansion required to understand numerous coding elements and detect bugs.

No matter how you’re testing, you should be considering how your site both looks and performs across numerous browsers. Whether you manually test or automate the process, CrossBrowserTesting has a multitude of options to help save time, cut costs and improve the overall quality of your web application.

Intro to Selenium Testing

March 2, 2017 By Alex McPeak Leave a Comment

Selenium is an open source testing tool that allows users to test web applications across different browsers and platforms. Selenium includes a suite of software that developers can use to automate web applications including IDE, RC, WebDriver and Selenium grid, which all serve different purposes.

In this post, we’ll explore the multiple components of the Selenium framework and its evolution as well as how it’s used and why developers and testers depend on it for browser testing.

What’s the Use?

The uses of Selenium increase as you advance in the suite, and each component has advantages and disadvantages. The basic fundamentals of Selenium though revolve around browser testing, as developers and testers alike love the open source framework’s ability to mimic a real user’s actions in the browser. Developers create test scripts that run locally or in the cloud to assess basic functional correctness and verify visual layouts. These test scripts can range from simple JavaScript unit tests around an action like a button click, or fully automated end-to-end functional tests that resemble the full actions of an everyday user, like the checkout or login process.

What makes Selenium so easy to learn is its ability to bind to several popular languages, letting you write in your preferred programming language including Java, Javascript, PHP, Perl, C#, Ruby, .Net, and Python.

A Brief History of Selenium

Selenium has a dynamic history since it came to being in 2004 when Jason Huggins, who was a ThoughtWorks programmer at the time, decided to make a tool for testing web applications. By the end of the year, it had become a wildly popular open source tool.

The first version of Selenium had many limiting factors like its Javascript based automation engine, causing Huggins to seek other contributors. Simon Stewart, another ThoughtWorks employee, stepped up to the plate and they produced Selenium Webdriver, or Selenium 2.0

The combination of Selenium and WebDriver in 2006 became a master web application tool know as Selenium 2.0 and is one of the leading free testing tools for programmers around the world.

While Selenium 3.0 has been released, it’s basically a new and improved version of Selenium 2.0 with the same fundamental purpose and structure.

Breaking Down the Selenium Suite

The Beginning – Selenium IDE

Selenium IDE (Integrated Development Environment) is the Firefox plug-in tool that allows you to record, playback, edit and debug test scripts to export them in a preferred programming language. By providing a test domain specific language, Selenese, IDE lets you test without learning a new scripting language and makes it easier to send commands to the browser and automate testing. However, IDE is limiting in the way that you can only use it to test web applications in Firefox, and therefore it cannot be used to test other browsers.

Selenium IDE is mostly used now for bug reproduction scripts, exploratory testing, learning about Selenese commands and exporting simple test cases to WebDriver. Selenium Builder is also a Firefox extension that can import HTML files recorded in Selenium IDE to run tests locally or on a Selenium server and further assist automation.

The Evolution – Selenium 1 and 2

Selenium RC (Remote Control), or Selenium 1, was a method used to run the test script in different browsers by launching them as a proxy server, but has since been abandoned since the creation of WebDriver, or Selenium 2.0. Selenium 2 is what you would use in testing today, as RC is no longer supported.

This is because WebDriver doesn’t rely on JavaScript for automation like RC does, and instead, directly communicates with the browser from the operating system making it faster and more accurate. WebDriver also supports more browsers and operating environments than RC including the HtmlUnit browser, and has a more concise API.

Selenium 2 is popular among developers and QA teams for regressions tests and cross browser testing, since it allows you to test on a spectrum of browsers and operating systems using the native operating system functionality. This means that commands perform as a human would if they were using the same browser.

Optimization – Selenium Grid

Selenium Grid allows you to multiple run tests at the same time on different machines and browsers in order to speed up testing. For CI motions, testing demands to be done at a rapid pace, and running tests sequentially can be painfully slow. Running in parallel on a grid can improve the speed of your testing greatly. Selenium Grid is comprised of one machine called a hub and multiple nodes for execution. The more nodes you have, the more tests you can run simultaneously.

While you can make your own Selenium grid with a costly device lab or VMs, many have found moving Selenium grid to a third-party cloud is faster and more productive, allowing more tests to run on more OS systems and browsers simultaneously.