test automation Archives | CrossBrowserTesting.com

The Anatomy of a Comprehensive Selenium Testing Strategy

August 28, 2017 By Bas Dijkstra 2 Comments

Now that test automation has established itself as a key enabler of shorter software development and delivery cycles, being fluent in Selenium WebDriver (Selenium) will give test automation engineers an edge in the current IT job market. However, there’s a lot more you need to know to create a successful Selenium testing implementation than just the Selenium API itself. In this article, we’ll take a closer look at what it takes to create, use, and maintain a Selenium-based test automation solution that’s able to live up to expectations.

Is Selenium testing right for you?

Before you implement Selenium, it’s wise to ask yourself whether it’ll be a good fit for your test automation efforts in the first place. While it’s true that Selenium implementations are available for a multitude of programming languages and there’s a wealth of information around Selenium testing available online, creating a solid solution around Selenium and implementing robust, reliable, and maintainable test scripts is not an easy task. It requires both solid development and testing skills to write, run, and maintain a Selenium testing suite that lives up to its promise of faster feedback and shortened delivery cycles, instead of being a time and effort drain that slows down your software development efforts.

The remainder of this article illustrates the most important skills needed and the factors that should be taken into consideration to make your Selenium testing strategy successful once you’ve decided it’s the right direction.

Choose your tests wisely

Once you’ve got the hang of creating automated Selenium tests, it might be tempting to replace all of your current tests with Selenium-based counterparts. However, before you start automating away, it might be a good idea to consider the following:

Despite being one of the most commonly used user interface-driven automated test tools, creating high quality, reliable Selenium testing is hard. Maintaining a large set of these tests, especially in fast moving environments, is even harder.
Selenium-based tests offer you the broadest scope since they exercise an application all the way from the user interface through API and business logic layers all the way down to the database (and back again). This has a downside, too. Whenever a defect occurs at a lower level in the application, there’s a significant chance that your Selenium test isn’t able to pinpoint the root cause of the failure, only its effect on the end user experience. This is also known as shallow feedback, and it’s something you want to avoid when you’re trying to pinpoint and fix defects fast.
Selenium testing is much slower in terms of execution compared to unit or API-level tests. When you’re striving for fast feedback, you want your automated test suite to give you this feedback in the shortest amount of time possible.

What the above boils down to is simple: never use Selenium (or any other user interface-driven tool, for that matter) to automate a test that can be done at a lower level such as the API layer. As a general rule, Selenium and similar tools should only be used to verify that an end user of the application can successfully complete a sequence of actions in the application.

Apply good programming practices

Despite the ongoing advertisement of ‘codeless’ test automation tools that ‘allow everyone to create automated tests’, test automation still is a form of software development and should be treated as such. This applies to Selenium testing as well. Since you’re writing your tests in code, it makes sense to apply good programming practices to this code as well:

Make your tests maintainable by leveraging patterns that promote modularity and reuse. A single change in your application under test such as the introduction or removal of a text field should require only a single change in your test code. Probably the best known and most widely used pattern in Selenium is the Page Object pattern.
Keep your tests readable by using descriptive method and variable names. The intent of a test method named withdrawMoney_ResultsInError_IfBalanceIsInsufficient() is clear, while a test method named testCase147() or testWithdraw() is far less so. You can find more suggestions for useful test method names here. Be sure to apply readable standards to variables as well. You’ll see that a well-structured test with a descriptive name is pretty much self-documenting.
Optimize the stability of your tests to prevent false positives. Failing tests can be quite costly in terms of time needed to analyze the root cause, and time spent analyzing false positives is time wasted. Unfortunately, Selenium tests are among the tests most prone to false positives. One tried and tested way to create more stable tests is by creating wrapper methods around the Selenium API to create a single place where synchronization, exception handling, and additional logging and reporting can be implemented. If you want to be able to rely on the results of Selenium testing — something that’s becoming more important now that Continuous Integration and Continuous Delivery are being widely adopted — you’ll need to make that sure you can trust your test execution to be stable.

Create a solid test data management strategy

One of the hardest aspects of creating a solid Selenium testing strategy is the question of how to deal with test data. For any sufficiently complex application, there will be test cases that require the application to have data such as bank accounts, subscriptions — you name it — in a specific state before the test can be executed. The test data problem only gets harder when there are connected dependencies that have their own specific requirements with regards to the test data used. There are several ways to tackle this problem:

Creating the test data anew before the start of every test. This ensures that the right test data is available at all times, but it has the downside of increased test execution time especially when there are no hooks or access points that allow you to create test data below the user interface, and there’s risk of test execution failing before the part that comprises the actual test has even started.
Querying the system for existing test data. This minimizes the time needed to execute tests because they use data that is already in the system. There is a risk of test data not being present, resulting in a failing test, or test data being selected that is not suitable for the test under hand. This can lead to defects that might be hard to analyze or tests that pass when they shouldn’t.
Reusing test data from previous tests. This might look like a solid approach, however, it creates dependencies between tests. This can lead to several interdependent tests failing just because of a single defect that prevented a test that was responsible for creating test data to pass. It might also lead to false positives in case of parallel test execution.

As you can see, there is no one true way to manage test data. I would recommend tackling this problem as soon as possible, though, since it’s always easier to change the approach when you’ve got a relatively small test automation suite.

Create a solid test environment strategy

With applications becoming ever more distributed in nature, it’s getting harder to set up and manage test environments not only for your application under test, but also for all dependencies that your application interacts with during test execution. And that doesn’t even include having the right test data in place in all of these dependencies. So, whenever you’re starting on your journey to create automated tests using Selenium, it is a good idea to also take into account the requirements that your tests place on the test environment.

Is a test environment available and accessible for all of the required dependencies, all of the time? Will they contain the required test data, or is there a way you can take care of that? If the answer to any of these questions is ‘no’, it might have an effect on the scope of your tests or the frequency with which you can run your tests. If your test environments become a serious roadblock, you might consider applying a technique such as mocking or service virtualization to replace the dependencies with simulations that are under your own control instead.

As you can see, there’s much to consider when you’re creating automated tests or thinking about Selenium testing. Take care of the above aspects, and the chance that your journey will be a successful one increases significantly.

About the Author: Bas Dijkstra a test automation and service virtualization consultant who’s always looking for more intelligent ways to use tools to improve test processes and software quality. He’s been in the field for over 10 years, designing and developing test automation and service virtualization solutions that enhance and improve test teams and test processes. You can read more about him by reading his blog or following him on Twitter.

How to Use Selenium to Test Pop-Ups, Alerts, and Overlays

June 12, 2017 By Alex McPeak Leave a Comment

Whether it be for an advertisement, a call-to-action, or a warning, pops-up are a common way to draw the attention of users.

However, when these alerts and overlays are implemented, it may interfere with testers who are keen on automation. So, how do you use Selenium to close out of a window, or automate testing to interact with a pop-up? We demonstrated the commands for basic pop-ups to make test automation as seamless as possible, without reverting to manual.

Alert Types

There are a few distinct types of alert messages that pop-up with varying user asks.

Simple – A simple alert will usually give minimal information or a warning and only require one, such as clicking an “OK” button to confirm you understand the message.
Prompt– Usually a prompt alert will require the user to input some information such as a name, email, or password before being able to click “OK” to continue.
Confirmation – Confirmation alerts will request permission, which can be answered by clicking “OK” or “Cancel”.

Pop-ups and alerts are usually either Windows-based or web-based. Windows-based alerts are beyond Webdriver capabilities because they’re part of the OS instead of the browser, so you’ll probably need to rely on third-party tools such as Sikuli Script or Robot class.

Using WebDriver for Web-based Pop-ups

WebDriver, on the other hand, does the ability to interact with multiple windows, including alerts, with a method called switchTo. This allows you to switch control to the pop-up while keeping the browser in the background:

  popup = @driver.switch_to.alert

From there, these are the simple alert commands for web-based pop-ups:

To click “Cancel”

       popup.dismiss

To click “OK”

       popup.accept

To capture the alert message

       popup.get_text

To enter information

       popup.send_keys('Text')

To exit

       popup.close

An example of this might be:

   require 'selenium-webdriver'
   require 'rspec/expectations'
   require 'TimeoutException'
   include RSpec::Matchers

   def setup
      @driver = Selenium::WebDriver.for :firefox
   end
   run do 
      @driver.get 'URL'
      @driver.maximize.window
      @driver.find_elements(css: 'button')[1].click

      popup = @driver.switch_to.alert
      popup.accept

      textreturned = driver.find_element 'result'

      print(textreturned.text)

      assertTrue(textreturned.text, 'You Clicked OK')
      
   except TimeoutException:
      print 'no alert'
  
   def teardown
   end

These commands should take care of any functions you want to perform when automating testing where there are any simple, prompt, or confirmation alert-pop-ups in your browser.

What are the most common challenges that you face testing with Selenium? Tell us in the comments!

SeConf Austin: Ashley Hunsberger Talks Transformative Culture

April 17, 2017 By Alex McPeak Leave a Comment

Ashley Hunsberger is a Test Automation Architect at Blackboard, a virtual learning environment and course management system for top institutions worldwide.

Her Selenium Conference presentation, “Transformative Culture” reviewed how Blackboard established departmental goals, built teams that would ensure quality and success, and slashed execution times, while transforming the way the company thinks about software quality.

Not only was this presentation a favorite among conference attendees, it also serves as highly valuable information for any company that’s undergoing a culture change.

Ashley Hunsberger Transformative Culture at Selenium Conf

Ashley Hunsberger at Selenium Conf

A Cause for Change

A year ago, it took an hour to run Blackboard’s 400 tests, and 370 were unreliable. So, not only was the time to execute high, but the builds were constantly failing and developers had stopped caring about them.

Hunsberger quickly realized that the root of the issues they were having centered around culture and the way different departments thought about quality and communicated with each other to deploy projects.

She wanted to make a company-wide change that would enable the development team to own testing and quality and hoped to get the entire organization to put more care into test suites and maintenance.

The very first step to transform the department was a simple one. They would not longer be referred to as QA, but instead Engineering Productivity.

Hunsberger also wanted to re-establish the department mission to “Reduce the time from concept to deliverable by providing our product development teams with the tools, practices, and support to increase their productivity while maintaining high quality standards.”

She also wanted to clarify that Engineering Productivity would not take over writing tests for other teams, but instead be a resource to support the automation strategy.

The next move was to set goals including:

Provide an easily maintainable and extensible framework that would enable scrum teams to add and remove tests.
Enable the automatic and early detection of failures within the software under development.
Prevent the source of detected failures from moving any further downstream.
Accommodate all of this without impacting the engineer’s time.

Enter, Team Shangri-la

After realizing this transformation would only be possible with a proper leadership team to own the automation strategy, Hunsberger introduced Team Shangri-la, where she acted as the project owner alongside a scrum master, SETs, a feature engineer, and a CI/CD team member.

Team Shangri-la had to first establish their definition of “Done”, as Hunsberger found that in the past, a lot of teams were saying projects were done without them actually being testing, reviewed, documented, and released.

They also had to outline test suite definitions to get everyone speaking the same language:

Goals – What was the intent for each suite?
Trigger – What would trigger tests?
Gates – What happens if tests fail?
Requirements – What are the requirements for each suite?

Once the guidelines were determined, they had to work on containerizing environments for faster feedback and implementing project guardrails to provide support for nine different scrum teams.

Additionally, one prominent area of Shangri-la’s strategy was risk analysis, where they aimed to prioritize what tests would be executed and how they would be written.

By rating the likelihood of a bug occurring and what the impact on customers would be if the feature didn’t work, it became easier to evaluate and what areas need to be included in the test suite and which did not.

Charting and tracking the results also helped the team decide what kind of testing needed to be executed, as well as how the tests would be monitored and maintained.

Transformative Results

Hunsberger and her teams saw the 33 stable tests and 370 unreliable tests that ran in an hour turn on its head to now run 165 stable tests in 30 minutes.

Along the way, Hunsberger learned that not everyone has the same idea about what’s critical, and you can’t test everything. These small realizations ultimately initiated a complete culture change that encouraged them to be better communicators so that they could establish goals and meet them.

By focusing departments to be a company-wide effort, Hunsberger was able to affect better testing to support development, optimize testing, and ensure uncompromised quality across Blackboard’s projects.

The Build That Cried Broken: Building Trust in Your CI Tests with Angie Jones at Selenium Conf Austin

April 10, 2017 By Alex McPeak Leave a Comment

The CrossBrowserTesting team hung out in Austin last week for three days of Tex-Mex and test automation talk as an official sponsor at the 2017 Selenium Conference. We thoroughly enjoyed meeting our customers and conversing with other testing enthusiasts as well as attending speaker presentations to hear a few insights from the experts.

Angie Jones is a Consulting Automation Engineer providing strategies for scrum teams at LexisNexis who came to this year’s Selenium Conf to recite some of Aesop’s fables and teach us how we can apply the lessons to test automation and continuous integration to increase productivity, accuracy, and trust between departments.

These are just the few of the lessons we learned from Jones’ storytelling session.

Angie Jones at Selenium Conf

The Boy Who Cried Wolf – In this familiar story, a shepherd boy is tasked with keeping watch over the sheep and alarming the village people if a wolf comes. However, the shepherd boy yells “Wolf! Wolf” just to laugh as the villagers all come running in panic to realize there is no wolf. The boy does this quite a few times, until one day a wolf actually comes. The shepherd boy yells “Wolf! Wolf!” but nobody believes him, and the wolf eats all the sheep.

Jones asserts that all our tests are becoming like the shepherd boy and crying “Wolf” but often with false negatives. All the while, the developers — our villagers — are no longer paying attention to the alarms, and in turn, problematically ignoring broken tests as well.

The Lioness and the Vixen – A lioness and a vixen are bragging to each other about their children. The vixen tries to one-up the lioness by saying that she has a whole litter of kids, while the lioness only has one. In response, the lioness says that vixen may have more, but she has a lion, implying that her child is of higher value than the vixen’s many.

In test automation, Jones advances that quality should always prioritize quantity — the number of tests don’t matter, but how efficient they are does. Especially in a CI cycle, the more tests, the slower the process, and the fastest way to lose trust is with flaky tests. You want to make sure you’re automating quality tests, providing the best possible value to developers, and ensuring fast feedback.

The Astrologer – An astrologer went out every night looking at stars, analyzing constellations, and trying to predict the future. Unfortunately, he wasn’t paying attention to what was in front of him because he was so focused on looking up at the sky, and he fell into a deep hole.

Often times, teams will solely focus on automating new features, but it’s important to pay attention to your present builds, as well. “Everyone wants to get the newer features out the door, but no one’s looking at the state of automation right now, and it’s broken,” said Jones. “Why are we building new code when the builds are red? Don’t value new code over working code.”

The Farmer and the Stork – A farmer finds that cranes have been trespassing to eat his crops, so he devises a plan to catch them in the act. The cranes recruit a stork to come eat with them, but the stork doesn’t realize the food would be stolen and doesn’t participate. However, when the farmer casts his net to catch the cranes, he captures the stork too. The stork pleads to be let go, saying he’s done nothing wrong, but the farmer says that it doesn’t matter because he had still been there with the rest of the cranes and so would receive the same punishment.

Similarly, all it takes in testing is one flaky test to ruin the rest of the build. Once a build turns red, no one trusts the rests of the tests even if they might be reliable. Jones advises separating tests into different paths — one for stable tests that only fail when something is wrong and one for unstable tests with flaky fails. This way you can divide attention between the red builds that need to be fixed or rewritten and the green builds that developers should focus on.

The Happy Ending

Jones has 4,000 automated tests, less than 1.5 percent of which are unstable — a number so low it trump’s Google’s 3 percent. The main build is always green, unless there’s a change with the application, and everyone trusts it for feedback.

To learn more about managing flaky tests, monitoring builds, knowing when to rewrite tests, and other CI lessons based on classic tales, take advantage of the full recording of The Build That Cried Broken: Building Trust in Your CI Tests Selenium Conf presentation.

Getting to Know Selenium 3

April 7, 2017 By Alex McPeak Leave a Comment

We’ve discussed the evolution of Selenium, and the major shift from Selenium 1 to Selenium 2 in “Intro to Selenium Testing”. We also noted that latest version of developers’ preferred open source automation testing tool is actually Selenium 3.

But what are the differences present between the two versions, and how did it affect development and testing? In this post, we’ll examine the features that are components of Selenium 3.

Announcing Selenium 3.0 Beta

When Simon Stewart, the founding contributor of WebDriver, announced a new beta version of Selenium would soon be released at SeleniumConf in 2013, it caused quite a stir among the already large community of open source contributors.

After advising users to switch from the original RC APIs to WebDriver, Selenium 3 would stop supporting RC completely to be replaced by a similar, revised function that was integrated with WebDriver.

Of course, since most people had already abandoned RC at this point, this change didn’t affect too many developers. Those who still were using RC, however, would have to prepare the transition to WebDriver.

More noticeable changes would include support for Firefox through Mozilla’s geckodriver, support for Safari on macOS through Apple’s safaridriver, and support for Edge through MS’s webdriver server.

The update also required Java 8 to be running to access certain elements and stopped supporting IE prior to version 9.

In a second release note, Stewart specified with the following details:

WebDriver users will just find bug fixes and a drop-in replacement for 2.x.
Selenium Grid users will also find bug fixes and a simple update.
The WebDriver APIs are now the only APIs actively supported by the Selenium project.
The Selenium RC APIs have been moved to a “legacy” package.
The original code powering Selenium RC has been replaced with something backed by
WebDriver, which is also contained in the “legacy” package.
By a quirk of timing, Mozilla have made changes to Firefox that mean that from Firefox 48 you must use their geckodriver to use that browser, regardless of whether you’re using Selenium 2 or 3.

However, developers continued to wait three years for the release of Selenium 3, which was supposed to come out at Christmas 2013.

The Official Release of Selenium 3.0

Finally, in October of 2016, Selenium 3 was released.

Before, Selenium was responsible for driving each browser. With the update, all major browser vendors would ship their own implementations. While the specification was still in the works, it meant that Selenium was moving more toward becoming a testing framework that would operate as the default browser.

“Because the browser vendors know their browsers better than anyone, their WebDriver implementations can be tightly coupled to the browser, leading to a better testing experience for you,” said Stewart in the final release note.

Selenium 3.0 would also be integrating a W3C WebDriver specification for automation. In a follow-up interview, testing specialist Juho Perälä noted that this was a relevant tool in mobile testing through Appium and iOS Driver and it would “encourage compatibility across different software implementations of the WebDriver API.”

The purpose of the update wasn’t going to make any huge changes to the common use of Selenium. Instead, it was largely to make Selenium Core obsolete in exchange for an alternative implementation old RC interfaces to meet a rapidly advancing environment of web testing.

Looking Forward with Selenium

Selenium 3 continues to be exceedingly favored in test automation today, and SeleniumConf returns for 2017 as developers and enthusiasts maintain conversation around best practices in testing and advancing the future of Selenium. And we can only hope that when Selenium 4 is announced, it doesn’t take as long to launch.