The Anatomy of a Comprehensive Selenium Testing Strategy

Now that test automation has established itself as a key enabler of shorter software development and delivery cycles, being fluent in Selenium WebDriver (Selenium) will give test automation engineers an edge in the current IT job market. However, there’s a lot more you need to know to create a successful Selenium testing implementation than just the Selenium API itself. In this article, we’ll take a closer look at what it takes to create, use, and maintain a Selenium-based test automation solution that’s able to live up to expectations.

Is Selenium testing right for you?

Before you implement Selenium, it’s wise to ask yourself whether it’ll be a good fit for your test automation efforts in the first place. While it’s true that Selenium implementations are available for a multitude of programming languages and there’s a wealth of information around Selenium testing available online, creating a solid solution around Selenium and implementing robust, reliable, and maintainable test scripts is not an easy task. It requires both solid development and testing skills to write, run, and maintain a Selenium testing suite that lives up to its promise of faster feedback and shortened delivery cycles, instead of being a time and effort drain that slows down your software development efforts.

The remainder of this article illustrates the most important skills needed and the factors that should be taken into consideration to make your Selenium testing strategy successful once you’ve decided it’s the right direction.

Choose your tests wisely

Once you’ve got the hang of creating automated Selenium tests, it might be tempting to replace all of your current tests with Selenium-based counterparts. However, before you start automating away, it might be a good idea to consider the following:

Despite being one of the most commonly used user interface-driven automated test tools, creating high quality, reliable Selenium testing is hard. Maintaining a large set of these tests, especially in fast moving environments, is even harder.
Selenium-based tests offer you the broadest scope since they exercise an application all the way from the user interface through API and business logic layers all the way down to the database (and back again). This has a downside, too. Whenever a defect occurs at a lower level in the application, there’s a significant chance that your Selenium test isn’t able to pinpoint the root cause of the failure, only its effect on the end user experience. This is also known as shallow feedback, and it’s something you want to avoid when you’re trying to pinpoint and fix defects fast.
Selenium testing is much slower in terms of execution compared to unit or API-level tests. When you’re striving for fast feedback, you want your automated test suite to give you this feedback in the shortest amount of time possible.

What the above boils down to is simple: never use Selenium (or any other user interface-driven tool, for that matter) to automate a test that can be done at a lower level such as the API layer. As a general rule, Selenium and similar tools should only be used to verify that an end user of the application can successfully complete a sequence of actions in the application.

Apply good programming practices

Despite the ongoing advertisement of ‘codeless’ test automation tools that ‘allow everyone to create automated tests’, test automation still is a form of software development and should be treated as such. This applies to Selenium testing as well. Since you’re writing your tests in code, it makes sense to apply good programming practices to this code as well:

Make your tests maintainable by leveraging patterns that promote modularity and reuse. A single change in your application under test such as the introduction or removal of a text field should require only a single change in your test code. Probably the best known and most widely used pattern in Selenium is the Page Object pattern.
Keep your tests readable by using descriptive method and variable names. The intent of a test method named withdrawMoney_ResultsInError_IfBalanceIsInsufficient() is clear, while a test method named testCase147() or testWithdraw() is far less so. You can find more suggestions for useful test method names here. Be sure to apply readable standards to variables as well. You’ll see that a well-structured test with a descriptive name is pretty much self-documenting.
Optimize the stability of your tests to prevent false positives. Failing tests can be quite costly in terms of time needed to analyze the root cause, and time spent analyzing false positives is time wasted. Unfortunately, Selenium tests are among the tests most prone to false positives. One tried and tested way to create more stable tests is by creating wrapper methods around the Selenium API to create a single place where synchronization, exception handling, and additional logging and reporting can be implemented. If you want to be able to rely on the results of Selenium testing — something that’s becoming more important now that Continuous Integration and Continuous Delivery are being widely adopted — you’ll need to make that sure you can trust your test execution to be stable.

Create a solid test data management strategy

One of the hardest aspects of creating a solid Selenium testing strategy is the question of how to deal with test data. For any sufficiently complex application, there will be test cases that require the application to have data such as bank accounts, subscriptions — you name it — in a specific state before the test can be executed. The test data problem only gets harder when there are connected dependencies that have their own specific requirements with regards to the test data used. There are several ways to tackle this problem:

Creating the test data anew before the start of every test. This ensures that the right test data is available at all times, but it has the downside of increased test execution time especially when there are no hooks or access points that allow you to create test data below the user interface, and there’s risk of test execution failing before the part that comprises the actual test has even started.
Querying the system for existing test data. This minimizes the time needed to execute tests because they use data that is already in the system. There is a risk of test data not being present, resulting in a failing test, or test data being selected that is not suitable for the test under hand. This can lead to defects that might be hard to analyze or tests that pass when they shouldn’t.
Reusing test data from previous tests. This might look like a solid approach, however, it creates dependencies between tests. This can lead to several interdependent tests failing just because of a single defect that prevented a test that was responsible for creating test data to pass. It might also lead to false positives in case of parallel test execution.

As you can see, there is no one true way to manage test data. I would recommend tackling this problem as soon as possible, though, since it’s always easier to change the approach when you’ve got a relatively small test automation suite.

Create a solid test environment strategy

With applications becoming ever more distributed in nature, it’s getting harder to set up and manage test environments not only for your application under test, but also for all dependencies that your application interacts with during test execution. And that doesn’t even include having the right test data in place in all of these dependencies. So, whenever you’re starting on your journey to create automated tests using Selenium, it is a good idea to also take into account the requirements that your tests place on the test environment.

Is a test environment available and accessible for all of the required dependencies, all of the time? Will they contain the required test data, or is there a way you can take care of that? If the answer to any of these questions is ‘no’, it might have an effect on the scope of your tests or the frequency with which you can run your tests. If your test environments become a serious roadblock, you might consider applying a technique such as mocking or service virtualization to replace the dependencies with simulations that are under your own control instead.

As you can see, there’s much to consider when you’re creating automated tests or thinking about Selenium testing. Take care of the above aspects, and the chance that your journey will be a successful one increases significantly.

About the Author: Bas Dijkstra a test automation and service virtualization consultant who’s always looking for more intelligent ways to use tools to improve test processes and software quality. He’s been in the field for over 10 years, designing and developing test automation and service virtualization solutions that enhance and improve test teams and test processes. You can read more about him by reading his blog or following him on Twitter.

Vishal Patidar

Just this morning I was having this discussion with one of the project managers of a big banking platform and we both agreed that going for Automation without taking other details in account is like walking on a cliff blindfolded. He said I raised my concern among other stakeholders that first we need to assess whether we need automation at this point. They work on huge complex banking business logic. Its really long hauling process.
Thank you Bas for such an amazing guidelines for Automation Strategy.
Anand Jayaram

Test Data management is a key point. I have seen multiple customers failing on that front. Reporting is another key area which I think is not given enough importance. Unless you have an easily accessible and readable report, your automation is not going to be worth much. People forget that -“ones who write the automation tests need not be the ones analyzing it”.

Join Over 200,000 Testers & Developers And Start Testing Today