testing frameworks Archives | CrossBrowserTesting.com

There’s a new kid on the block for open source test automation tools, and everyone’s talking about how Cypress may be an alternative to Selenium.

But while some have posed that Cypress signals the end of an era, we’re not so sure that Selenium is going anywhere anytime soon.

To better understand the similarities and differences between these two tools, let’s take a better look at what they both do.

Selenium

Selenium WebDriver is the open source automation tool we all know and love. With a history starting from the humble beginnings of IDE and transforming into Selenium 2.0 a.k.a. Selenium WebDriver, Selenium has become a foundation of many modern software teams’ test automation toolkits.

Most are familiar with Selenium’s use — it makes direct calls to the browser using each browser’s native support for automation. By writing a script in Selenese using the compatible programming languages, you can run tests that work as if you were driving the browser yourself.

While the learning curve can be steep for Selenium and requires the knowledge of at least one programming language, it has a vast community of active users and even boasts an annual user conference.

One of the foundations of Selenium is the ability to test on the most popular browsers, making it essential to cross-browser testing.

As Selenium has grown in popularity, it’s not only become a basis of many test automation strategies but has also become an integral part of many third-party tools such as CrossBrowserTesting. Once you know how to use Selenium, you can use these cloud testing tools to access thousands of browsers and leverage parallel testing to cut down on testing time.

Cypress

So how is Cypress different from Selenium, and how is it the same?

First, Cypress is said to have an easier onboarding than Selenium, where all you need to do is install the .exe to get up and running sine all the drivers and dependencies are automatically installed. With Selenium, on the other hand, you need to spend a little more time getting everything up and running.

One of the advantages of Selenium is that it supports most of the major programming languages including Java, Perl PHP, Python, Ruby, and C#, so if your team is already using one of those languages, Cypress may not be ideal since it only supports JavaScript.

In addition, since Cypress tests are written using Mocha and Chai, the syntax echoes what most Javascript users understand. This means that if you work with Javascript, it will be especially easy to start using Cypress. However, if you’re more familiar with another language, it will be fairly complex.

Cypress almost acts as a “freemium” model. Similiar to Selenium which is entirely open sourced unless you integrate with a paid tool, you can also add to Cypress’s core capabilities by leveraging the paid dashboard which includes a test runner that allows you to easily debug tests with DOM snapshots and includes video storage for private projects.

Selenium and Cypress differ in the way of their architecture, as well. Selenium is made up of bindings, or libraries, and the WebDriver, which controls the browsers. These two components work through the JSON network.

Alternatively, while Cypress is used for UI testing, it uses its own unique DOM manipulation and runs directly in the browser with no network communication. Cypress is often cited as a developer-friendly test automation tool for this reason since this often causes frustrations for them in Selenium.

Having a different architecture on top of an Electron app also allows Cypress to be so close to the web application under test that we can do things that you can’t do in Selenium such as stubbing DOM APIs.

One of the biggest gaps between Selenium and Cypress is that while Selenium runs against different browsers, as mentioned before, Cypress only supports Chrome. Selenium’s browser support is often integral to those who use it, making Cypress severely lacking in the area of cross-browser testing.

For software teams that find importance in checking their website on multiple browsers instead of just leaving their code up to chance, finding their way around this deficiency could prove challenging to Cypress users.

Cypress vs Selenium

Cypress and Selenium serve a similar purpose that is achieved in two different ways.

Selenium is a library but requires a unit testing framework or a runner plus an assertions library to build out its capabilities. Cypress provides a robust, complete framework for running automated tests but takes some of the freedom out of Selenium by confining the user to specific frameworks and languages.

If you don’t know what you’re doing when it comes to building out a test automation framework, Cypress may be a good option.

Another thing to keep in mind is that if you look at how Cypress is built, it is largely a unit testing tool that is will fit best in Javascript-focused development teams. Once you stray from these details and your team decides to experiment with other methods of test automation, you’ll find that Selenium can better accommodate those growing pains.

As a tool made by developers for developers, it’s less a replacement for Selenium and more so a way to introduce developers to test automation.

Like most instances when it comes to test automation, the tool you pick and the method you follow will largely depend on what your team decides and what works best for your organization. However, the notion that Selenium is on its last legs is misled.

If you’re still confused as to the differences between Cypress and Selenium, Automation Rhapsody provides a really great breakdown of how each performs in different categories.

Since JavaScript is widely considered the “language of the web”, it comes to no surprise that test automation frameworks for the language are the most plentiful and most popular. However, this can also make it more difficult to choose the right ones for your team.

By considering the attributes of different frameworks, it’ll become more clear as to which one is the right fit for your current workflow. Follow our guide for selecting JavaScript testing frameworks in order to make the right choice that will lead your organization to lasting success.

Criteria for Selection

When choosing JavaScript testing frameworks, there are going to be many options and different factors to consider. The following criteria are going to most influential in your decision:

E2E or Unit – An end-to-end framework is specialized to run through an entire human interaction on a website. A unit testing framework will give you different capabilities since it’s for testing single functions.

Assertions – Every end-to-end testing framework needs an assertion library. Assertions make sure that our tests return expected results. When choosing a framework, you also need to have a reason for choosing which assertion library you’re going to use and why you’re going to use it.

BDD/TDD Friendly – Some frameworks are not as easy to use with a BDD Girken style language as others, and you will want to stay away from those if your organization is following a certain style. Choose BDD or TDD and pick a framework that makes sense for your team and fits your organizational flow.

Asynchronous testing – When testing JavaScript, it’s important to worry about asynchronous testing and promises. Some frameworks will do this better out of the box than others, which will take longer to grab a promise.

Visual Testing – Some frameworks are better suited for visual testing and capturing browser screenshots than others, and some actually come out of the box with it. This will give you the ability look at visual differences in your regression suite.

CI Integrations – If your team is following Continuous Integration and Delivery practices, it’s no surprise that you’ll also probably want a framework that integrates with Jenkins, Team City, and Bamboo. Most of the CI systems run on a standard runner, so they work hand in hand with many of the larger JavaScript frameworks to make sure that they work together as well as possible, but it’s still something you want to double-check when choosing.

App Language – The app language criteria has risen recently with the dynamic nature of JavaScript. Previously it was fine to have test code and production code not be the same language. Now, with the way app languages are going in frameworks, we want to pick a framework that is best suited for the actual language that our application is written in.

Speciality – You may be searching for a framework with a certain specialty. Is it easy to set up and get running out of the box? Is there a clean and simple syntax? Is it compatible with Angular testing or a Selenium binding? Does it have good customization? Is it flexible, and can it fit into other assertion libraries, code coverage tools, and frameworks? These are all considerations you’ll want to assess with your team.

Cloud Support – Pay attention to which frameworks work with cloud providers because not every framework will out of the box. Running tests in the cloud will allow you to increase test coverage and speed while eliminating hassle, cost, and maintenance, so compatibility should be part of your consideration.

Size of the Community – The size of the community matters because it’s where you look for answers and find tutorials and documentation, but this also depends on personal preference. For example, one framework may be good because it’s newer so more people are talking about it, but some might prefer a more mature framework that already has a good community base and documentation.

Top JavaScript Frameworks

To get a better idea of specific frameworks and how they meet these criteria, we listed some of the top frameworks used by teams that work in JavaScript.

Assertion Libraries

Jasmine – Jasmine is a BDD/TDD compatible unit testing framework with an out-of-the-box assertion library, an easy to read syntax, and broad Angular support.
Should.js – Should.js has a BDD assertion library and often pairs with Mocha.
Assert.js – Assert.js works with Angular and often is used with Chai to provide a more simple set of assertions.

Unit Testing Frameworks

Mocha – The self-proclaimed simple, flexible and fun framework is one of the most popular choices. It depends on other frameworks like Chai for assertions, but it is BDD/TDD compatible and includes a runner. Mocha is a very flexible, extendable, and feature-rich configuration that’s also good for asynchronous testing.
Chai – Chai often goes hand-in-hand with Mocha as the most popular assertion library used alongside it. It is also BDD/TDD compatible.
Tape – Tape is also a newer framework for unit testing that some might uses as an alternative to Mocha. It uses a traditional syntax, making it very simple to use.
Jest – Jest is a runner for unit testing that can adapt to any JavaScript library or framework. It comes with its own assertion library and supports BDD. It’s a newer framework, so it’s less widely used than others.

E2E Frameworks

Protractor – Protractor has risen in popularity as a good choice for Angular apps because it puts a 10-second wait at the end of each command for a JavaScript single page app. This is an example of something you’d want to consider when thinking about app language. It would also depend on assertion libraries from another framework.
PhantomJS– Phantom.js deals with headless browsers but has had a drop-off in popularity since Chrome came out with their own and real browsers have been optimized to be comparably fast to headless ones.
Webdriver.io – Webdriver.io includes visual testing with WebDriver CSS. It also specializes in customization and is a flexible framework where you can fit assertion libraries, code coverage tools, and other frameworks.
Nightwatch – Nightwatch is an E2E test framework with its own assertion library and runners. It’s an easy, out of the box setup that a simple syntax, which couples together with a configuration file. Nightwatch is also a custom implementation of W3C WebDriver and works with the JSON wire protocol.
CapserJS – Casper.js also works for headless browser testing with a runner for Mocha, Jasmine, and Chai for end-to-end testing. It has added capabilities like assertions and testing utilities that PhantomJS lacks.
Karma – Karma is a test runner created by the AngularJS team, making it often the preferred test runner for apps written in Angular. It’s frequently used alongside Jasmine and integrates with popular CI servers as well.
Cucumber – Cucumber is a popular BDD framework for functional testing using the Gerkin syntax that works with a lot of these tools and runners, like WebDriver.io. It’s being used more and more as more teams use BDD.
InternJS – Intern is a fairly new framework that’s BDD/TDD compatible. As a test runner, it also integrates with CI tools like Jenkins and TeamCity as well the cloud. Don’t let the immaturity of this framework fool you, it has comprehensive documentation and a strong community.

Selenium vs. Cypress: Is WebDriver on Its Way Out?