continuous testing Archives | CrossBrowserTesting.com

Parallel Testing: What It Is and Why You Should Adopt It

December 13, 2019 By Victoria.Bezsmolna Leave a Comment

Our friends over at Bitbar recently wrote another fantastic blog about parallel testing, what it is and why you should adopt it. Take a look below!

The continuous integration and delivery methodology is a great way to keep you releasing high-quality products in the shortest possible time. All of this relies on continuous testing and requires a faster test cycle. While the traditional testing methodology – sequential testing – means a long queue of test jobs and longer time-to-market, parallel testing is the favored approach for higher test efficiency and quicker turnaround time in software deliveries.

What is Parallel Testing?

Parallel testing is an automated testing process that developers and testers can launch multiple tests against different real device combinations and browser configurations simultaneously. The goal of parallel testing is to resolve the constraints of time by distributing tests across available resources.

For example, if 20 test cases take a total of 100 minutes to complete, then 10 parallel execution could run 2 test cases each and bring the total testing time down to 10 minutes. Ideally, if you have sufficient resources, say 20 real mobile devices and real browsers for simultaneous execution of 20 test cases, then you’ll be able to significantly shrink the run time to 5 minutes.

On the other hand, sequential testing or serial testing is the opposite of parallel testing. In the above case, 20 test cases are executed one after another, and you will need to waiting 100 minutes for the test results.

Benefits of Parallel Testing

As we have scratched the surface of what parallel testing is and how it influences the test cycle, there are a couple of more reasons why you should consider testing in parallel.

Speed

Sequential testing is time-consuming, while parallel testing will allow you to divide invested time by the number of environments. In order to test your application against ten devices, all you need to do is write only ONE script and run it against all your target devices, cutting your testing time by ten times.

Cost-Efficiency

The building, maintaining and keeping your own test environment up-to-date can burn a hole in your pocket. When it comes to parallel testing, maintenance isn’t a headache anymore — in fact, you lease the needed testing environment, always updated. Plus, cloud-based testing grids allow you to run tests at high concurrency, making the cost per test significantly lower.

Better Coverage

It’s always a good idea to run your application through as many platform-device-browser combinations as possible so that no bug sneaks in. Parallel testing will take your test coverage to the next level, giving you a significant ROI boost.

Optimization of Your CI/CD Processes

Parallel testing is the best friend of continuous integration and delivery. By testing in parallel, you can run tests as soon as developers submit new code updates throughout the entire SDLC. Timely reporting and quick feedback in parallel testing will also facilitate better communication between various departments.

Improvement of Testing Practices

Parallel testing improves the QA routine in your company. The reason is crystal clear: by testing at high speed, you can test more. This gives your QA team a chance to improve their testing practices and pinpoint bugs faster.

Gradual Transition from Sequential Testing

Transitioning your QA regime from sequential to parallel testing is a huge undertaking if you try to do all at once. Starting small might be your best bet. You can begin with the test cases that are most adaptable to the parallel testing environment, giving your engineers enough time to adjust the rest of the tests.

Tips for Successful Parallel Testing

Many companies switched to parallelism only in some instances and avoid parallelizing all their QA processes. Why? Infrastructure limitations, data dependencies, poor test cases data management and hard-coding are the most common constraints. The good news is there are ways to nail them down and reap all the benefits of it properly.

Test in the Cloud-Based Environment

Implementing a parallel testing strategy using in-house resources is one of the most typical mistakes. Building and maintaining your own infrastructure is not efficient. It is not just time- and cost-consuming – you won’t be able to make the most of parallel testing where the ability to test at high concurrency is a major advantage. Also, keeping your testing environment up-to-date requires a lot of resources. To this end, it’s a good idea to turn to cloud-based services that allow you to access the necessary devices at any time.

Avoid Dependencies Between Tests

Dependencies between different test cases are a primary reason why transitioning to parallel testing is so challenging for many teams. Simply put, when test cases are dependent on each other, you should run them in a particular order, which can destroy any parallel testing strategy. So, it is critical to creating your test cases to be atomic, independent from each other. Only then, you will be able to run them at any time and in any order, making your testing processes free of constraints.

Avoid Hard-Coding

Hard-coding is embedding data directly into the source code instead of generating it at runtime. This notion is an enemy of efficient parallelization since it creates dependencies between test cases, mentioned above. It is strongly recommended to avoid hard-coding values when scripting your test cases to ensure that each of your tests is self-sufficient and can be run whenever necessary. Use the data-driven approach to configure your test cases when they are running in parallel.

Manage Test Data Efficiently

Efficient test data management is the key to successful parallel test execution. But first, you need a strategy. There are two components you might want to add to it: a creational strategy (which creates test data needs) and a cleanup strategy (which cleans it up). Still, the only thing that matters is that your strategy is adapted to your particular case. The following ones are quite basic, and perhaps that’s what makes these data management strategies so effective:

Elementary approach, which has no creational and cleanup strategies
Refresh-your-data approach, which resets your source code in between test executions but has no creational strategy
Selfish data generation approach, which has a creational strategy but no clean up one.

These are the most basic strategies. You can mix and match them to serve your own case or explore alternatives, such as generating other data or refreshing specific data.

Build Tests to be Run in Parallel

You don’t want to end up with a bunch of tests that can’t be run in parallel right before the release date. Planning correctly from the get-go will save you from nasty surprises when time’s up or when a deadline is due. Create your test cases with parallelization in mind should be your way to go. And believe us, it sounds harder than it actually is: test cases created for running in parallel are simpler, shorter and faster to build.

Conclusion

Parallel testing can save your team from delays in delivery without compromising the quality, which makes it superior to sequential testing.

With parallelization, you can cut your QA expenses, run cases at high concurrency, optimize your CI/CD processes and constantly improve your scripts to get more accurate results. At the same time, all these advantages are out of reach without a correct strategy. Testing in a cloud-based environment, avoiding hard-coding, eliminating data dependencies and scripting with parallelization in mind will help you enjoy all the benefits of parallel testing.

3 Methods for Hands-free Continuous Testing

December 4, 2018 By Alex McPeak Leave a Comment

In order to strike the perfect balance between speed and quality, teams that hope to achieve Continuous Integration and Continuous Delivery must also implement Continuous Testing.

This is often easier said than done. It’s virtually impossible to have your team running tests at every moment — they do have lives outside of work after all, and it can be difficult to put dinner on the table while running a Selenium script at the same time. So how do you run your scripts, even if you aren’t at your desk?

Fortunately, there are various hands-free options for continuous testing that you can implement, which will keep tests running for you while you eat, sleep, or maybe even get some exploratory testing done.

Why Continuous Testing

What’s the point of continuous testing? There are a few major advantages that will enable you to achieve more in your day-to-day.

Test while you sleep – With continuous testing, you can literally test in your sleep. By setting tests to run overnight, you can experience the luxury of coming in the next morning to have your test results waiting for you and knowing with a glance whether they passed. Just ask America’s Test Kitchen about how convenient it is.
Keep up with Agile, DevOps and CI/CD – When software development teams are constantly integrating, you need a testing strategy to ensure those new features, fixes, and changes are put through the quality process, no matter how frequently they’re implemented. Continuous Testing is the only way QA teams can keep up with constantly changing features, frequent integrations, and fast workflows.
Catch bugs in each regression – When there are constantly small changes to an application’s code, it can break something that was previously functioning. And you don’t want your customers to be the one to catch it. Continuous Testing will help you be first to spot new bugs, so you can quickly debug and release to users with confidence.
Level up your automation game – Automate your automation with continuous testing, and set your tests to run as often as you need. Then come back to test results or screenshots that tell you exactly what you need to know about the health of your applications. Plus, testing in parallel allows you to further accelerate Continous Testing to increase browser coverage without compromising runtime.

Continuous Testing With a CI Server

Selenium WebDriver has long provided teams with the ability to automate tests across browsers. But in order to run those tests continuously, a Continuous Integration tool such as Jenkins, VS Team Services, Buildbot, Bamboo, CircleCI, TravisCI, Codeship, or TeamCity is needed.

By far, the most popular option is Jenkins, an open source automation server written in Java. In fact, according to the SmartBear 2018 State of Testing Survey, 66 percent of participants choose Jenkins to be their CI server. For development teams adopting Agile and DevOps workflows, Jenkins is a reliable tool for continuous integration and delivery.

But what about testing teams that want to leverage Jenkins for Continuous Testing with Selenium? The CrossBrowserTesting integration with Jenkins lets you run automated functional tests with Selenium, Appium, and visual screenshot tests, which means that every time you run your build process, you can simultaneously run your testing suite across thousands of browsers.

This is ideal for teams that already work with Selenium and a CI tool for deployment to start testing on a continuous basis. While Jenkins tends to be the most popular choice for CI and automated test scheduling, CrossBrowserTesting also integrates with TeamCity and VS Team Services, so no matter what tools you’re already using, it’s easy to integrate CrossBrowserTesting into your current workflow.

Continuous Testing With a Record and Playback Tool

For teams that may be less familiar with Selenium but still wish to automate, a record and playback tool might be a more practical option.

Record & Replay enables teams to achieve automation across browsers, just like they would with Selenium, without requiring knowledge of scripting and frameworks. This has provided teams with a more accessible option for less technical team members, or those who are just getting started with automation, to create automated tests. As tools like Selenium IDE and its alternatives are brought back into the spotlight, these options are becoming more and more attractive to modern testing teams.

But where record and playback tools will normally only run the test once to check if the test passes, how can testers implement Continuous Testing?

With new Scheduling in CrossBrowserTesting, you can schedule a Record & Replay suite to run as often as you need them. For organizations that have less technical testers, this allows you to achieve continuous testing without requiring the skills to run Selenium and Jenkins or another automation server.

Continuous Screenshot Testing

Teams that solely focus on functional testing without incorporating visual testing will be disappointed when all their tests pass and users are still pointing out visual bugs.

When visual elements can change from browser to browser or device to device and make the user’s experience difficult or impossible while still functioning correctly, it’s important to run compare screenshots in regression. Comparing these pages side by side and with their historical versions means you can be more aware of changes that break the application in order to debug before they reach the end user.

While you could always run automated screenshots in CrossBrowserTesting, it required a manual motion to rerun those tests on a certain cadence. You can now set those screenshot tests to run with the new Scheduling feature, just like you can with Record & Replay.

Continuous Testing for Every Team

No matter what tools you’re already using, what technical level you’re at, or how far your team is in their test automation journey, there are options to help every team achieve continuous testing in CrossBrowserTesting.

With options to schedule tests with Selenium, Record & Replay, and automated screenshots, you can take your automation to the next level and receive test results through email or Slack on a daily, weekly, or monthly basis that you decide.

Understanding Continuous Integration Testing

November 27, 2017 By Alex McPeak Leave a Comment

As software development practices increasingly trend towards Agile development and DevOps, Continuous Integration and Delivery have become the go-to methods for receiving fast feedback, meeting changing requirements, and optimizing quality.

But what exactly is CI/CD and how does testing fit in? As more organization adopt these practices, some will increase testing, while some will let it fall through the cracks. Find out how teams are including continuous integration testing to balance speed and quality more successfully.

What is Continuous Integration?

Continuous Integration is a practice where a team of developers intermittently merges their code changes into a shared repository. Developers are integrating multiple times throughout the day and ideally committing to changes about once a day or a few times a week.

Continuous Integration is preferred by software teams because it allows daily constant feedback in development from all contributors, while it’s easier to catch and resolve bugs early on in the process.

In turn, this ideally leads to Continuous Delivery — since builds are constantly managed and shipped regularly, new features can be delivered to the end user on a daily or weekly basis instead of a quarterly or yearly one.

Additionally, by integrating small, frequent changes, it’s easier to catch issues early in the SDLC. Teams often find that this CI/CD results in higher quality software with fewer bugs, and when bugs are found, they’re usually much easier to fix.

Where CI/CD and Agile Intersect

Continuous Integration piggy-backs onto Agile development since Agile is often achieved through the continuous integration of features, as it was created so developers could solve issues as they come instead of trying to predict and solve every change up front and only test right before the product launch.

While teams following an Agile methodology strive to support more change throughout the development process, Continuous Integration is a way that developers to do this by making changes multiple times throughout the day or week. Additionally, sprint planning allows for the constant feedback from both customers and developers as well as other team members.

However, though Continuous Integration often works best as part of Agile or Extreme Programming, it can also be leveraged in other environments such as Waterfall or RUP.

The Need for Continuous Integration Testing

Another element of CI and CD that is less commonly acknowledged is Continuous Testing, or the practice of testing for every integration. You may even consider continuous testing a prerequisite for Continuous Integration and Delivery.

It makes sense that Continuous Testing goes hand-in-hand with Continuous Integration to make sure that bugs are found early and are easy to fix. While many changes are made on a daily basis, there’s an opportunity each time for those changes to disrupt a previously working part of the code.

While teams implement Continuous Integration and Delivery to gain a competitive edge, it’s important to acknowledge that ensuring quality is equally important in the eyes of the end user — no one wants to use a new feature that has a bug. Continuous Integration testing is a critical step for organizations that want to differentiate from their competition.

By automating tests to match the speed of Continuous Integration, rapid delivery can be more effectively achieved and acceptable standards of quality can be met simultaneously.

Open Source CI Tools

Utilizing a Continuous Integration strategy is often complemented with the use of one or more open-source tools.

Jenkins – Jenkins is the by far the most popular open source continuous integration tool. It’s a Java-based automation server that has hundreds of CI/CD plugins to support building, deploying, and automation. Basically, Jenkins works to check for changes on the server, trigger a build when it detects one to deploy it for testing, and notify test teams of found results. It has a robust, supportive community, easy installation, and numerous integrations that make it a leading Continuous Integration server. Learn about installing and configuring Jenkins here.
Buildbot – Buildbot is primarily used to manage and schedule different jobs in the Continuous Integration process. It supports job execution across multiple platforms and monitors source-code repositories while reporting the results. Buildbot can only be configured through a Python script, though it does support all major operating systems.
TeamCity – TeamCity allows you to develop in Java, .NET, or for mobile platforms, while giving you additional functionalities and features such as gated commits, build grids, and integrations as well as more than 100 plugin options, while the Open API allows you to create your own plugins. Read about installing TeamCity.

For Continuous Integration testing, open source tools like Selenium and Appium are most popular for automating tests. Additionally, tools like CrossBrowserTesting can also be used to execute test automation and create an environment for continuous testing in the cloud.

Additionally, while automation is essential for keeping up with the speed of a well-oiled CI process, it won’t always be the answer to testing. For brand new features, it’s best to resort to exploratory testing. For other instances where teams should trade in automation for live testing, check out our blog about deciding which tests to automate.

Conclusion

As more QA and DevOps professionals begin to implement Continuous Integration and Delivery, Continuous Testing will have to be quick to follow in order to better meet customer demands, keep a competitive edge, and secure a standard of quality.

As technology advances and web testing trends change, being able to adapt to change and follow faster development cycles will become the differentiator for many software teams. Continuous testing is how those teams will meet demands of agile practices, frequent integrations, and rapid delivery cycles.

Using Jenkins and CrossBrowserTesting For Continuous Testing

January 1, 2017 By Daniel Giordano Leave a Comment

Today, CrossBrowserTesting has launched our redesigned Jenkins integration for running automated functional and visual tests during your continuous integration and continuous testing process. Jenkins is an open-source CI build and deploy system that helps software teams all over the world automate their build processes.

CrossBrowserTesting’s new integration allows you to access our entire real browser and mobile device cloud made up of over 1500 configurations to conduct and report on your automated browser testing right from the Jenkins Build Report. Run Selenium, Appium, and other testing frameworks to test your websites cross-browser functionality, or use our REST API to automate our powerful visual testing capabilities.

Core Capabilities:

When using Jenkins with CrossBrowserTesting in your CI process, you can assure that your Selenium tests written in the language of your choice will run fast, securely, and reliably.

During test automation, run visual regression and comparison tests using our Screenshot Comparison Engine. Set a baseline browser and continually make sure its DOM elements and UI layout matches that of new builds on different browsers and devices before your website deploys.

Make your debugging simple by providing test reports, video recordings, network packets and more to your team – all automated by Jenkins and CBT.