parallel testing Archives | CrossBrowserTesting.com

The Benefits of Increased Parallel Testing

June 2, 2020 By Nick Bonofiglio Leave a Comment

Quality at speed is the new mantra for most CIOs. In response, Development Operations has evolved their software lifecycle- becoming both more agile and more scalable while attempting to deliver a flawless experience. In response, companies are constantly delivering new products and updates to consumers, and the speed with which they can deliver quality code and features is limited by a dependency on slow and intricate testing process. This rapid pace of change directly effects the functional testing space. How can you deliver products at speed while maintaining the highest standards of experience?

Enter parallel testing.

Instead of running single tests sequentially, parallelization is the process of running multiple automated test cases concurrently. With parallel testing, organizations can drastically save on the amount of time it takes to execute automated tests while dramatically improving the scope and scale of said tests.

The ROI is immediate and can be easily quantified because there is a direct correlation between the number of concurrent executions and the time that it takes to run the test suite. For example, let’s say a team has a suite of 100 automated tests and each test take an average of one minute to execute. That would mean that the test suite takes 100 minutes to run, if executed sequentially. However, if that team distributes that test suite across 10 parallels running concurrently, they could reduce the execution time ten-fold to just 10 minutes total. This results in a significant amount of time being put back in the developers’ hands.

Example code for implementing parallel test execution on SmartBear’s CrossBrowserTesting platform, using JavaScript via NodeJS. This sample, as well as samples for other languages and frameworks can be found at https://github.com/crossbrowsertesting/.

And, dramatically reducing testing time isn’t the only benefit of parallel testing:

Improve Fluidity in your Development Process

An increasing number of DevOps teams have adopted the process of Continuous Integration and Continuous Delivery, a type of process that requires frequent and reliable functional testing. Dev teams who get test results back faster are able to maintain a more constant flow of code writing, testing, deploying to production all of this repeated as fast as possible.

Implementing a parallel testing process can optimize that continuous flow. Getting results back quickly is a core part of CI/CD, because the sooner developers do, the sooner they can begin debugging and refactoring, or pushing to production. In addition to that, organizations can reduce cost and increase accessibility by moving test environments to the cloud.

Example code for implementing execution into a Jenkins pipeline in NodeJS using the CrossBrowserTesting Jenkins plug-in. This plug-in allows users to specify their desired environment configurations in the pipeline, itself, separate from the test framework for increased flexibility. Learn More

Substantially Reduce Testing Costs

By running parallel tests concurrently on virtual machines in a cloud environment, you can dramatically reduce your costs per test. Working with a partner like SmartBear – who offers more combinations of virtual machines and real devices in both public and private cloud environments – helps remove the burden of maintaining test environments and devices, which is costly. Parallel testing is the way to take advantage to all those browser combinations and devices. Speaking of which…

Drastically Increased Test Coverage

Because parallel testing is faster – and with access to thousands of device and browser combinations in SmartBear’s public cloud – your team can make sure that the breadth of your testing efforts is much wider. If your customers are a diverse group spanning all sorts of mobile devices, and with their own preferred operating systems and browsers, you want to make sure that the code you push covers all of those bases. One unhappy user with an edge case device or browser type can be enough to harm your business’ reputation. With parallel testing, you’re much more likely to avoid that scenario.

Changing from sequential testing or manual testing to an automated testing environment with multiple parallels can be intimidating for some teams, but it doesn’t have to be scary! You can take baby steps, increasing first to two parallels and setting up a few automated tests there, before continuing to scale up. SmartBear is here to help you explore the many benefits of parallel testing.

Parallel Testing: What It Is and Why You Should Adopt It

December 13, 2019 By Victoria.Bezsmolna Leave a Comment

Our friends over at Bitbar recently wrote another fantastic blog about parallel testing, what it is and why you should adopt it. Take a look below!

The continuous integration and delivery methodology is a great way to keep you releasing high-quality products in the shortest possible time. All of this relies on continuous testing and requires a faster test cycle. While the traditional testing methodology – sequential testing – means a long queue of test jobs and longer time-to-market, parallel testing is the favored approach for higher test efficiency and quicker turnaround time in software deliveries.

What is Parallel Testing?

Parallel testing is an automated testing process that developers and testers can launch multiple tests against different real device combinations and browser configurations simultaneously. The goal of parallel testing is to resolve the constraints of time by distributing tests across available resources.

For example, if 20 test cases take a total of 100 minutes to complete, then 10 parallel execution could run 2 test cases each and bring the total testing time down to 10 minutes. Ideally, if you have sufficient resources, say 20 real mobile devices and real browsers for simultaneous execution of 20 test cases, then you’ll be able to significantly shrink the run time to 5 minutes.

On the other hand, sequential testing or serial testing is the opposite of parallel testing. In the above case, 20 test cases are executed one after another, and you will need to waiting 100 minutes for the test results.

Benefits of Parallel Testing

As we have scratched the surface of what parallel testing is and how it influences the test cycle, there are a couple of more reasons why you should consider testing in parallel.

Speed

Sequential testing is time-consuming, while parallel testing will allow you to divide invested time by the number of environments. In order to test your application against ten devices, all you need to do is write only ONE script and run it against all your target devices, cutting your testing time by ten times.

Cost-Efficiency

The building, maintaining and keeping your own test environment up-to-date can burn a hole in your pocket. When it comes to parallel testing, maintenance isn’t a headache anymore — in fact, you lease the needed testing environment, always updated. Plus, cloud-based testing grids allow you to run tests at high concurrency, making the cost per test significantly lower.

Better Coverage

It’s always a good idea to run your application through as many platform-device-browser combinations as possible so that no bug sneaks in. Parallel testing will take your test coverage to the next level, giving you a significant ROI boost.

Optimization of Your CI/CD Processes

Parallel testing is the best friend of continuous integration and delivery. By testing in parallel, you can run tests as soon as developers submit new code updates throughout the entire SDLC. Timely reporting and quick feedback in parallel testing will also facilitate better communication between various departments.

Improvement of Testing Practices

Parallel testing improves the QA routine in your company. The reason is crystal clear: by testing at high speed, you can test more. This gives your QA team a chance to improve their testing practices and pinpoint bugs faster.

Gradual Transition from Sequential Testing

Transitioning your QA regime from sequential to parallel testing is a huge undertaking if you try to do all at once. Starting small might be your best bet. You can begin with the test cases that are most adaptable to the parallel testing environment, giving your engineers enough time to adjust the rest of the tests.

Tips for Successful Parallel Testing

Many companies switched to parallelism only in some instances and avoid parallelizing all their QA processes. Why? Infrastructure limitations, data dependencies, poor test cases data management and hard-coding are the most common constraints. The good news is there are ways to nail them down and reap all the benefits of it properly.

Test in the Cloud-Based Environment

Implementing a parallel testing strategy using in-house resources is one of the most typical mistakes. Building and maintaining your own infrastructure is not efficient. It is not just time- and cost-consuming – you won’t be able to make the most of parallel testing where the ability to test at high concurrency is a major advantage. Also, keeping your testing environment up-to-date requires a lot of resources. To this end, it’s a good idea to turn to cloud-based services that allow you to access the necessary devices at any time.

Avoid Dependencies Between Tests

Dependencies between different test cases are a primary reason why transitioning to parallel testing is so challenging for many teams. Simply put, when test cases are dependent on each other, you should run them in a particular order, which can destroy any parallel testing strategy. So, it is critical to creating your test cases to be atomic, independent from each other. Only then, you will be able to run them at any time and in any order, making your testing processes free of constraints.

Avoid Hard-Coding

Hard-coding is embedding data directly into the source code instead of generating it at runtime. This notion is an enemy of efficient parallelization since it creates dependencies between test cases, mentioned above. It is strongly recommended to avoid hard-coding values when scripting your test cases to ensure that each of your tests is self-sufficient and can be run whenever necessary. Use the data-driven approach to configure your test cases when they are running in parallel.

Manage Test Data Efficiently

Efficient test data management is the key to successful parallel test execution. But first, you need a strategy. There are two components you might want to add to it: a creational strategy (which creates test data needs) and a cleanup strategy (which cleans it up). Still, the only thing that matters is that your strategy is adapted to your particular case. The following ones are quite basic, and perhaps that’s what makes these data management strategies so effective:

Elementary approach, which has no creational and cleanup strategies
Refresh-your-data approach, which resets your source code in between test executions but has no creational strategy
Selfish data generation approach, which has a creational strategy but no clean up one.

These are the most basic strategies. You can mix and match them to serve your own case or explore alternatives, such as generating other data or refreshing specific data.

Build Tests to be Run in Parallel

You don’t want to end up with a bunch of tests that can’t be run in parallel right before the release date. Planning correctly from the get-go will save you from nasty surprises when time’s up or when a deadline is due. Create your test cases with parallelization in mind should be your way to go. And believe us, it sounds harder than it actually is: test cases created for running in parallel are simpler, shorter and faster to build.

Conclusion

Parallel testing can save your team from delays in delivery without compromising the quality, which makes it superior to sequential testing.

With parallelization, you can cut your QA expenses, run cases at high concurrency, optimize your CI/CD processes and constantly improve your scripts to get more accurate results. At the same time, all these advantages are out of reach without a correct strategy. Testing in a cloud-based environment, avoiding hard-coding, eliminating data dependencies and scripting with parallelization in mind will help you enjoy all the benefits of parallel testing.

Maximum Concurrency with Nightwatch.js

January 9, 2019 By John Reese Leave a Comment

You made it. After weeks writing a dozen Selenium tests with Nightwatch.js, you are done. Your elements have been found, your buttons have been clicked, and your page objects have been modeled. The only thing left is to run your suite of tests across all the browsers you care about. That’s where parallel testing comes in handy. Twelve tests across five platforms mean 60 test cases, but you have a 25-parallel Automation plan, so this shouldn’t take long.

$ nightwatch --env ie,edge,chrome,ff,safari

You press enter and…

Started child process for: ie environment
Started child process for: edge environment
Started child process for: chrome environment
Started child process for: ff environment
Started child process for: safari environment

Huh, that’s funny. It looks like it’s only running five tests at a time, not 20.

Yup, CBT shows that there are only five tests running. What gives?

It turns out that if you use the --env flag to specify the browsers to run on, then Nightwatch will run each environment in parallel, but each test case will execute sequentially.

That’s not what we want! What’s the point of having a big beefy Automation plan if you can’t use it?

So the question becomes, how do we run with MAXIMUM CONCURRENCY?

Well, I couldn’t figure out how to do it in Nightwatch, but I hacked together a workaround!

The following Node.js code will take the --env (or -e) flag and launch an instance of Nightwatch for each specified environment. If no environment is specified, then it will use all environments specified in your Nightwatch configuration file.

parallel-nw.js:

const cbt = require('cbt_tunnels');
const nwConfig = require('./nightwatch')

const spawn = require('child_process').spawn;

const tunnelOptions = {
  'username': nwConfig.test_settings.default.username,
  'authkey': nwConfig.test_settings.default.access_key
}

async function startTunnel(tunnelOptions){
  return new Promise( (resolve, reject) => {
    cbt.start(tunnelOptions, function(err, data) {
      if (err) {
        reject(err)
      } else {
        resolve(data)
      }
    });
  })
}

function envNames(config){
  if(config && config.test_settings){
    return Object.keys(config.test_settings);
  }
  else {
    return [];
  }
}

async function spawnNW(envName){
  return new Promise( (resolve, reject) => {
    let err;
    let proc = spawn('nightwatch', ['-e', envName], {shell:true});
    proc.stdout.on('data', data => {
      if (data.toString('utf8').replace(/\s/g, '') !== ""){
        console.log(`${envName}:\n ${data.toString('utf8')}`)
      }
    })
    proc.stderr.on('data', data => {
      if (data.toString('utf8').replace(/\s/g, '') !== ""){
        console.error(data.toString('utf8'))
      }
    })
    proc.on('close', (code) => {
      if(code === 0){
        console.log(`${envName}: Finished!`)
        resolve(code)
      }
      else {
        reject(new Error(`${envName}: failed ` + err));
      }
    })
  })
}



( async () => {

  // make sure tunnel is started before starting selenium tests
  await startTunnel(tunnelOptions);


  // look for environment names in command line args...
  let names;
  for (let i=0; i < process.argv.length; i++){
    if (process.argv[i] === '--env' || process.argv[i] === '-e'){
      names = process.argv[i+1].split(',');
    }
  }

  // or pull from the config file
  if (names === undefined){
    names = envNames(nwConfig);
  }

  // spawn nightwatch for each environment
  let procs = names.map(spawnNW)

  console.log(`Starting to run Nightwatch on ${names.join(', ')}`)

  // wait for all nightwatch processes to return
  Promise.all(procs)
    .then( () => {
        console.log('Nightwatch finished!')
        // stop tunnel when all tests are done
        cbt.stop()
    })
    .catch( (err) => {console.log(err)})
})()

Once that file is in place, we can use it to run our tests!

node parallel-nw.js --env ie,edge,chrome,ff,safari

Much better!

Nightwatch.js is an End-to-End testing platform for writing and running Selenium tests with Node.js.

The 5 Ways Parallel Testing Can Instantly Improve Your Workflow

April 24, 2018 By Alex McPeak Leave a Comment

You’ve probably heard of parallel testing before, but the ability to multiply your web testing power without multiplying your testing time seems something of a QA fantasy if you haven’t tried it yet. But once you experience parallel testing, you’ll never want to go back to life before it.

While there are probably hundreds of benefits from parallel testing, everyone will gain the value in different ways. In case you need to be convinced (or you need to convince your boss), we rounded up the top five advantages you can expect from parallel testing.

More Browser Coverage – Take one test, run it across multiple browsers at once — that’s the beauty of parallel testing. If you’ve been running tests on different browsers one after the other, or even just skipping browser testing all together, you’ll realize how valuable parallel testing is after your first time trying it. Instead of only testing on one or two browsers, you can substantially increase your cross-browser coverage with parallel testing, hitting configurations you didn’t even realize were an issue. Plus, with CrossBrowserTesting’s broad selection of browsers, operating systems, and devices, you have seemingly endless options for checking the functionality for any and every user.
Less Manual Testing – We’re not saying manual testing is a bad thing — in fact, it’s a pillar of a successful testing strategy. However, when you’re manually performing the same regression tests over and over on different browsers, you’re not making the most of your time. Not only is this boring, tedious work, you’re also risking human error and inaccurate test data by trying to repeat the same tests. Parallel testing allows you to repeat these tests in regression without worrying about skipping a step so you can be confident you’re reaching all the right parts of your application.
Saves Valuable Time – Besides increasing test coverage, parallel also decreases the amount of time you’re actually spending on each test. Since you’re running multiple tests at the same time, rather than one after the other, you’re testing gets done in the fraction of the time it normally takes. In fact, just running 2 tests in parallel doubles your speed, and the more you run in parallel, the more time you save that can be spent elsewhere. This means you can redirect that energy to go back to performing the exploratory tests and looking at new parts of your application at while still meeting your deadlines.
Improving the Development Workflow – Not only does parallel testing make your job easier, it allows for better communication between departments and improves the productivity of your entire team. By allowing for faster feedback and better reporting, parallel testing equips you with the ability to relay results from testing back to your team earlier in the development cycle. This means that parallel testing will be useful in your organization’s effort to shift left and implement Continuous Testing to get iterations to market faster and with fewer bugs.
Go Scriptless – With CrossBrowserTesting, you now have even more ways to run your tests in parallel. With our Selenium grid, you can use your Selenium skills to write tests and run them in parallel. With Record & Replay, you can also achieve parallel testing with zero coding knowledge needed by executing a test manually then rerunning your suites across the browsers your choose. This means teams who are both new to automation and experienced in Selenium can benefit from the capabilities of parallel testing.

We could talk all day about how amazing parallel testing is, but don’t take our word for it. Whether you’ve automated 100 tests or 1, you can easily start running tests in parallel now to experience more code coverage with less time spent on manual regressions.

Try your hand at running a test in two browsers at once then take it to the next level by adding more, and witness pure testing magic right before your eyes.

How CrossBrowserTesting Enhances Your Selenium Suite

March 27, 2018 By Alex McPeak Leave a Comment

Learning how to work Selenium is the key to mastering test automation and writing suites that will provide valuable insights for your team. However, becoming fully acclimated with the ins-and-outs of Selenium takes time, and there will always be certain limitations of what the open source tool can help you do.

While Selenium has its many challenges, you don’t have to go it alone. Cloud testing tools like CrossBrowserTesting provide additional abilities to help make the most of your testing and integrate more automation throughout the software development lifecycle.

Take a look at the following ways that CrossBrowserTesting enhances your Selenium suite (besides, of course, maintaining a Selenium grid infrastructure with over 1,500 real desktop and mobile browsers):

Parallel Testing – Parallel testing allows you to test multiple browser and OS combinations at the same time rather than sequentially. This allows you to radically shorten test execution time by running long cross-browser tests at the same point in time across different environments instead of one after the other. Even just testing two browsers in parallel means that you’re testing twice as much, being two times and productive, and finding double the bugs in the same amount of time — just imagine what you could get accomplished with 5, 10, or 15 tests running in parallel.
Debugging with Recording – CrossBrowserTesting has extended capabilities that not only allow you to automate your Selenium scripts but also to record live tests and debug with inspection tools like FireBug, Chrome Dev Tools, IE Developer Toolbar and more for control over CSS, HTML, and JavaScript. This makes it easier to identify errors and allows you to further investigate what went wrong once a test fails. Once you know what you need to fix, you can do so right in the browser.
Out-of-the-Box Integrations – Automation gives teams the opportunity to insert more testing into the CI/CD workflow. CrossBrowserTesting further enables faster deployments through out-of-the-box integrations like Jenkins and TeamCity, allowing you to improve test coverage and application quality every build. Additionally, integrations like Slack, HipChat, and Jira make it even more simple to integrate CrossBrowserTesting into your current practices and communicate results with your teammates.
Automatic Translation – As discussed, learning Selenium takes a lot of time and patience, but learning Appium presents a whole different set of obstacles. Fortunately, CrossBrowserTesting has automatic Selenium to Appium translation that allows you to easily use the scripts you’ve already written to perform mobile web testing across different devices and browsers. With mobile testing taking center stage as users adopt more devices, you can’t afford to skip testing on a selection of Androids and iPhones. CrossBrowserTesting makes it possible for you to automate across these devices without learning Appium, which means you can just focus on making sure your web app is mobile friendly.

Once you understand how to leverage Selenium tools like CrossBrowserTesting to enhance your automation efforts, your quickly begin to notice how much more testing you can get done.

Familiarizing yourself with these capabilities will make automation more practical for you and more productive for your team as you’re able to increase the speed of testing cycles and spend more time on finding new bugs rather than manually executing tests.

Selenium 101: Running Your Test Against Two Browsers at Once

February 6, 2018 By Alex McPeak Leave a Comment

If you are familiar with automated testing but haven’t yet explored the possibilities of parallel testing, we can tell you right now that you’ve been missing out. Rather than testing sequentially, or running each browser test one after the other, parallel testing allows you to run cross-browser tests simultaneously.

This means if you’re running a test in just two browsers at once, you’re already doubling your speed and productivity. When time is a factor to every software development team, parallel testing is a no-brainer. Selenium is able to run tests in parallel, but CrossBrowserTesting makes it even simpler – allowing you to test across multiple browsers and mobile devices with just a few extra lines of code.

Our example below will be using the Selenium’s Python library, and we’ll run our test against two browsers at once in CrossBrowserTesting with Selenium testing.

First, you want to choose the two browsers you’ll be running the test on. There are a few ways to choose which configuration to test, but the best way to do it for this purpose is to choose the two most common browser versions and operating systems your customers use to visit your web application with Google Analytics.

For this example, we’ll use IE 10 and Chrome 50 on Windows operating systems. Our capabilities will look something like this:

browsers = [
   {“platform”: “Windows 7 64-bit”, “browserName”: “Internet Explorer”,”version”: “10”, “name”: “Python Parallel”}, {“platform”: “Windows 8.1”, “browserName”: “Chrome”, “version”: “50”, “name”: “Python Parallel”},
]

Next, we’ll actually create the test itself, printing out relevant information like browserName to make our debugging experience easier. Our test will simply open CrossBrowserTesting.com and just wait for 3 seconds.

browsers_waiting = [ ]

def get_browser_and_wait(browser_data):
   print ("starting %s\n" % browser_data["browserName"])
   browser = get_browser(browser_data)
   browser.get("http://crossbrowsertesting.com")
   browsers_waiting.append({"data": browser_data, "driver": browser})
   print ("%s ready" % browser_data["browserName"])
   while len(browsers_waiting) < len(browsers):
     print ("working on %s.... please wait" % browser_data["browserName"])
browser.get("http://crossbrowsertesting.com")
     time.sleep(3)

For parallel testing, we’re going to use something called thread dependency, which you’ll see below. Multithreading is the implementation of parallel execution that communicates multiple processes on the CPU.

threads = []
for i, browser in enumerate(browsers):
   thread = Thread(target=get_browser_and_wait, args=[browser])
   threads.append(thread)
   thread.start()

for thread in threads:
   thread.join()

print ("all browsers ready")
for i, b in enumerate(browsers_waiting):
   print ("browser %s's title: %s" % (b["data"]["name"], b["driver"].title))
   b["driver"].quit()

Parallel testing becomes really handy when trying to increase the speed of your testing or actual testing coverage without sacrificing quality. Running a test against two browsers is a really great place to start with this script and already allows you to test twice as fast as doing the same testing sequentially on different browsers.

Once you begin using this method of testing on multiple browsers to extend the reach of longer test cases, the value will be evident. A test suite that previously ran for 5 hours can now be affordably decreased to around 15 minutes, giving your development and testing teams quicker feedback. You can also take your testing suite and boost your coverage immediately, running it against the browsers and devices your customers are actually using. If you extend this over day, weeks, and months, you’ll be benefiting from hours on top of hours of productivity.

For more of our Selenium 101 Series, take a look at the following:

To see more examples in other languages:

6 Ways to Speed Up Your Testing Cycles

November 16, 2017 By Alex McPeak Leave a Comment

In the battle between speed vs quality, teams that practice test automation know that there’s a practical way to aim for both. However, while testing inherently places value on precise observation and careful consideration over making mistakes for the benefit of speed, how can you make testing rapid enough the meet the needs of Agile development demands without skipping over essential steps?

Faster testing is easily attainable by refining your current automation strategy and adjusting the processes that are already in place. Not only will the following tips make your life easier, they’ll also accelerate your testing to help make the most out of your time.

Get organized – Keeping your tests organized may not seem like a priority day-to-day, but it will make things much more streamlined for you in the future. You should be keeping your test cases lean and reusable, automating features at the unit level. By practicing good test management methods, maintaining accurate data, and keeping clean automation reports, it’ll be easier for you and your team to run tests whenever they’re needed.
Talk to your teammates – Speaking of team, your teammates are some of your most important tools to driving testing that’s effective and efficient. Whether you’re the solo tester on a software team mostly made up of developers or you’re one of fifty other automation engineers, there are always other stakeholders in the software. Communicating with these people will help everyone stay on the same page as far as documentation and reporting, as well of the status of the software included what’s new, what needs to be tested, what’s been tested, and what needs to be fixed.
Know when to automate – Of course, automating your tests will make things exponentially easier and faster, but automating the wrong test cases is just going to slow your progress down. While testing manually might seem to take longer, new test cases require exploratory testing, and taking the time to do it at the beginning will save you from having to go back and do it later after you’ve already tried to automate it. Ultimately, understanding the best cases to automate before you actually start running them will prevent you from wasting time — a software tester’s worst nightmare.
Try parallel testing – While automation will unquestionably help you test more parts of your application faster, it falls short in the way that it is only able to accomplish this in a sequential fashion — no more than one automated test can generally be run at one time. However, with parallel testing, automation gets a much-needed boost by allowing different tests to run simultaneously. If you’re smart enough to understand the importance of the same tests across multiple browsers, operating systems, and devices, you can also see the value in doing this all at once with parallel testing instead of running the script over and over on different configurations.
Use tools to your advantage – Being a great tester will only take you so far if you don’t know how to leverage the right tools to help you. Between adopting open source testing frameworks and investing in automation tools, there’s no shortage of ways to make your life a little easier. With the previously mentioned parallel testing, for example, you need to rely on a tool like CrossBrowserTesting for a cloud-based Selenium grid or Record & Replay.
Cut out flakiness – While tools like Selenium are an asset in automation, it can also lead to flakiness. False negatives will just slow you down — no one likes seeing red all the time and dealing with inconsistent results. In order to avoid this, it’s best to attack the problem head-on by isolating flaky Selenium tests from the build to fix them before rerunning the test again.

Conclusion

Working towards faster test automation doesn’t have to mean starting over. Balancing speed and quality is something you just have to learn along the way, but getting to know test automation hacks will get you there even sooner.

Upcoming Webinar: Accelerating Your Test Execution Pipeline

August 30, 2017 By Alex McPeak Leave a Comment

Do you find your testing process can’t keep up with the demands and expectations of quickening software development and delivery cycles? Do you mark tests as “done” just to discover certain users are still finding bugs? Maybe, you wouldn’t mind finding a solution that would cut your testing time and increase quality, without having to choose just one or the other.

On September 7th, SmartBear will be hosting a free webinar entitled “Accelerating Your Test Execution Pipeline” in two sessions (8:30 AM and 2:00 PM EDT) to talk about just that. Gabriella Grandgard and Daniel Giordano will discuss ways to identify key bottlenecks in your testing processes that will help to deliver software on-time without compromising quality.

As test automation processes increase among organizations, testing efforts can lag behind largely because of scalability issues. Test environments are often the root cause for a number of these problems due to the time it takes to create them and maintain them, and the cost to reflect real users. Incorporating a test automation platform can cut your testing time in half by leveraging the cloud to test multiple devices and browsers at one time, and fix bugs as fast as they’re found rather than waiting until after deployment.

This method referred to as parallel testing allows testers to run scripts in multiple environments concurrently to save time, increase coverage, and speed up tests. This means that instead of running one script sequentially across different device and browser configurations, it can be done at the same point in time. As browsers, devices, and operating systems continue to release and update, keeping up with these versions calls upon the need for both parallel testing and cloud testing to support scalability.

Join our software experts to learn how testing teams can accelerate their testing efforts by easily scaling and reporting on their tests across multiple environments using parallel testing techniques.

Highlights include:

Real world demonstration of testing across multiple environments.
Techniques that can help you easily speed up your testing efforts.
Strategies to enable parallel testing for both manual and automated testing processes.
New ways to quickly pinpoint problem areas in your tests.

Register for the free webinar here.

What is Regression Testing?

June 22, 2017 By Alex McPeak Leave a Comment

When a developer goes to make adjustments in their software, the smallest change to code can disrupt previously flawless functionality. You know it worked before, so if it doesn’t work now, it must be something we introduced with our latest push, right?

Regression testing is the process used to ensure that a recent code change hasn’t impacted these features that were already working, and is also used to monitor and maintain production software that has already passed testing.

Not only do you want to make sure everything looks good visually, but you also have to make sure that it performs quickly and functions without bugs. By re-executing these test cases, it’s easier to evaluate whether everything still works the way it’s expected.

When thinking about regression testing, there are three types including unit, partial, and complete. Unit regression tests are used to test a single unit of code, partial regression tests are used for code modification, and complete regression is for on-going code changes.

While many testers acknowledge the importance of comprehensive regression testing, it may be difficult for a team to agree on the best method for its execution.

The Problem with Regression Testing

While regression testing is a necessary part of QA, it’s probably not your tester’s favorite part of the job. In fact, the repetition involved with manual regression testing can be really tedious after running cases the first few times.

While you’ll likely have to run the first regression test manually, executing regression tests manually is a huge waste of time and energy over an extended period of time. And it’s really boring to repeat those same actions over and over – really boring. Manually running regression tests is an open window for human error, since it’s difficult to perform the actions exactly the same over and over.

In addition, the best QA teams know that an accurate regression test has to be done across more than one browser to ensure functionality across different configurations, increasing the time and energy involved in the process multiple times over.

Fortunately, testers that take advantage of automated testing can program those repetitive regression test cases to run in the background on a nightly basis, or with every deployment. By using tools likes Selenium, Appium, Record & Replay, or other testing tools and frameworks it becomes much easier to run regression tests, while your testers have more time to focus on their other priorities.

Additionally, automating parallel tests in Selenium takes this a step further, allowing you to approach cross-browser testing by running separate tests at the same time.

To read more about parallel testing with Selenium, read this resource.

Should I Automate All of My Tests?

No. Automated testing will only assert whether the actions you tell it to are true or false, which is why it’s great for regression tests — you’re going back to make sure nothing has changed over time.

However, automated tests cannot and will not find new bugs in unexplored areas of software. As your system scales, so must your exploratory and unit tests, as well as your regression tests. And, as we mentioned, your regression test cases will have to stem from manual test execution.

Once a test case is stable, it’s a good idea to perform regression tests on them in the future, but they will also need maintenance and validation as the system changes over time. You’ll probably have to create new regression test cases to compliment this change as well.

Additionally, a great benefit of automated regression testing is that it frees up a lot of time that can be spent manually testing other unexplored areas. You don’t want to use automation to take over testing completely, but to aid testing efforts that need to be maintained more than they need to be inspected.

Regression Testing Today

While regression testing is not a new term in software testing, new test management practices, tools, and methodologies have changed the way testers approach these tasks.

As organizations become more Agile and begin to continuously integrate new code, it’s important that those changes are verified and approved before new features and functionalities are delivered to end-users.

In fact, the industry shift from Waterfall to Agile has actually made more regression testing possible more often.

Since Agile teams have also adopted the idea of testing early and often, it’s allowed smaller regression tests cases to be performed intermittently rather than running long case at the end of development.

Instead of re-doing an entire test suite, running regression test cases as needed saves a lot of time a creating a baseline of quality. Additionally, regression testing becomes a crucial part of reducing risk and eliminating bugs before they reach the customer.

Luckily because of the high demand for regression testing, there are many tools that strive to make this easier. Whether you’re parallel testing or using a visual regression tool to evaluate inconsistencies, there’s no excuse to skip out on regression tests, and as your organization grows, you won’t want to.

By crafting a testing strategy, prioritizing automation, and using the right tools, you can utilize regression testing in the best interest of your software development and QA teams.

How much have you experimented with regression testing? What are the best ways you’ve found to streamline the process? Tell us your experience in the comments!

9 Signs You Should be QA Testing in the Cloud

June 7, 2017 By Alex McPeak Leave a Comment

We’re all familiar with the cloud. As a topic that’s clearly trending and growing in software, our appreciation of the cloud testing stems from its innovation as well as its effectiveness.

The cloud is only projected to become more popular and more common in years to come. In fact, 95 percent of organizations are using a cloud in some way, and the market is supposed to increase 22 percent annually according to SD Times.

In software development, cloud testing provides an easy-access, remote testing environment to produce real user scenarios. As organizations consider the benefits of the cloud testing, here are some signs that tell you it’s time to use it for browser testing.

You’re starting a business – One of the best times to check your web design across browsers are in the early stages a.k.a the exploratory tests. New companies should ensure they establish the UI of their website to look and function responsively across multiple devices when quality can make or break a business reputation.
You’re moving from manual to automation – Cloud testing offers an easy onboarding for organizations whose testers are a little newer to programming and automation. As you continue to recognize the benefits of automating your tests, the cloud is a practical way to make the process pain-free. With on-demand access, it’s easy for your team to test whenever they need and learn as they go. Additionally, browser screenshots, extensions, frameworks, debugging tools, and integrations are included to optimize usability and make testing more enjoyable. Plus, customizable preferences and customer-centric user interface assure an optimal user experience.
You’re realizing there aren’t enough hours a day to test everything – If you find yourself with test cases that are taking more time than there are hours in the day, it might be time to think about adding some more machines to your parallel testing. Parallel testing is a feature exclusive to cloud testing, which will allow you to significantly cut down your testing time by testing different cases on multiple machines at the same time, instead of one after the other. Additionally, the cloud is available all day every day, so you can test whenever you want without having to be on-site. In fact, you can even run automated tests in your sleep. This will make it easier to test all the configurations you need to, while still meeting deadlines.
You have a hard time keeping up with all the tech – Software testing is not what it used to be. With so many new browsers, OS versions, and devices coming out all the time, it can be difficult to keep track of all the configurations you need to test on. With cloud testing, new releases are added for you so you don’t even have to think about it.
You think new smartphones are way overpriced – If your device lab budget isn’t quite allowing all the configurations you need, cloud testing might be just the right answer. When an average Android is about $670 and an iPhone rounds out at around $700, the numbers can add up fast when you try to test on every device your user has — not to mention maintenance.
You recognize the benefits of simulators, emulators, and real devices – While emulators and simulators have their place in testing, real devices are crucial in order to test on the machines that your users are on. To get the most comprehensive breadth of simulators and emulators plus real devices for testing, cloud testing is the best way to access all three for different purposes, and the combination ensures that you always have access to testing.
You’re scaling your team – You had QA all figured out when you had just one (albeit really talented) tester. But, as your company scales, so must your testing team. When everyone’s trying to complete different tasks that all seem to be of top priority at the same time, the cloud gives you more room to accommodate growing pains. Since everyone has access to the same devices at once, you can ensure you’re supplying your growing team with the resources they require.
You have testers in different offices – Similarly, it’s a little bit difficult to share the capabilities of your device lab among a QA team that’s spread out among different cities, countries, and continents. Cloud testing provides one platform that further encourages communication between different departments and remote teams with integrations like Slack sharing.
Your customers are getting demanding – Has your customer support team been extra stressed out lately? Are you getting user reports saying that your website is slow, or there’s a bug, or it’s not working on their browser? Your customers have high expectations of you. They might be using a wider variety of devices than you think, and it’s compromising their interaction with your web application.

What pushed you to make the decision to adopt a cloud testing service? Tell us in the comments!