CrossBrowserTesting.com

A Design, Development, Testing Blog

The Junction of Browser Testing and Test Management

By Leave a Comment

Software development – and subsequently software testing – is happening at speeds which we’ve never seen before. With many tests being set up and deployed automatically in order to keep up with pace and maintain quality and diligence throughout the testing process, keeping track of those tests has become even more paramount.  

This is the junction of browser (and app) testing and test management. Test management is crucial to ensure collaboration and traceability throughout the testing lifecycle, especially given the increase in volume and complexity of these automated tests.  

Unifying test automation and test management capabilities can be done using SmartBear’s industry leading test automation software CrossBrowserTesting, and Zephyr for Jira, the most widely used test management tool to work natively inside Jira.  

The CrossBrowserTesting and Zephyr for Jira integration allows teams to easily manage all of their manual and automated tests. Users and testers can quickly navigate to the respective Jira card to view results of each test, analyze pictures and video recordings of the test, and add any notes that may have come up — all within that native Jira experience.  

By including test management in your Agile process, you’ll have much more control and visibility into your team’s efforts. During Sprint and Release meetings, you can make sure tests are properly executed, while giving your testers the ability to quickly change or re-prioritize tests accordingly.  

Want to learn more about where browser testing fits in the testing pyramid and how to include test management in your Agile process, both with the power and support of award-winning SmartBear tools? Check out our On-demand session to learn more about both tools and how they work together!

Ensuring Mobile Excellence with SmartBear Recap

By Leave a Comment

Digital transformation has been embraced by most organizations.  More than ever, people turn to mobile devices to stay connected with the world and each other, and that includes engaging with businesses.  With this expansion of users on all devices and platform, businesses are striving for digital excellence while they manage transformation and product roll out.  According to Hackermoon, internet usage has increased by as much as 70% as organizations adopt a more distributed workforce model with working from remote locations becoming the norm and usage on mobile devices growing quickly.   

As a result, it is more important than ever for apps and websites to operate at peak performance and deliver a seamless and satisfying user experience. If a user comes across a bug on your site or app or isn’t getting the experience they’re looking for, they’re likely to leave you without taking your desired action.

  • 88% of online shoppers say they won’t return to a website after having a bad user experience 
  • 44% of shoppers will tell their friends about a bad online experience 

With so much competition for a user’s engagement, time, and dollars, less than ideal experiences are risks you simply can’t afford, emphasizing the importance of testing to ensure your mobile app delivers what it’s supposed to will help you avert these bad experiences. For your developers and QA teams, there are a variety of testing execution challenges that they must overcome in order to contribute to delivering the seamless app or website experience your customers have come to expect. 

Here is a short list: 

  • Browser and device coverage – This is ultimately the most important aspect; after all, user come in all shapes and sizes and have thousands of different mobile devices, operating systems, or browsers. How can you be sure your app or website looks good on all of them? By testing on all of them! That means having access to robust device lab that contains a multitude of different device and browser options, unfortunately, maintaining this infrastructure is expensive.  
  • Ensuring your checkout process is fully operational – Converting a customer is difficult enough as it is. If that would-be customer has any bumps on the road on their way to checking out, they can change their mind – cart abandonment is a major problem for any ecommerce retailer.  
  • How do things look? – An engaging visual experience is so important for the savvy mobile and internet user. For a dynamic website that changes daily or more regularly, your testers need to be able to track those changes to make sure everything looks as good as it should be.  

Those are three not insignificant challenges for testers and software developers to deal with. Whether they’re doing live-testing manually or setting up automated tests through various frameworks, these are real challenges that could prevent them from engaging an effective testing process.  

Fortunately, two SmartBear tools CrossBrowserTesting and Bitbar are designed to address exactly those challenges facing testers. If you’re a software developer or QA tester looking to uplevel your manual or automated tests, work more efficiently, and gain access to a huge access of coverage in terms of browsers and devices through the SmartBear Real Device Cloud, these tools can significantly improve your testing capabilities. 

You’ve seen the critical business impact that having a poor mobile app or website experience can have, and how that business impact has only accelerated with the digital transformation spurred on by testing. Now see how these SmartBear tools can take your testing capabilities to the next level and ensure that your customers are always getting the experience you want them to have.  

Check out our webinar recording to see a demo of these tools in action, or sign up for a free trial of CrossBrowserTesting and BitBar to take them for a spin yourself!  

How to Successfully Automate Your Cross-Browser Tests

By Leave a Comment

Testing your software or application before releasing it into the wild isn’t just a nice-to-have. Developers and QA teams spend hours making sure their company releases high-quality products. However, testing practices and requirements for web apps have changed in recent years, and new complexities require efficient solutions, such as test automation. So let’s talk more about automating your cross-browser tests, what to automate and what not, and which tools are best suited for the job.

What is test automation, and how can you benefit from it?

When Internet Explorer 6 was the most common browser, hardly anyone used Google Chrome, smartphones, or tablets. Developers only had a few browsers of which they had to test their website on, and there was usually nothing more to do – testing was not as demanding as it is now. Also, the products built back then weren’t as sophisticated as what you can find online today. E-commerce sites, online banking, complicated UI structures, and animations with CSS required developers to step up their testing game. Add all the different smartphones, tablets, different browsers, and their versions into the mix, and you can see, cross-browser testing is much more critical these days.

If our testing practices looked the same as they did in the IE 6 era, we would likely publish bug-infested web apps that most people wouldn’t even look at. Of course, by now, we all know how necessary thorough testing is and that we need to run various types of tests (e.g., functional tests, performance tests, security tests, etc.) to ensure that our app works and looks perfectly. Yet, testing can be time-consuming – the more tests you need to run, the longer it will take to release your app. And that’s where test automation comes in to save the day.

Test automation helps you save time by letting developers configure test cases and have them run automatically by a tool. If you’re cross-browser testing, instead of manually interacting with your web application, you can prepare a script, define and configure the browsers you want to test, and then have a browser automation tool do everything for you.

However, saving time isn’t the only benefit you can enjoy from automating your cross-browser tests! Read on to discover more.

Know what to automate and what not

You have probably heard that manual testing leads to bottlenecks, worsens time-to-market, and possibly bores your test teams out of their minds. At the moment, though, we can hardly execute 100% of our tests automatically (although that would be nice). For example, User Experience testing requires a human tester, other tests may be completed faster manually, etc.

Regardless of which tools you use for cross-browser testing, you and your team must define what should and should not be automated, to be able to achieve the desired results efficiently.

In general, these types of tests can be automated:

  • Tests that you expect to get a specific result, e.g., unit testing
  • Repetitive tests that have to be carried out frequently, e.g., login (authentication)
  • Tests that you have to run with different options and parameters, e.g., a contact form with mandatory fields, email address verification, etc.

Yet, manual testing can be a better choice for:

  • Tests that get flaky due to automation. For example, if your test has to wait for input from a third-party source that doesn’t always deliver on time and therefore can’t produce consistent results, you’d be better off running the test manually.
  • Tests that are too complicated or computationally intensive, but can be carried out quickly by a human.
  • Everything that protects your web app or itself from automation, such as captchas. It would make no sense if you could test and pass a captcha automatically, right?
  • Tests based on a user’s subjective opinion, such as UX tests and any other types of analysis that goes more in the look-and-feel direction of your app.

After talking about what can be automated, we need to talk about which tool to use. One of the most popular and useful frameworks for cross-browser test automation is Selenium – let’s find out what it is and how it can help you.

Use Selenium to automate your cross-browser tests

Selenium is a robust framework that consists of various tools and was initially developed by Jason Huggins in 2004. Allegedly, the name comes from one of his emails, in which he made fun of a competitor named Mercury by writing that selenium mineral supplements can cure mercury poisoning.

The part of the framework we’re most interested in now is Selenium WebDriver, which is used to automate cross-browser testing. WebDriver accepts commands and sends them to a browser to execute them (hence the name). Teams can automate tests for their web applications by instructing WebDriver to perform specific actions in any browser and device. It’s also worth noting that the WebDriver specification made it into the W3C recommendations in June 2018, further increasing the tool’s popularity.

Selenium helps you and your team save valuable time that you can spend on other tasks that still require a human’s full attention, such as analyzing test results and performing manual tests. However, before you blindly start automating your test cases with Selenium, you need to determine which tests can and cannot be automated. To do this, you can use our lists from above to get started. If you want to learn more about test automation with Selenium, be sure to check out our Selenium Toolkit!

Run your automated tests in parallel

Automating your Selenium tests is an essential step in improving test execution times, but you can do more! The real magic unfolds when you run your automated tests in parallel on multiple browsers and/or devices. Let’s say a test case takes about 10 minutes to execute, and you need to test 10 different browsers and devices. If you check one browser and device combo at a time, it will take about 100 minutes to finish. However, if you run the script in all required environments at once, you’ll be finished in about 10 minutes!

Parallel tests enable you to test your web app on more devices and browsers in the shortest possible time. For this, you can use Selenium Grid, which allows you to execute your Selenium tests on multiple environments at once. But, the right infrastructure is required for Selenium Grid because it is based on a hub-node concept. This means that you need one machine – the hub – to host the tests, and several machines – the nodes – to execute tests on different browsers, browser versions, etc.

Make sure you have the resources to set up and maintain the necessary infrastructure if you want to run Selenium Grid on-premises. If you don’t want to manage the infrastructure yourself and have additional requirements, e.g., running your tests on real devices as opposed to simulators, you can try a web-based test service.

Conclusion

Selenium is the best choice if you want to automate your cross-browser tests and make them future-proof. Before you start automating, however, you should spend some time thinking about your product and its features to find out what you can and should automate, and what needs to be tested manually.

Make sure you run your automated tests in parallel for the best possible efficiency. Run hundreds of tests in parallel on thousands of real desktop and mobile browsers with our optimized test infrastructure in the cloud. Start your free trial and take advantage of executing your automated tests in parallel on real devices and browsers.

Blueprint for Automation using Selenium

By Leave a Comment

Getting started with Selenium can be a tough task!  But to help you build a solid foundation, you’ll need to understand the ins and outs of Selenium, how it works, and some tips on how to get started.

What is Selenium?

The Selenium tool set consists of the Selenium IDE, Selenium Grid, and the Selenium WebDriver. Selenium is an open source tool designed to create browser-based automated suites and tests for web applications across environments.

What Are the Limitations of Selenium?

Before you get started, it is important to understand what Selenium can and cannot do when automating your tests.

Selenium is designed for testing your web applications in a variety of browsers. The Selenium WebDriver can only manipulate web pages inside of the browser.

What Kind of Skills Will You Need?

Once you’ve settled on using Selenium, you will need to choose a programming language. Selenium supports many different language bindings including Java, JavaScript, PHP, Ruby, C#, and Python, allowing you to write your tests in the language of your choice. Each of these languages provide several frameworks that integrate with Selenium to make testing easier.

Selenium does offer an IDE that requires minimal coding, however we will focus on the Selenium WebDriver and its programming language bindings.

Choosing a Testing Framework

As mentioned, there are many testing frameworks that you could work with to get started with Selenium, but we will go over two common types of testing frameworks as a starting point.

BDD (Behavior Driven Development)

BDD has become increasingly popular over the past couple of years in helping agile development teams be more collaborative among developers, QA and non-technical or business participants in software projects.

With BDD testing, it’s Gherkin language allows you to write your tests in a way that can be easily read by anyone on your team. Creating simple scenarios to test your application’s behavior from the end users’ perspective that allow a shared understanding of user requirements, making test automation easier and faster. These frameworks read as plain text specifications and can generate a report, validating success or failure.

Some examples of BDD frameworks are Cucumber, Behave, and Jasmine. Check them out!

Unit

Another type of testing framework is Unit Testing frameworks. Unit testing frameworks allow you to test individual components of your site with the objective of isolating each section and verifying its correctness. These frameworks allow the use of pre-defined assertion methods to help with this verification process.

Some examples of unit testing frameworks are  TestNG, JUnit, NUnit and PHPUnit, Check them out!

Get Started Testing

Now that you have a good blue print on what makes up the Selenium stack, benefits of using Selenium, along with questions to ask along the way as you try to implement this tool into your work flow, it’s time to get started!  Since Selenium is an open source tool, you can download it and try it out at SeleniumHQ.

For all listed languages and frameworks mentioned above, you can find detailed documentation with examples to help you get started at the CrossBrowserTesting Help site!

6 Key Factors That Improve Cross Browser Testing

By Leave a Comment

Our friends over at Bitbar recently wrote a fantastic blog about 6 key factors that can improve your cross browser testing. Take a look below!

If you start thinking about testing after the completion of your web application, you’re already too late. The decisions you make before and during development impact and shape testing more than one might think. And not involving testing from the start may lead to some severe problems that can prevent a timely release. In particular, cross browser tests can be a bottleneck. That’s why we’ve put together a list of 6 essential factors that can help you improve cross browser testing and plan ahead!

1. New Isn’t Necessarily Better

Web technology advances swiftly, and innovative features are being introduced with each new browser version. We know the urge to use fancy new tech in a project can be massive. But you have to keep in mind that new doesn’t automatically mean better! Some of these functionalities are still in beta, experimental, or not yet supported in all major browsers. As a result, you deliberately introduce cross browser variations into your product and face a more difficult time testing.

Our advice:
If using a particular new technology that brings no real value to your users, think twice about integrating it into your app. Avoid unnecessary cross browser variations in your web application.

2. Stop Reinventing the Wheel

When your team plans the technology stack for your next web-based project, see if you can use existing frameworks, libraries, and so on. Use React, Vue.js, or Angular, and don’t depend on Vanilla JS in your app too much. You not only save a considerable amount of time in development but also during testing! The same is true for CSS frameworks like Bootstrap or Foundation. They are already tested on a wide range of browsers, and devices, saving you time testing your app.

Our advice:
Go with established frameworks and libraries to write your CSS and JavaScript code. They are usually thoroughly tested, and you can safely use them in your project. That way, you reduce the code you need to write and validate. Additionally, you can reuse already tested components, further reducing testing time.

3. Only Test Clean Code

Have your developers follow coding and style guidelines and let them deliver clean, consistent code. As a result, your code is less error-prone, and bugs can be found more efficiently. Additionally, anyone in the team can work on the source code regardless of who wrote it. If you don’t have guidelines or don’t have the resources to create them, use existing ones! Why not use style guidelines from other companies such as the Airbnb React/JSX Style Guide?

Our advice:
Using and enforcing coding and style guidelines makes developing and testing your app more straightforward. Ensure your team delivers clean code by having your developers install code linters in their IDEs. These linters should automatically check the written code against your guidelines and find minor errors, like spelling mistakes or missing brackets.
As a result, cleaner code – without the usual minor errors – arrives at your testers without investing too much extra time!

4. Make Use of Automation

Test automation can help you shorten testing time significantly so you can release updates and bug-fixes more often. Although automating cross browser testing can be a bit tricky in some places, we can’t not take notice of the benefits. Automating tests also mean that your devs and testers can eliminate repetitive and tedious tasks from their to do list. This gives them more time to create new features and perform necessary manual tests.

Our advice:
First, see what you can automate and which parts or browsers you should manually test. Here are some ideas to get you started:

  • Follow TDD for your JavaScript code. Write and automate unit tests by using a framework like Mocha.
  • Use browser automation and inspection tools like Selenium WebDriver to automate your end-to-end tests.
  • Take a look at the different tools in the Selenium suite that help you test your apps. Learn more about the suite in our blog article series about Selenium.
  • Make sure your app looks immaculate on different screen resolutions by automating image regression testing. Cloud-based services like CrossbrowserTesting provide the functionality you need.

5. Test on a Large Scale

Automating your tests is nearly worthless if you don’t run them on real browsers and devices, preferably at the same time. Sure, test automation saves time, but if you have to run these tests on one device and one browser at a time, the process takes forever. Also, keep in mind that your users are real people who use real computers, smartphones, tablets, and browsers. Can you imagine anyone using emulators to browse the internet? Consequently, make sure to run your automated tests on multiple real-world devices and browsers in parallel. This lets you test more efficiently so you can spend more time on other things.

Our advice:
Choose a cloud-based testing service, that lets you run tests on multiple real-world devices and browsers simultaneously. As a result, you can run more tests in less time while ensuring your app works as intended in a variety of real-world scenarios. In the end, you can release new features to your customers with more confidence.

6. Bring Continuous Integration into Play

If you’ve integrated all previous steps into your workflow to improve cross browser testing, then Continuous Integration (CI) is the icing on the cake. If you are unfamiliar with CI, be sure to read some information on this topic, as you will undoubtedly benefit greatly.

In a nutshell, CI works as follows: Whenever a developer pushes code into the project’s central repository, several jobs start running. Usually, these jobs perform a variety of tests. In the end, CI helps you determine whether the committed code is releasable – only commits that pass the validation get accepted.

Our advice:
Using CI does not require your developers to start the tests manually, so no test can be forgotten. Additionally, only perfectly valid code is accepted for release. Not only does CI automate repetitive processes, but it also makes sure your app performs at its best. You can find many different CI tools that meet your specific needs. For example, use Jenkins if you want to run jobs on a dedicated server. Or use TravisCI if you wish to use a cloud service instead. However, many, but not all, cross browser tests, can be integrated into a CI job. Consequently, you need to pick the service that’s most compatible with your other tools! Check out the integrations CrossBrowserTesting supports!

Wrapping up – Improve Cross Browser Testing by Using the Right Tools

Test early and often, use tools that fit your needs, and make sure you run automated tests on real devices as often as possible! Use different tools and services that match your situations and needs. Don’t hesitate to use a cloud-based service to make your test processes more efficient and resource-saving.

Why Should I Run My Selenium Tests in Headless?

By Leave a Comment

Headless testing is greatly underutilized – here are three quick ways it can increase performance

Selenium has grown to be one of the most popular automation tools available today. It automates many tasks needed for online testing, including checking titles, navigating pages, clicking links, and much more. Many developers already use the full set of capabilities Selenium offers, but few know about running their tests in headless browsers. And even fewer actually deploy this approach in their daily testing. With headless testing you can increase your testing performance to an all-time high.

What is Headless testing?

Headless testing is simply running your Selenium tests using a headless browser. It operates as your typical browser would, but without a user interface, making it excellent for automated testing.

How does Headless testing benefit developers?

There are several benefits, actually. They can create new testing opportunities, as well as accelerate tests you’re already running.

Benefits include:

  • Greater testing reach
  • Improved speed and performance
  • Multitasking

Greater testing reach

When running Selenium tests, you typically need a machine that supports the graphics of the web browser that you’re testing on. With headless testing we get rid of this need and open up a whole new set of devices to test on. Ex. Servers, docker containers, etc.

Improved speed and performance

Selenium tests can take a while to complete, due to elements on a page that the browser needs to load. Headless testing gets rid of this load time, allowing you to cut your testing times significantly. In our tests with headless testing, we’ve seen a 30% reduction of test execution times. And you can use other techniques, like running more tests in parallel, to amplify that benefit.

Multitasking

Running normal Selenium tests take up your screen time, keeping you from being able to accomplish anything else on that device. With the UI disabled, headless testing lets you continue to use your computer while the tests execute in the background.

In our conversations with customers, we’ve found that headless testing has let developers experience all of these benefits – and many more!

How have you used headless testing to improve your processes? And where do you see opportunities to take advantage of them? Respond in the comments!

To help you get started Headless testing with CrossBrowserTesting, visit our help page: Headless Testing Documentation

Are These 5 Gaps Missing From Your UI Test Automation Strategy?

By Leave a Comment

UI test automation

As more and more organizations integrate test automation into their workflows, teams have benefitted from shortened feedback loops and less time between development and delivery.

However, while automation has been an asset for effective and efficient testing, many of us are still getting acquainted with the skills required to build out tests that give us maximum value for our time.

What many don’t realize is that there could be gaps in your test automation strategy. These common oversights could be the difference between valuable automation suites and inadequate test results.

Recently, SmartBear hosted a webinar to address a few of these gaps and their solutions. Here are the top insights.

  1. Spending too much time maintaining test scripts –  If you’re putting too much time, effort, and energy into creating test scripts, it may be time to reconsider your approach. Often this can be caused by improper planning or poor structuring of testing frameworks. However, by making tests more reusable, maintenance time can be reduced significantly. Rather than making large tests in an attempt to cover as much of your application as possible, break them up into smaller tests cases. You can then use these tests to build out larger, more comprehensive test cases. This way if the UI or the user flow changes, it’ll be easier to go back and use your components.
  2. Zero or little mobile browser testing – In 2018, half of website traffic came from mobile devices, yet companies still tend to neglect testing on mobile. Just having a mobile version of your web application is not enough, you need to verify that it also works correctly on different devices. Without testing on real devices and tablets, you have no idea what your users are experiencing every time they visit your website on a smaller screen. This is not something you want to leave up to chance as mobile web usage continues to increase.
  3. Lack of environment options – Even the best intentions to increase test coverage through automation can fall short without a substantial number of environment options to test on. Different environments can render code in different ways, which means it’s crucial to look at your website on different browsers and devices. Having limited environment options can lead to issues going unnoticed. If you’re only testing one or two browsers and devices, you’re not going to know where there are issues in the others your customers are using. You don’t want preventable bugs slipping through to customers when the fix is as easy as increasing the number of environments you test on.
  4. Spending too much time and money maintaining devices – Some may assume that the best answer to a lack of environment options is to build a device lab. In theory, this seems like a good idea — you’re team has 24/7, onsite access to all the machines and configurations they could possibly need. However, the reality is that these devices are expensive to buy and maintain. Tens of new mobile devices are released every year, meanwhile, existing devices must be repaired and maintained. Not only is the time commitment not practical for most organizations, but it’s also expensive. Most of the time, it’s more trouble than it’s worth to build out a device lab when there are expansive cloud options available.
  5. Lack of parallel testing – Parallel testing is your key to increasing test coverage without compromising time. The issue? This practice is often underutilized by teams. While thinking about the time it takes to test in all these environments might be overwhelming, parallel testing allows you to significantly cut down on the hours it would normally take to run one test after the other. By testing multiple configurations simultaneously, you can condense 30 hours of testing on three different machines down to 10. In this way, parallel testing will reduce the time spent running tests, while still allowing you to hit the environments your users are on.

While automation is no easy task, there are steps we can all take to improve efficiency. Bridging these five gaps in your test automation strategy with the help of tools like TestComplete and CrossBrowserTesting will allow you to get more value out your time.

If you’d like to learn more about these common UI test automation gaps, check out the full webinar below.

Maximum Concurrency with Nightwatch.js

By Leave a Comment

Nightwatch.js parallel testing

You made it. After weeks writing a dozen Selenium tests with Nightwatch.js, you are done. Your elements have been found, your buttons have been clicked, and your page objects have been modeled. The only thing left is to run your suite of tests across all the browsers you care about. That’s where parallel testing comes in handy. Twelve tests across five platforms mean 60 test cases, but you have a 25-parallel Automation plan, so this shouldn’t take long.

$ nightwatch --env ie,edge,chrome,ff,safari

You press enter and…

Started child process for: ie environment
Started child process for: edge environment
Started child process for: chrome environment
Started child process for: ff environment
Started child process for: safari environment

Huh, that’s funny. It looks like it’s only running five tests at a time, not 20.

Yup, CBT shows that there are only five tests running. What gives?

It turns out that if you use the --env flag to specify the browsers to run on, then Nightwatch will run each environment in parallel, but each test case will execute sequentially.

That’s not what we want! What’s the point of having a big beefy Automation plan if you can’t use it?

So the question becomes, how do we run with MAXIMUM CONCURRENCY?

Well, I couldn’t figure out how to do it in Nightwatch, but I hacked together a workaround!

The following Node.js code will take the --env (or -e) flag and launch an instance of Nightwatch for each specified environment. If no environment is specified, then it will use all environments specified in your Nightwatch configuration file.

parallel-nw.js:

const cbt = require('cbt_tunnels');
const nwConfig = require('./nightwatch')

const spawn = require('child_process').spawn;

const tunnelOptions = {
  'username': nwConfig.test_settings.default.username,
  'authkey': nwConfig.test_settings.default.access_key
}

async function startTunnel(tunnelOptions){
  return new Promise( (resolve, reject) => {
    cbt.start(tunnelOptions, function(err, data) {
      if (err) {
        reject(err)
      } else {
        resolve(data)
      }
    });
  })
}

function envNames(config){
  if(config && config.test_settings){
    return Object.keys(config.test_settings);
  }
  else {
    return [];
  }
}

async function spawnNW(envName){
  return new Promise( (resolve, reject) => {
    let err;
    let proc = spawn('nightwatch', ['-e', envName], {shell:true});
    proc.stdout.on('data', data => {
      if (data.toString('utf8').replace(/\s/g, '') !== ""){
        console.log(`${envName}:\n ${data.toString('utf8')}`)
      }
    })
    proc.stderr.on('data', data => {
      if (data.toString('utf8').replace(/\s/g, '') !== ""){
        console.error(data.toString('utf8'))
      }
    })
    proc.on('close', (code) => {
      if(code === 0){
        console.log(`${envName}: Finished!`)
        resolve(code)
      }
      else {
        reject(new Error(`${envName}: failed ` + err));
      }
    })
  })
}



( async () => {

  // make sure tunnel is started before starting selenium tests
  await startTunnel(tunnelOptions);


  // look for environment names in command line args...
  let names;
  for (let i=0; i < process.argv.length; i++){
    if (process.argv[i] === '--env' || process.argv[i] === '-e'){
      names = process.argv[i+1].split(',');
    }
  }

  // or pull from the config file
  if (names === undefined){
    names = envNames(nwConfig);
  }

  // spawn nightwatch for each environment
  let procs = names.map(spawnNW)

  console.log(`Starting to run Nightwatch on ${names.join(', ')}`)

  // wait for all nightwatch processes to return
  Promise.all(procs)
    .then( () => {
        console.log('Nightwatch finished!')
        // stop tunnel when all tests are done
        cbt.stop()
    })
    .catch( (err) => {console.log(err)})
})()

Once that file is in place, we can use it to run our tests!

node parallel-nw.js --env ie,edge,chrome,ff,safari

Much better!

Nightwatch.js is an End-to-End testing platform for writing and running Selenium tests with Node.js.

SmartBear Named a Leader in Gartner Magic Quadrant for Software Test Automation

By Leave a Comment

SmartBear was named a Leader in the 2018 Gartner Magic Quadrant for Software Test Automation. Gartner, the world’s leading information technology research and advisory company, has recognized SmartBear for the fourth consecutive year, with this being the first as a Leader.

Amidst emerging trends in DevOps and Agile Development, SmartBear has solidified its commitment to accelerate delivery by enabling continuous testing, making it a key player in this space.

SmartBear has encouraged teams to adopt intelligent automation practices across the entire software delivery lifecycle, providing a portfolio of tools to easily design, test, and monitor at both the API and UI layer such as TestComplete, TestLeft, Hiptest, ReadyAPI, and CrossBrowserTesting.

CrossBrowserTesting is proud to be part of SmartBear since being acquired in 2016. The CrossBrowserTesting provides the ability to run Selenium- and Appium-based tests across over 1500 real desktop and mobile devices in the cloud. Additionally, capabilities for record and replay, remote interactive testing and debugging, and visual testing in CrossBrowserTesting make it an asset to teams are practicing both manual and automated testing.

The strong support for open source communities like Selenium in CrossBrowserTesting as well as the OpenAPI Initiative, Swagger, and SoapUI has further propelled the success of SmartBear. By contributing to these initiatives, SmartBear has a crucial hand in the open source tools that are used by thousands of software teams every day.

Additionally, new offerings over the last year have broadened the portfolio and enabled SmartBear to lead the test automation market. With the acquisitions of Zephyr for test management and Hiptest for continuous testing with native BDD support, as well as the development of LoadNinja, SmartBear is innovating at an unparalleled speed.

“We’re proud to be recognized by Gartner as a Leader in the Magic Quadrant,” said Christian Wright, Chief Product Officer and Executive GM at SmartBear. “We believe our Leader position reaffirms our vision of enabling quality throughout the software development lifecycle through a comprehensive test automation portfolio. We see Gartner’s recognition reflects the excellent feedback we receive from our vast customer base and our teams’ ability to deliver open, collaborative, and easy-to-use tools to our users.”

To access a complimentary copy of the 2018 Gartner Magic Quadrant for Software Test Automation, visit https://smartbear.com/resources/white-papers/gartner-magic-quadrant-2018/.

Gartner, Magic Quadrant for Software Test Automation, Joachim Herschmann, Thomas Murphy, Jim Scheibmeir, 27 November 2018

Gartner does not endorse any vendor, product or service depicted in its research publications, and does not advise technology users to select only those vendors with the highest ratings or other designation. Gartner research publications consist of the opinions of Gartner’s research organization and should not be construed as statements of fact. Gartner disclaims all warranties, express or implied, with respect to this research, including any warranties of merchantability or fitness for a particular purpose.

This graphic was published by Gartner, Inc. as part of a larger research document and should be evaluated in the context of the entire document. The Gartner document is available upon request from [insert client name or reprint URL].

3 Methods for Hands-free Continuous Testing

By Leave a Comment

continuous testing with jenkinscontinuous testing with jenkins

In order to strike the perfect balance between speed and quality, teams that hope to achieve Continuous Integration and Continuous Delivery must also implement Continuous Testing.

This is often easier said than done. It’s virtually impossible to have your team running tests at every moment — they do have lives outside of work after all, and it can be difficult to put dinner on the table while running a Selenium script at the same time. So how do you run your scripts, even if you aren’t at your desk?

Fortunately, there are various hands-free options for continuous testing that you can implement, which will keep tests running for you while you eat, sleep, or maybe even get some exploratory testing done.

Why Continuous Testing

What’s the point of continuous testing? There are a few major advantages that will enable you to achieve more in your day-to-day.

  • Test while you sleep – With continuous testing, you can literally test in your sleep. By setting tests to run overnight, you can experience the luxury of coming in the next morning to have your test results waiting for you and knowing with a glance whether they passed. Just ask America’s Test Kitchen about how convenient it is.
  • Keep up with Agile, DevOps and CI/CD – When software development teams are constantly integrating, you need a testing strategy to ensure those new features, fixes, and changes are put through the quality process, no matter how frequently they’re implemented. Continuous Testing is the only way QA teams can keep up with constantly changing features, frequent integrations, and fast workflows.
  • Catch bugs in each regression – When there are constantly small changes to an application’s code, it can break something that was previously functioning. And you don’t want your customers to be the one to catch it. Continuous Testing will help you be first to spot new bugs, so you can quickly debug and release to users with confidence.
  • Level up your automation game – Automate your automation with continuous testing, and set your tests to run as often as you need. Then come back to test results or screenshots that tell you exactly what you need to know about the health of your applications. Plus, testing in parallel allows you to further accelerate Continous Testing to increase browser coverage without compromising runtime.

Continuous Testing With a CI Server

Selenium WebDriver has long provided teams with the ability to automate tests across browsers. But in order to run those tests continuously, a Continuous Integration tool such as Jenkins, VS Team Services, Buildbot, Bamboo, CircleCI, TravisCI, Codeship, or TeamCity is needed.

By far, the most popular option is Jenkins, an open source automation server written in Java. In fact, according to the SmartBear 2018 State of Testing Survey, 66 percent of participants choose Jenkins to be their CI server. For development teams adopting Agile and DevOps workflows, Jenkins is a reliable tool for continuous integration and delivery.

But what about testing teams that want to leverage Jenkins for Continuous Testing with Selenium? The CrossBrowserTesting integration with Jenkins lets you run automated functional tests with Selenium, Appium, and visual screenshot tests, which means that every time you run your build process, you can simultaneously run your testing suite across thousands of browsers.

This is ideal for teams that already work with Selenium and a CI tool for deployment to start testing on a continuous basis. While Jenkins tends to be the most popular choice for CI and automated test scheduling, CrossBrowserTesting also integrates with TeamCity and VS Team Services, so no matter what tools you’re already using, it’s easy to integrate CrossBrowserTesting into your current workflow.

Continuous Testing With a Record and Playback Tool

For teams that may be less familiar with Selenium but still wish to automate, a record and playback tool might be a more practical option.

Record & Replay enables teams to achieve automation across browsers, just like they would with Selenium, without requiring knowledge of scripting and frameworks. This has provided teams with a more accessible option for less technical team members, or those who are just getting started with automation, to create automated tests. As tools like Selenium IDE and its alternatives are brought back into the spotlight, these options are becoming more and more attractive to modern testing teams.

But where record and playback tools will normally only run the test once to check if the test passes, how can testers implement Continuous Testing?

With new Scheduling in CrossBrowserTesting, you can schedule a Record & Replay suite to run as often as you need them. For organizations that have less technical testers, this allows you to achieve continuous testing without requiring the skills to run Selenium and Jenkins or another automation server.

Continuous Screenshot Testing

Teams that solely focus on functional testing without incorporating visual testing will be disappointed when all their tests pass and users are still pointing out visual bugs.

When visual elements can change from browser to browser or device to device and make the user’s experience difficult or impossible while still functioning correctly, it’s important to run compare screenshots in regression. Comparing these pages side by side and with their historical versions means you can be more aware of changes that break the application in order to debug before they reach the end user.

While you could always run automated screenshots in CrossBrowserTesting, it required a manual motion to rerun those tests on a certain cadence. You can now set those screenshot tests to run with the new Scheduling feature, just like you can with Record & Replay.

Continuous Testing for Every Team

No matter what tools you’re already using, what technical level you’re at, or how far your team is in their test automation journey, there are options to help every team achieve continuous testing in CrossBrowserTesting.

With options to schedule tests with Selenium, Record & Replay, and automated screenshots, you can take your automation to the next level and receive test results through email or Slack on a daily, weekly, or monthly basis that you decide.