CrossBrowserTesting.com

A Design, Development, Testing Blog

The 5 Ways Parallel Testing Can Instantly Improve Your Workflow

By Leave a Comment

benefits parallel testing

You’ve probably heard of parallel testing before, but the ability to multiply your testing power without multiplying your testing time seems something of a QA fantasy if you haven’t tried it yet. But once you experience parallel testing, you’ll never want to go back to life before it.

While there are probably hundreds of benefits from parallel testing, everyone will gain the value in different ways. In case you need to be convinced (or you need to convince your boss), we rounded up the top five advantages you can expect from parallel testing.

  1. More Browser Coverage – Take one test, run it across multiple browsers at once — that’s the beauty of parallel testing. If you’ve been running tests on different browsers one after the other, or even just skipping browser testing all together, you’ll realize how valuable parallel testing is after your first time trying it. Instead of only testing on one or two browsers, you can substantially increase your cross-browser coverage with parallel testing, hitting configurations you didn’t even realize were an issue. Plus, with CrossBrowserTesting’s broad selection of browsers, operating systems, and devices, you have seemingly endless options for checking the functionality for any and every user.
  2. Less Manual Testing – We’re not saying manual testing is a bad thing — in fact, it’s a pillar of a successful testing strategy. However, when you’re manually performing the same regression tests over and over on different browsers, you’re not making the most of your time. Not only is this boring, tedious work, you’re also risking human error and inaccurate test data by trying to repeat the same tests. Parallel testing allows you to repeat these tests in regression without worrying about skipping a step so you can be confident you’re reaching all the right parts of your application.
  3. Saves Valuable Time – Besides increasing test coverage, parallel also decreases the amount of time you’re actually spending on each test. Since you’re running multiple tests at the same time, rather than one after the other, you’re testing gets done in the fraction of the time it normally takes. In fact, just running 2 tests in parallel doubles your speed, and the more you run in parallel, the more time you save that can be spent elsewhere. This means you can redirect that energy to go back to performing the exploratory tests and looking at new parts of your application at while still meeting your deadlines.
  4. Improving the Development Workflow – Not only does parallel testing make your job easier, it allows for better communication between departments and improves the productivity of your entire team. By allowing for faster feedback and better reporting, parallel testing equips you with the ability to relay results from testing back to your team earlier in the development cycle. This means that parallel testing will be useful in your organization’s effort to shift left and implement Continuous Integration and Delivery practices that will get iterations to market faster and with fewer bugs.
  5. Go Scriptless – With CrossBrowserTesting, you now have even more ways to run your tests in parallel. With our Selenium grid, you can use your Selenium skills to write tests and run them in parallel. With Record & Replay, you can also achieve parallel testing with zero coding knowledge needed by executing a test manually then rerunning your suites across the browsers your choose. This means teams who are both new to automation and experienced in Selenium can benefit from the capabilities of parallel testing.

We could talk all day about how amazing parallel testing is, but don’t take our word for it. Whether you’ve automated 100 tests or 1, you can easily start running tests in parallel now to experience more code coverage with less time spent on manual regressions.

Try your hand at running a test in two browsers at once then take it to the next level by adding more, and witness pure testing magic right before your eyes.

How CrossBrowserTesting Enhances Your Selenium Suite

By Leave a Comment

crossbrowsertesting selenium test suite

Learning how to work Selenium is the key to mastering test automation and writing suites that will provide valuable insights for your team. However, becoming fully acclimated with the ins-and-outs of Selenium takes time, and there will always be certain limitations of what the open source tool can help you do.

While Selenium has its many challenges, you don’t have to go it alone. Cloud testing tools like CrossBrowserTesting provide additional abilities to help make the most of your testing and integrate more automation throughout the software development lifecycle.

Take a look at the following ways that CrossBrowserTesting enhances your Selenium suite (besides, of course, maintaining a Selenium grid infrastructure with over 1,500 real desktop and mobile browsers):

  • Parallel Testing – Parallel testing allows you to test multiple browser and OS combinations at the same time rather than sequentially. This allows you to radically shorten test execution time by running long cross-browser tests at the same point in time across different environments instead of one after the other. Even just testing two browsers in parallel means that you’re testing twice as much, being two times and productive, and finding double the bugs in the same amount of time — just imagine what you could get accomplished with 5, 10, or 15 tests running in parallel.
  • Debugging with Recording – CrossBrowserTesting has extended capabilities that not only allow you to automate your Selenium scripts but also to record live tests and debug with inspection tools like FireBug, Chrome Dev Tools, IE Developer Toolbar and more for control over CSS, HTML, and JavaScript. This makes it easier to identify errors and allows you to further investigate what went wrong once a test fails. Once you know what you need to fix, you can do so right in the browser.
  • Out-of-the-Box Integrations – Automation gives teams the opportunity to insert more testing into the CI/CD workflow. CrossBrowserTesting further enables faster deployments through out-of-the-box integrations like Jenkins and TeamCity, allowing you to improve test coverage and application quality every build. Additionally, integrations like Slack, HipChat, and Jira make it even more simple to integrate CrossBrowserTesting into your current practices and communicate results with your teammates.
  • Automatic Translation – As discussed, learning Selenium takes a lot of time and patience, but learning Appium presents a whole different set of obstacles. Fortunately, CrossBrowserTesting has automatic Selenium to Appium translation that allows you to easily use the scripts you’ve already written to perform mobile web testing across different devices and browsers. With mobile testing taking center stage as users adopt more devices, you can’t afford to skip testing on a selection of Androids and iPhones. CrossBrowserTesting makes it possible for you to automate across these devices without learning Appium, which means you can just focus on making sure your web app is mobile friendly.

Once you understand how to leverage Selenium tools like CrossBrowserTesting to enhance your automation efforts, your quickly begin to notice how much more testing you can get done.

Familiarizing yourself with these capabilities will make automation more practical for you and more productive for your team as you’re able to increase the speed of testing cycles and spend more time on finding new bugs rather than manually executing tests.

Selenium 101: Running Your Test Against Two Browsers at Once

By Leave a Comment

how to run a test on multiple browsers in parallel

If you are familiar with automated testing but haven’t yet explored the possibilities of parallel testing, we can tell you right now that you’ve been missing out. Rather than testing sequentially, or running each browser test one after the other, parallel testing allows you to run cross-browser tests simultaneously.

This means if you’re running a test in just two browsers at once, you’re already doubling your speed and productivity. When time is a factor to every software development team, parallel testing is a no-brainer. Selenium is able to run tests in parallel, but CrossBrowserTesting makes it even simpler – allowing you to test across multiple browsers and mobile devices with just a few extra lines of code.

Our example below will be using the Selenium’s Python library, and we’ll running our test against two browsers at once in CrossBrowserTesting with Selenium.

First, you want to choose the two browsers you’ll be running the test on. There are a few ways to choose which configuration to test, but the best way to do it for this purpose is to choose the two most common browser versions and operating systems your customers use to visit your web application with Google Analytics.

For this example, we’ll use IE 10 and Chrome 50 on Windows operating systems. Our capabilities will look something like this:

browsers = [
   {“platform”: “Windows 7 64-bit”, “browserName”: “Internet Explorer”,”version”: “10”, “name”: “Python Parallel”}, {“platform”: “Windows 8.1”, “browserName”: “Chrome”, “version”: “50”, “name”: “Python Parallel”},
]

Next, we’ll actually create the test itself, printing out relevant information like browserName to make our debugging experience easier. Our test will simply open CrossBrowserTesting.com and just wait for 3 seconds.

browsers_waiting = [ ]

def get_browser_and_wait(browser_data):
   print ("starting %s\n" % browser_data["browserName"])
   browser = get_browser(browser_data)
   browser.get("http://crossbrowsertesting.com")
   browsers_waiting.append({"data": browser_data, "driver": browser})
   print ("%s ready" % browser_data["browserName"])
   while len(browsers_waiting) < len(browsers):
     print ("working on %s.... please wait" % browser_data["browserName"])
browser.get("http://crossbrowsertesting.com")
     time.sleep(3)

For parallel testing, we’re going to use something called thread dependency, which you’ll see below. Multithreading is the implementation of parallel execution that communicates multiple processes on the CPU.

threads = []
for i, browser in enumerate(browsers):
   thread = Thread(target=get_browser_and_wait, args=[browser])
   threads.append(thread)
   thread.start()

for thread in threads:
   thread.join()

print ("all browsers ready")
for i, b in enumerate(browsers_waiting):
   print ("browser %s's title: %s" % (b["data"]["name"], b["driver"].title))
   b["driver"].quit()

Parallel testing becomes really handy when trying to increase the speed of your testing or actual testing coverage without sacrificing quality. Running a test against two browsers is a really great place to start with this script and already allows you to test twice as fast as doing the same testing sequentially on different browsers.

Once you begin using this method of testing on multiple browsers to extend the reach of longer test cases, the value will be evident. A test suite that previously ran for 5 hours can now be affordably decreased to around 15 minutes, giving your development and testing teams quicker feedback. You can also take your testing suite and boost your coverage immediately, running it against the browsers and devices your customers are actually using. If you extend this over day, weeks, and months, you’ll be benefiting from hours on top of hours of productivity.

For more of our Selenium 101 Series, take a look at the following:

To see more examples in other languages:

6 Ways to Speed Up Your Testing Cycles

By Leave a Comment

faster automated testing

In the battle between speed vs quality, teams that practice test automation know that there’s a practical way to aim for both. However, while testing inherently places value on precise observation and careful consideration over making mistakes for the benefit of speed, how can you make testing rapid enough the meet the needs of Agile development demands without skipping over essential steps?

Faster testing is easily attainable by refining your current automation strategy and adjusting the processes that are already in place. Not only will the following tips make your life easier, they’ll also accelerate your testing to help make the most out of your time.

  1. Get organized – Keeping your tests organized may not seem like a priority day-to-day, but it will make things much more streamlined for you in the future. You should be keeping your test cases lean and reusable, automating features at the unit level. By practicing good test management methods, maintaining accurate data, and keeping clean automation reports, it’ll be easier for you and your team to run tests whenever they’re needed.
  2. Talk to your teammates – Speaking of team, your teammates are some of your most important tools to driving testing that’s effective and efficient. Whether you’re the solo tester on a software team mostly made up of developers or you’re one of fifty other automation engineers, there are always other stakeholders in the software. Communicating with these people will help everyone stay on the same page as far as documentation and reporting, as well of the status of the software included what’s new, what needs to be tested, what’s been tested, and what needs to be fixed.
  3. Know when to automate – Of course, automating your tests will make things exponentially easier and faster, but automating the wrong test cases is just going to slow your progress down. While testing manually might seem to take longer, new test cases require exploratory testing, and taking the time to do it at the beginning will save you from having to go back and do it later after you’ve already tried to automate it. Ultimately, understanding the best cases to automate before you actually start running them will prevent you from wasting time — a software tester’s worst nightmare.
  4. Try parallel testing – While automation will unquestionably help you test more parts of your application faster, it falls short in the way that it is only able to accomplish this in a sequential fashion — no more than one automated test can generally be run at one time. However, with parallel testing, automation gets a much-needed boost by allowing different tests to run simultaneously. If you’re smart enough to understand the importance of the same tests across multiple browsers, operating systems, and devices, you can also see the value in doing this all at once with parallel testing instead of running the script over and over on different configurations.
  5. Use tools to your advantage – Being a great tester will only take you so far if you don’t know how to leverage the right tools to help you. Between adopting open source testing frameworks and investing in automation tools, there’s no shortage of ways to make your life a little easier. With the previously mentioned parallel testing, for example, you need to rely on a tool like CrossBrowserTesting for a cloud-based Selenium grid or Record & Replay.
  6. Cut out flakiness – While tools like Selenium are an asset in automation, it can also lead to flakiness. False negatives will just slow you down — no one likes seeing red all the time and dealing with inconsistent results. In order to avoid this, it’s best to attack the problem head-on by isolating flaky Selenium tests from the build to fix them before rerunning the test again.

Conclusion

Working towards faster test automation doesn’t have to mean starting over. Balancing speed and quality is something you just have to learn along the way, but getting to know test automation hacks will get you there even sooner.

Upcoming Webinar: Accelerating Your Test Execution Pipeline

By Leave a Comment

parallel testing webinar smartbear

Do you find your testing process can’t keep up with the demands and expectations of quickening software development and delivery cycles? Do you mark tests as “done” just to discover certain users are still finding bugs? Maybe, you wouldn’t mind finding a solution that would cut your testing time and increase quality, without having to choose just one or the other.

On September 7th, SmartBear will be hosting a free webinar entitled “Accelerating Your Test Execution Pipeline” in two sessions (8:30 AM and 2:00 PM EDT) to talk about just that. Gabriella Grandgard and Daniel Giordano will discuss ways to identify key bottlenecks in your testing processes that will help to deliver software on-time without compromising quality.

As test automation processes increase among organizations, testing efforts can lag behind largely because of scalability issues. Test environments are often the root cause for a number of these problems due to the time it takes to create them and maintain them, and the cost to reflect real users. Incorporating a test automation platform can cut your testing time in half by leveraging the cloud to test multiple devices and browsers at one time, and fix bugs as fast as they’re found rather than waiting until after deployment.

This method referred to as parallel testing allows testers to run scripts in multiple environments concurrently to save time, increase coverage, and speed up tests. This means that instead of running one script sequentially across different device and browser configurations, it can be done at the same point in time. As browsers, devices, and operating systems continue to release and update, keeping up with these versions calls upon the need for both parallel testing and cloud testing to support scalability.

Join our software experts to learn how testing teams can accelerate their testing efforts by easily scaling and reporting on their tests across multiple environments using parallel testing techniques.

Highlights include:

  • Real world demonstration of testing across multiple environments.
  • Techniques that can help you easily speed up your testing efforts.
  • Strategies to enable parallel testing for both manual and automated testing processes.
  • New ways to quickly pinpoint problem areas in your tests.

 

Register for the free webinar here.

Sign Up

What is Regression Testing?

By Leave a Comment

what is regression testing

When a developer goes to make adjustments in their software, the smallest change to code can disrupt previously flawless functionality. You know it worked before, so if it doesn’t work now, it must be something we introduced with our latest push, right?

Regression testing is the process used to ensure that a recent code change hasn’t impacted these features that were already working, and is also used to monitor and maintain production software that has already passed testing.

Not only do you want to make sure everything looks good visually, but you also have to make sure that it performs quickly and functions without bugs. By re-executing these test cases, it’s easier to evaluate whether everything still works the way it’s expected.

When thinking about regression testing, there are three types including unit, partial, and complete. Unit regression tests are used to test a single unit of code, partial regression tests are used for code modification, and complete regression is for on-going code changes.

While many testers acknowledge the importance of comprehensive regression testing, it may be difficult for a team to agree on the best method for its execution.

The Problem with Regression Testing

While regression testing is a necessary part of QA, it’s probably not your tester’s favorite part of the job. In fact, the repetition involved with manual regression testing can be really tedious after running cases the first few times.

While you’ll likely have to run the first regression test manually, executing regression tests manually is a huge waste of time and energy over an extended period of time. And it’s really boring to repeat those same actions over and over – really boring. Manually running regression tests is an open window for human error, since it’s difficult to perform the actions exactly the same over and over.

In addition, the best QA teams know that an accurate regression test has to be done across more than one browser to ensure functionality across different configurations, increasing the time and energy involved in the process multiple times over.

Fortunately, testers that take advantage of automated testing can program those repetitive regression test cases to run in the background on a nightly basis, or with every deployment. By using tools likes Selenium, Appium, Record & Replay, or other testing tools and frameworks it becomes much easier to run regression tests, while your testers have more time to focus on their other priorities.

Additionally, automating parallel tests in Selenium takes this a step further, allowing you to approach cross-browser testing by running separate tests at the same time.

To read more about parallel testing with Selenium, read this resource.

Should I Automate All of My Tests?

No. Automated testing will only assert whether the actions you tell it to are true or false, which is why it’s great for regression tests — you’re going back to make sure nothing has changed over time.

However, automated tests cannot and will not find new bugs in unexplored areas of software. As your system scales, so must your exploratory and unit tests, as well as your regression tests. And, as we mentioned, your regression test cases will have to stem from manual test execution.

Once a test case is stable, it’s a good idea to perform regression tests on them in the future, but they will also need maintenance and validation as the system changes over time. You’ll probably have to create new regression test cases to compliment this change as well.

Additionally, a great benefit of automated regression testing is that it frees up a lot of time that can be spent manually testing other unexplored areas. You don’t want to use automation to take over testing completely, but to aid testing efforts that need to be maintained more than they need to be inspected.

Regression Testing Today

While regression testing is not a new term in software testing, new test management practices, tools, and methodologies have changed the way testers approach these tasks.

As organizations become more Agile and begin to continuously integrate new code, it’s important that those changes are verified and approved before new features and functionalities are delivered to end-users.

In fact, the industry shift from Waterfall to Agile has actually made more regression testing possible more often.

Since Agile teams have also adopted the idea of testing early and often, it’s allowed smaller regression tests cases to be performed intermittently rather than running long case at the end of development.

Instead of re-doing an entire test suite, running regression test cases as needed saves a lot of time a creating a baseline of quality. Additionally, regression testing becomes a crucial part of reducing risk and eliminating bugs before they reach the customer.

Luckily because of the high demand for regression testing, there are many tools that strive to make this easier. Whether you’re parallel testing or using a visual regression tool to evaluate inconsistencies, there’s no excuse to skip out on regression tests, and as your organization grows, you won’t want to.

By crafting a testing strategy, prioritizing automation, and using the right tools, you can utilize regression testing in the best interest of your software development and QA teams.

How much have you experimented with regression testing? What are the best ways you’ve found to streamline the process? Tell us your experience in the comments!

9 Signs You Should be QA Testing in the Cloud

By Leave a Comment

cross-browser testing cloud testing

We’re all familiar with the cloud. As a topic that’s clearly trending and growing in software, our appreciation of the cloud testing stems from its innovation as well as its effectiveness.

The cloud is only projected to become more popular and more common in years to come. In fact, 95 percent of organizations are using a cloud in some way, and the market is supposed to increase 22 percent annually according to SD Times.

In software development, cloud testing provides an easy-access, remote testing environment to produce real user scenarios. As organizations consider the benefits of the cloud testing, here are some signs that tell you it’s time to use it for browser testing.

  • You’re starting a business – One of the best times to check your web design across browsers are in the early stages a.k.a the exploratory tests. New companies should ensure they establish the UI of their website to look and function responsively across multiple devices when quality can make or break a business reputation.
  • You’re moving from manual to automation – Cloud testing offers an easy onboarding for organizations whose testers are a little newer to programming and automation. As you continue to recognize the benefits of automating your tests, the cloud is a practical way to make the process pain-free. With on-demand access, it’s easy for your team to test whenever they need and learn as they go. Additionally, browser screenshots, extensions, frameworks, debugging tools, and integrations are included to optimize usability and make testing more enjoyable. Plus, customizable preferences and customer-centric user interface assure an optimal user experience.
  • You’re realizing there aren’t enough hours a day to test everything – If you find yourself with test cases that are taking more time than there are hours in the day, it might be time to think about adding some more machines to your parallel testing. Parallel testing is a feature exclusive to cloud testing, which will allow you to significantly cut down your testing time by testing different cases on multiple machines at the same time, instead of one after the other. Additionally, the cloud is available all day every day, so you can test whenever you want without having to be on-site. In fact, you can even run automated tests in your sleep. This will make it easier to test all the configurations you need to, while still meeting deadlines.
  • You have a hard time keeping up with all the tech – Software testing is not what it used to be. With so many new browsers, OS versions, and devices coming out all the time, it can be difficult to keep track of all the configurations you need to test on. With cloud testing, new releases are added for you so you don’t even have to think about it.
  • You think new smartphones are way overpriced – If your device lab budget isn’t quite allowing all the configurations you need, cloud testing might be just the right answer. When an average Android is about $670 and an iPhone rounds out at around $700, the numbers can add up fast when you try to test on every device your user has — not to mention maintenance.
  • You recognize the benefits of simulators, emulators, and real devices – While emulators and simulators have their place in testing, real devices are crucial in order to test on the machines that your users are on. To get the most comprehensive breadth of simulators and emulators plus real devices for testing, cloud testing is the best way to access all three for different purposes, and the combination ensures that you always have access to testing.
  • You’re scaling your team – You had QA all figured out when you had just one (albeit really talented) tester. But, as your company scales, so must your testing team. When everyone’s trying to complete different tasks that all seem to be of top priority at the same time, the cloud gives you more room to accommodate growing pains. Since everyone has access to the same devices at once, you can ensure you’re supplying your growing team with the resources they require.
  • You have testers in different offices – Similarly, it’s a little bit difficult to share the capabilities of your device lab among a QA team that’s spread out among different cities, countries, and continents. Cloud testing provides one platform that further encourages communication between different departments and remote teams with integrations like Slack sharing.
  • Your customers are getting demanding – Has your customer support team been extra stressed out lately? Are you getting user reports saying that your website is slow, or there’s a bug, or it’s not working on their browser? Your customers have high expectations of you. They might be using a wider variety of devices than you think, and it’s compromising their interaction with your web application.

What pushed you to make the decision to adopt a cloud testing service? Tell us in the comments!

Join Over 200,000 Testers & Developers And Start Testing Today

Learn more Free Trial