automated testing Archives | CrossBrowserTesting.com

Upcoming Webinar: Accelerating Your Test Execution Pipeline

August 30, 2017 By Alex McPeak Leave a Comment

Do you find your testing process can’t keep up with the demands and expectations of quickening software development and delivery cycles? Do you mark tests as “done” just to discover certain users are still finding bugs? Maybe, you wouldn’t mind finding a solution that would cut your testing time and increase quality, without having to choose just one or the other.

On September 7th, SmartBear will be hosting a free webinar entitled “Accelerating Your Test Execution Pipeline” in two sessions (8:30 AM and 2:00 PM EDT) to talk about just that. Gabriella Grandgard and Daniel Giordano will discuss ways to identify key bottlenecks in your testing processes that will help to deliver software on-time without compromising quality.

As test automation processes increase among organizations, testing efforts can lag behind largely because of scalability issues. Test environments are often the root cause for a number of these problems due to the time it takes to create them and maintain them, and the cost to reflect real users. Incorporating a test automation platform can cut your testing time in half by leveraging the cloud to test multiple devices and browsers at one time, and fix bugs as fast as they’re found rather than waiting until after deployment.

This method referred to as parallel testing allows testers to run scripts in multiple environments concurrently to save time, increase coverage, and speed up tests. This means that instead of running one script sequentially across different device and browser configurations, it can be done at the same point in time. As browsers, devices, and operating systems continue to release and update, keeping up with these versions calls upon the need for both parallel testing and cloud testing to support scalability.

Join our software experts to learn how testing teams can accelerate their testing efforts by easily scaling and reporting on their tests across multiple environments using parallel testing techniques.

Highlights include:

Real world demonstration of testing across multiple environments.
Techniques that can help you easily speed up your testing efforts.
Strategies to enable parallel testing for both manual and automated testing processes.
New ways to quickly pinpoint problem areas in your tests.

Register for the free webinar here.

5 Steps for Taking Full-Page Screenshots of Your Website

August 28, 2017 By Alex McPeak Leave a Comment

Testers know the value of using screenshots for manual debugging and regression testing, but we take it a step further with capabilities for full-page automated screenshots. CrossBrowserTesting allows you to automate screenshots on real browsers and devices and compare them against a baseline to evaluate layout differences and find issues that need debugging more easily.

But what do we mean by a full-page screenshot? A full-page screenshot is more than what you see when you look at a web page on your device (a.k.a. a windowed screenshot), it’s a capture of the entire web page above and below the fold.

This means full-page screenshots account for any area you have to scroll to up, down, or sideways to view. As opposed to manually taking multiple windowed screenshots of one page and trying to stitch them together and compare them to multiple screenshots of another page, our algorithm scrolls the page, captures an image at each viewpoint, and merges them into one view in order to provide a fast and accurate screenshot of the entire page.

By evaluating different browser versions visually, side-by-side with our comparison engine, it’s easier to not only see layout differences but detect smaller problems that might go unnoticed.

Full-page screenshots are a great tool for

Exploratory testing – Exploratory testing is performed the first time a tester looks at new software or new features. Being able to multiply these efforts by visually exploring multiple browsers saves time while increasing coverage.
Regression testing – Any change to working code means the possibility that it’s been compromised. Regression testing is used to ensure that a recent change hasn’t impact any previously functioning features, and comparing them with pass/fail results makes the process easy, effective, and efficient.
Usability testing – Testing the UI/UX of a web page ensures that current and potential customers are happy with your application and will continue to come back. Usability testing across browser and devices configurations further considers diverse user experiences.
Responsive design – Responsive design by definition means that everything looks visually acceptable from screen to screen, device to device, and browser to browser, which means visually validating this on real devices is paramount.

Here are five easy steps to taking full-page screenshots of your website with CrossBrowserTesting:

Log into the CrossBrowserTesting app, click screenshots, and start a screenshot test by entering the URL of choice. Keep in mind, you can also take screenshots locally or behind a firewall so you can test and debug before your app reaches the public eye.
Choose from over 1,500 real browsers, devices, and operating systems to replicate real user scenarios. Hint: use Google Analytics to decide which browsers you should prioritize, or choose the “Popular Browsers” option.
Use the full-page setting and view high-resolution results within minutes. Compare to your baseline, filter by device, and easily examine highlighted layout differents that anyone could understand to find bugs, spot UI issues, and identify elements that are hidden or misplaced.
Download the results, save your reports, and share with your team through Slack, Hipchat, email, or Jira. These communication channel integrations make it easier to provide feedback, start discussions, and affect change by encouraging collaboration with different roles on the development team.
Take to live testing for remote debugging.

Full-page screenshots prove to be a great resource in many stages of testing, and automating them with a tool like CrossBrowserTesting makes the entire process much more streamlined. Or, you can just photoshop your own full-page screenshots together if you prefer, but we prefer our way.

By increasing test coverage and speeding up debugging time, you can more easily identify gaps in quality and quickly deliver better software to your customers. Additionally, the CrossBrowserTesting Comparison Tool provides the best of both worlds, striking a balance between manual and automated testing by a visual representation of your web page while eliminating shortcomings of human error.

The Anatomy of a Comprehensive Selenium Testing Strategy

August 28, 2017 By Bas Dijkstra 2 Comments

Now that test automation has established itself as a key enabler of shorter software development and delivery cycles, being fluent in Selenium WebDriver (Selenium) will give test automation engineers an edge in the current IT job market. However, there’s a lot more you need to know to create a successful Selenium testing implementation than just the Selenium API itself. In this article, we’ll take a closer look at what it takes to create, use, and maintain a Selenium-based test automation solution that’s able to live up to expectations.

Is Selenium testing right for you?

Before you implement Selenium, it’s wise to ask yourself whether it’ll be a good fit for your test automation efforts in the first place. While it’s true that Selenium implementations are available for a multitude of programming languages and there’s a wealth of information around Selenium testing available online, creating a solid solution around Selenium and implementing robust, reliable, and maintainable test scripts is not an easy task. It requires both solid development and testing skills to write, run, and maintain a Selenium testing suite that lives up to its promise of faster feedback and shortened delivery cycles, instead of being a time and effort drain that slows down your software development efforts.

The remainder of this article illustrates the most important skills needed and the factors that should be taken into consideration to make your Selenium testing strategy successful once you’ve decided it’s the right direction.

Choose your tests wisely

Once you’ve got the hang of creating automated Selenium tests, it might be tempting to replace all of your current tests with Selenium-based counterparts. However, before you start automating away, it might be a good idea to consider the following:

Despite being one of the most commonly used user interface-driven automated test tools, creating high quality, reliable Selenium testing is hard. Maintaining a large set of these tests, especially in fast moving environments, is even harder.
Selenium-based tests offer you the broadest scope since they exercise an application all the way from the user interface through API and business logic layers all the way down to the database (and back again). This has a downside, too. Whenever a defect occurs at a lower level in the application, there’s a significant chance that your Selenium test isn’t able to pinpoint the root cause of the failure, only its effect on the end user experience. This is also known as shallow feedback, and it’s something you want to avoid when you’re trying to pinpoint and fix defects fast.
Selenium testing is much slower in terms of execution compared to unit or API-level tests. When you’re striving for fast feedback, you want your automated test suite to give you this feedback in the shortest amount of time possible.

What the above boils down to is simple: never use Selenium (or any other user interface-driven tool, for that matter) to automate a test that can be done at a lower level such as the API layer. As a general rule, Selenium and similar tools should only be used to verify that an end user of the application can successfully complete a sequence of actions in the application.

Apply good programming practices

Despite the ongoing advertisement of ‘codeless’ test automation tools that ‘allow everyone to create automated tests’, test automation still is a form of software development and should be treated as such. This applies to Selenium testing as well. Since you’re writing your tests in code, it makes sense to apply good programming practices to this code as well:

Make your tests maintainable by leveraging patterns that promote modularity and reuse. A single change in your application under test such as the introduction or removal of a text field should require only a single change in your test code. Probably the best known and most widely used pattern in Selenium is the Page Object pattern.
Keep your tests readable by using descriptive method and variable names. The intent of a test method named withdrawMoney_ResultsInError_IfBalanceIsInsufficient() is clear, while a test method named testCase147() or testWithdraw() is far less so. You can find more suggestions for useful test method names here. Be sure to apply readable standards to variables as well. You’ll see that a well-structured test with a descriptive name is pretty much self-documenting.
Optimize the stability of your tests to prevent false positives. Failing tests can be quite costly in terms of time needed to analyze the root cause, and time spent analyzing false positives is time wasted. Unfortunately, Selenium tests are among the tests most prone to false positives. One tried and tested way to create more stable tests is by creating wrapper methods around the Selenium API to create a single place where synchronization, exception handling, and additional logging and reporting can be implemented. If you want to be able to rely on the results of Selenium testing — something that’s becoming more important now that Continuous Integration and Continuous Delivery are being widely adopted — you’ll need to make that sure you can trust your test execution to be stable.

Create a solid test data management strategy

One of the hardest aspects of creating a solid Selenium testing strategy is the question of how to deal with test data. For any sufficiently complex application, there will be test cases that require the application to have data such as bank accounts, subscriptions — you name it — in a specific state before the test can be executed. The test data problem only gets harder when there are connected dependencies that have their own specific requirements with regards to the test data used. There are several ways to tackle this problem:

Creating the test data anew before the start of every test. This ensures that the right test data is available at all times, but it has the downside of increased test execution time especially when there are no hooks or access points that allow you to create test data below the user interface, and there’s risk of test execution failing before the part that comprises the actual test has even started.
Querying the system for existing test data. This minimizes the time needed to execute tests because they use data that is already in the system. There is a risk of test data not being present, resulting in a failing test, or test data being selected that is not suitable for the test under hand. This can lead to defects that might be hard to analyze or tests that pass when they shouldn’t.
Reusing test data from previous tests. This might look like a solid approach, however, it creates dependencies between tests. This can lead to several interdependent tests failing just because of a single defect that prevented a test that was responsible for creating test data to pass. It might also lead to false positives in case of parallel test execution.

As you can see, there is no one true way to manage test data. I would recommend tackling this problem as soon as possible, though, since it’s always easier to change the approach when you’ve got a relatively small test automation suite.

Create a solid test environment strategy

With applications becoming ever more distributed in nature, it’s getting harder to set up and manage test environments not only for your application under test, but also for all dependencies that your application interacts with during test execution. And that doesn’t even include having the right test data in place in all of these dependencies. So, whenever you’re starting on your journey to create automated tests using Selenium, it is a good idea to also take into account the requirements that your tests place on the test environment.

Is a test environment available and accessible for all of the required dependencies, all of the time? Will they contain the required test data, or is there a way you can take care of that? If the answer to any of these questions is ‘no’, it might have an effect on the scope of your tests or the frequency with which you can run your tests. If your test environments become a serious roadblock, you might consider applying a technique such as mocking or service virtualization to replace the dependencies with simulations that are under your own control instead.

As you can see, there’s much to consider when you’re creating automated tests or thinking about Selenium testing. Take care of the above aspects, and the chance that your journey will be a successful one increases significantly.

About the Author: Bas Dijkstra a test automation and service virtualization consultant who’s always looking for more intelligent ways to use tools to improve test processes and software quality. He’s been in the field for over 10 years, designing and developing test automation and service virtualization solutions that enhance and improve test teams and test processes. You can read more about him by reading his blog or following him on Twitter.

Debunking 6 Automated Testing Myths

August 14, 2017 By Alex McPeak Leave a Comment

Automated testing has been the answer for many teams who wanted to speed up and increase their testing efforts, but misconceptions surrounding automation mean that testers don’t always get the most out of their tools.

In an attempt to correct frequent misunderstandings, we’re busting common automated testing myths with the more accurate truths.

Myth: You should automate 100% of your tests.

Fact: You can’t test everything. Though the ability to automate testing is a great skill, it’s impossible to automate every test. And even if you could, it wouldn’t be an efficient testing solution. Automated tests are based off source code that’s built based on manual tests, and some tests are better off being executed manually anyway. Plus, anytime a new integration is added, a new test has to be done because automated tests will only check previous scripts. Experienced automation engineers know that prioritizing tests and knowing when to automate is just as important as knowing how to automate. However, though you’re not able to automate every test, automation will greatly increase the number of tests you’re able to run at once.

Myth: Developers should lead automated testing efforts.

Fact: Automation Engineers should drive automation. While it does take programming and coding knowledge, most of the time it’s the testers that are building automated test scripts. As the software industry begins to depend on new testing methods and development processes, test automation architects and engineers will be in high demand. In fact, most of the time having an automation engineer is preferred to a developer because testers have the mindset, training, and specialized skills for different automation tasks, which include much more than just coding.

Myth: Automation is more expensive than manual testing

Fact: If utilized correctly, automation can save money. While the initial investment in automated testing tools and hiring testers may seem more expensive, the ROI of test automation is often worth it since it saves time that would usually be spent on manual testing and increases productivity. For example, if you’re manually running tests that could be automated, it’s costing you in wasted time because your testers could be doing something more constructive. If you think about this scenario over weeks, months, or years, that’s a huge opportunity cost. Test automation is also a cost effective way to extend testing coverage, scale testing efforts, mitigate risks as your application grows.

Myth: Automated testing is better than manual testing

Fact: Neither automated or manual testing is better or worse than each other, they just have different uses. The most effective testing strategy will find a happy medium between automation and manual testing. Manual testing is better for one-off tests or tests that require human observation, so exploratory tests, smoke tests, and usability testing should all be manual. Automation is usually best for repeated tests or things that need to be frequently tested, such as regression tests. Familiarizing yourself with the different uses and applying them to the appropriate test cases is the best way to get the most out of your time, money, and testing efforts.

Myth: Automated Testing is an easy fix.

Fact: Automated testing isn’t a one-and-done solution. You can’t set up an automated test and expect that you’ll never have to look at it again. In fact, the source code of your automated tests takes a good amount of maintenance to check that they’re doing their job. Not to mention, that changing an organization’s testing and development processes will require new training and onboarding to familiarize with automation and write sophisticated tests. Though the ROI automation will likely be apparent over time, the transition to a mostly-automated process from a mostly-manual one will take commitment, patience, and financial investment.

Myth: Automation will solve all of your problems.

Fact: Automation is not a silver bullet. Automation engineers experience false positives and flaky builds that come from a lack of human observation, maintenance, and company-wide collaboration. For example, common challenges using Selenium for automated testing might surround coding for cases with more complicated elements like dynamic content, multiple tabs, and pop-up windows that are easier to do manually. Testing will still require the specialized skills of a talented testing team to troubleshoot problems, write scripts, manage tools, and maintain test cases. For some best practices of Selenium testing, check out this guide.

Why Your Startup Needs Software Testing

August 7, 2017 By Alex McPeak Leave a Comment

The need for a software testing in large companies to protect security, functionality, and other components that can make or break their customers’ trust is obvious. Oftentimes, the focus is so large on testing that enterprises have entire teams of testers employed at different skill levels and specializing in different areas.

But for startups that have a minimal budget for development, nonetheless testing, why should budding businessmen invest in software testers? It may seem like a step that could be skipped without causing much impact, however, entrepreneurs that have been in similar situations know the truth isn’t that simple.

The ROI

Why does software testing matter to a startup? With so many other expenses being extended in sales, marketing, and day-to-day operations at a time where you’re probably just trying to break even, spending limited finances on another department may not seem very strategic at first.

However, there’s a big reason you’ll want a tester on your team, and that reason is quality.

While a few bugs early on in your software’s first years in the public eye might not seem like a huge deal, software defects are known to be costly when they make it past product release. In fact, it’s about 30 times more expensive to fix a bug after release than in design.

Debugging after a user has already noticed a problem is not just expensive, but it’ll cost you in productivity, reputation, and consumer trust as well, which is not something you can afford to lose with so few customers who are closely evaluating what your product has to offer them.

Instead, focusing on releasing high-quality software will create brand loyalty and help you stand out against competitors, making earning and retaining new customers much more attainable.

Hiring Software Testers

You may be asking that if you have a developer who clearly knows how to code and debug software, why do you need to hire a dedicated software tester? Why can’t your developer just test the software?

The biggest reason why having a developer test their own software is problematic is that they’re already so familiar with the application that it’s hard for them to act as an end-user and handle it in a way that it’s not made for.

On the other hand, the job of the tester is to test the limits of an application the way a user might. For example, since the developer built the application, they know how the checkout process is supposed to work and might not think to enter an invalid email like a tester would.

Since testers also have the technical knowledge and know the in’s and out’s of software development, their insight provides a highly accurate approach to the quality of software before it’s released. Developers and testers have two different skill sets, and each should be utilized for the job they’re best at in order to create a seamless software development life cycle.

Also, since developers obviously spend a lot of their time actually coding, it doesn’t provide them much time to be testing, too — or at least not as much as they should be. At the very least, there should always be one tester working alongside the developer to make sure appropriate testing can be done on the software at different stages of development and at every integration.

Additionally, if you’ve been hearing a lot about automated testing and think it may be a resource to replace actually hiring a human tester, you’d be sadly mistaken. Automated tests are great for following commands and pre-written tests, but won’t be able to find problems in unexplored regions of the software. This means that in order to do any testing, manual or automated, you’ll have to have a human that knows how to write tests.

The Tests Your Startup Should Be Doing

Below are the few of the different testing methods you’ll want to be doing, even at a startup:

Exploratory Testing

For a new application, exploratory testing is extremely important for laying a groundwork of quality. Exploratory testing is an ad-hoc process where the tester manually goes through the software in an attempt to find bugs.

Again, not just anyone can properly execute effective exploratory testing as there is usually a strict process and specific guidelines to follow. Exploratory testing is largely used to define and evaluate the usability of the app as a customer would experience it, and having an expert manually test the app is the best way to evaluate components like functionality and usability.

Regression Testing

Testing is not a one-off process — you can’t just run one test and be able to release new iterations whenever you want from then on. In fact, every time a change is made to the code, you’re going to want to run a regression test to make sure it hasn’t broken previously functioning features.

This is a great example of where you might have to utilize both manual and automated testing. Doing the same test over and over will take up a big chunk of time, so automation is usually a preferred way to compliment manual efforts. On the other hand, manually creating a test script and running it is necessary before automating regression tests. Either way, regression tests are important to make sure a bug doesn’t pop up after adding a new feature.

Performance Testing

When thinking about quality, fast performance is one of the best ways to establish a loyal user, but bad performance will just as quickly turn potential users away. When a 2-second delay in load time during a transaction results in abandonment rates of up to 87 percent, those are losses that you can’t afford.

You want to make sure your application loads quickly and can handle expected traffic. This is the best way to avoid frustrating your users and prevent them from leaving your website before they make it to check out.

Responsive Design

One of the most crucial details that can easily go un-examined without having a software tester that knows what to look for is the idea of responsive design.

Your website will most likely be accessed from an array of different devices and browsers — desktops, laptops, tablets, and smartphones on Chrome, Safari, or even the looming Internet Explorer. Gone untested, the user experience may be drastically different on each of these configurations, which isn’t a good thing.

One of the worst things you can do is go without optimizing your application for mobile devices — the web page simply won’t survive. Finding a tester who knows the importance of cross-browser testing and knows how to find inconsistencies in responsive design will be one of the most valuable assets your startup could invest in.

Conclusion

It’s pretty simple — when you’re a new player in the game, quality is what’s going to set your apart. Don’t make the mistake of skimping out of testing your product to save money because, in the end, it’ll probably cost you customers.

Hiring testers will ensure that someone is monitoring potential issues and catching bugs before release. While it’s nearly impossible to prevent every bug, having a testing strategy in place creates a safeguard to affect better software and happier customers.

The Most Common Challenges Testers Face with Selenium

August 3, 2017 By Alex McPeak Leave a Comment

Selenium automates the browser to mimic real user actions on the web. As the web gets more complex, using Selenium to test has become increasingly popular. Although it has made web testing far easier for many teams and organizations across the world, it still has its fair share of challenges due it’s open source nature.

Most of the issues that testers experience have fairly straight forward solutions, which is why we outlined the most common Selenium challenges and how to fix them.

Pop-up Windows

When a simple, prompt, or confirmation alert pops-up, it can be difficult to automate it to either accept or close. Additionally, Windows-based alerts are beyond Selenium’s capabilities since they’re part of the operating system instead of the browser. However, since Selenium can operate multiple windows, web-based alerts can usually be handled with the switchTo method to control the pop-up while keeping the browser in the background.

Learn more about handling pop-up windows with Selenium here.

Dynamic Content

A web page that has dynamically loaded content has elements that may not be at first visible when you visit the site. This can mean that the page is user specific and will display different content based on different users, new content appears on the page after a certain amount of time, or it appears after clicking something on the page. For when a locator cannot identify a new element present at that time, Selenium comes with an integrated Explicit Wait where you can specify an amount of time to wait before automating a command. This should help give the page enough time to load and identify the element to interact with.

Learn more about testing dynamic content with Selenium here.

Flakiness

Sometimes Selenium will give you flaky tests, which means they’ll come back as positive or negative when they actually are the opposite. Eventually, too many flaky tests may mean testers might ignore the results, though this isn’t a great option either. Oftentimes, unstable builds can be a product of one of many factors: poor test isolation, flaky external services, and timeouts. By examining the elements of your Selenium tests, it’ll be easier to find out why your builds are unstable so you can approach the problem head-on.

Learn more about managing flaky tests here.

Can’t Test Mobile Apps

While Selenium can test on any operating system and browser on desktops, it’s limited when it comes to mobile testing in that it cannot run on native operating systems like iOS and Android. So, for example, the version of Facebook you pull up on Safari on your Mac can be tested with Selenium, but not the one on mobile Safari – on your iPhone 7. The Selenium family has a solution. It’s cousin, Appium, is an open source test automation framework that drives iOS and Android native, mobile, and hybrid apps using the WebDriver protocol and can be used for this purpose for testing mobile apps instead of web applications. As users increasingly move to mobile devices, it’s no surprise that developers and testers are taking advantage of the functionalities of Appium for this purpose.

Learn more about Appium here.

Limited Reporting and Documentation

While Selenium will exponentially increase your automated testing capabilities, because it’s an open source tool it is limited and features and does not support much reporting on its own. For this purpose, Selenium is best complimented with a third-party tool like CrossBrowserTesting that can capture screenshots and share reports through integrations like Slack and HipChat. However, you can also set up a framework to generate an output folder after a test with report information like errors, execution time, pass/fail count, etc. like TestNG.

Learn more about automated screenshot testing here and generating reports with Test NG here.

Multi-tab Testing

Selenium has the ability to test in multiple tabs, but it might present as an obstacle if you don’t know the correct commands. For example, if you want to open a new tab to perform a specific action without leaving the original tab, you’ll still want to be able to control both tabs and go back to the parent tab if users are likely to perform these kinds of actions. The best method for this is to store the previous window handle in a variable and then continue in the new window and store that in a second variable. Then, you can use the switchTo method to go between both windows as needed.

Learn more about how to work with multiple windows here.

No Manual Testing

Many testers will get excited about Selenium’s capabilities and expect that they can automate everything when the reality is that this just isn’t possible. In fact, a lot of automation relies on basic manual testing efforts. Rather, learning to prioritize which tests to automate will be beneficial to any tester. Additionally, as we mentioned, Selenium lacks capabilities for visual testing, which is important for testing beyond functionality to assess overall user experience. The most efficient way to do visual testing is by outsourcing to a third-party screenshot tool that integrates with Selenium.

Learn more about the need for manual testing here, prioritizing automated tests here, and alternative visual testing with Selenium here.

Scalability

Again, while Selenium allows you to test on pretty much any browser or operating system, it’s still limited in how many tests it can run at once and how fast it can run them based on how many hub/node configurations the tester has. And again, as your software scales, so must your testing. Without a Selenium grid, you can only test sequentially. However, with a Selenium grid and a device lab or a third party cloud tool like CrossBrowserTesting, you can test in parallel. This means multiples tests on different configurations can run at the same time, reducing the time it takes to run automated tests and increasing the configurations you can test on.

Learn more about parallel testing here.

Conclusion

One of the best things about being a software tester is that there’s always something new to learn. As the industry shifted from manual to automation, testers were tasked with learning new skills such as programming.

Now, as they become more familiar with automation techniques and tools such as Selenium, it’s admissible that they’ll encounter more challenges as they explore and familiarize themselves with new testing trends. Fortunately, as software continues to evolve, experts can depend on insights and support from the rest of the testing community to make sure they’re keeping up. Pick your preferred programming language to get started with Selenium testing here.

What is Regression Testing?

June 22, 2017 By Alex McPeak Leave a Comment

When a developer goes to make adjustments in their software, the smallest change to code can disrupt previously flawless functionality. You know it worked before, so if it doesn’t work now, it must be something we introduced with our latest push, right?

Regression testing is the process used to ensure that a recent code change hasn’t impacted these features that were already working, and is also used to monitor and maintain production software that has already passed testing.

Not only do you want to make sure everything looks good visually, but you also have to make sure that it performs quickly and functions without bugs. By re-executing these test cases, it’s easier to evaluate whether everything still works the way it’s expected.

When thinking about regression testing, there are three types including unit, partial, and complete. Unit regression tests are used to test a single unit of code, partial regression tests are used for code modification, and complete regression is for on-going code changes.

While many testers acknowledge the importance of comprehensive regression testing, it may be difficult for a team to agree on the best method for its execution.

The Problem with Regression Testing

While regression testing is a necessary part of QA, it’s probably not your tester’s favorite part of the job. In fact, the repetition involved with manual regression testing can be really tedious after running cases the first few times.

While you’ll likely have to run the first regression test manually, executing regression tests manually is a huge waste of time and energy over an extended period of time. And it’s really boring to repeat those same actions over and over – really boring. Manually running regression tests is an open window for human error, since it’s difficult to perform the actions exactly the same over and over.

In addition, the best QA teams know that an accurate regression test has to be done across more than one browser to ensure functionality across different configurations, increasing the time and energy involved in the process multiple times over.

Fortunately, testers that take advantage of automated testing can program those repetitive regression test cases to run in the background on a nightly basis, or with every deployment. By using tools likes Selenium, Appium, or other testing frameworks it becomes much easier to run regression tests, while your testers have more time to focus on their other priorities.

Additionally, automating parallel tests in Selenium takes this a step further, allowing you to approach cross-browser testing by running separate tests at the same time.

To read more about parallel testing with Selenium, read this resource.

Should I Automate All of My Tests?

No. Automated tests will only assert whether the actions you tell it to are true or false, which is why it’s great for regression tests — you’re going back to make sure nothing has changed over time.

However, automated tests cannot and will not find new bugs in unexplored areas of software. As your system scales, so must your exploratory and unit tests, as well as your regression tests. And, as we mentioned, your regression test cases will have to stem from manual test execution.

Once a test case is stable, it’s a good idea to perform regression tests on them in the future, but they will also need maintenance and validation as the system changes over time. You’ll probably have to create new regression test cases to compliment this change as well.

Additionally, a great benefit of automated regression testing is that it frees up a lot of time that can be spent manually testing other unexplored areas. You don’t want to use automation to take over testing completely, but to aid testing efforts that need to be maintained more than they need to be inspected.

Regression Testing Today

While regression testing is not a new term in software testing, new test management practices, tools, and methodologies have changed the way testers approach these tasks.

As organizations become more Agile and begin to continuously integrate new code, it’s important that those changes are verified and approved before new features and functionalities are delivered to end-users.

In fact, the industry shift from Waterfall to Agile has actually made more regression testing possible more often.

Since Agile teams have also adopted the idea of testing early and often, it’s allowed smaller regression tests cases to be performed intermittently rather than running long case at the end of development.

Instead of re-doing an entire test suite, running regression test cases as needed saves a lot of time a creating a baseline of quality. Additionally, regression testing becomes a crucial part of reducing risk and eliminating bugs before they reach the customer.

Luckily because of the high demand for regression testing, there are many tools that strive to make this easier. Whether you’re parallel testing or using a visual regression tool to evaluate inconsistencies, there’s no excuse to skip out on regression tests, and as your organization grows, you won’t want to.

By crafting a testing strategy, prioritizing automation, and using the right tools, you can utilize regression testing in the best interest of your software development and QA teams.

How much have you experimented with regression testing? What are the best ways you’ve found to streamline the process? Tell us your experience in the comments!

Prioritizing Tests When You Can’t Automate Everything

May 25, 2017 By Alex McPeak Leave a Comment

Though automation is a great method for executing more frequent tests, teams that try to automate 100 percent of their tests will quickly find that it’s a herculean task. GUI tests should be carefully prioritized with regards to business critical processes, revenue generating flows, and common test cases.

As testers new to automation gain more experience, they should be able to prioritize tests, saving organizations time and money.

When Not to Automate

Usability Testing

User experience design (UX) is a human driven evaluation of the usability of a product, application, or feature. This means the UX can’t be evaluated by automation very accurately because that would take out the aspect of human observation. Using our Screenshot Engine, you can visually verify the UX across different devices and browsers, but we recommend serious Usability Testing be done manually.

One-time tests

This one is pretty self-explanatory, but if there’s a feature you only plan to test once or twice, there’s no reason to try and automate it. Automation is primarily used for regression and repeated tests, and best saved for more stable code bases. For example, an ad-hoc test may just be used to check something without the need of extensive documentation and a formal test structure.

Brand new features

Smoke testing and exploratory testing are better for testing a new feature than writing full end-to-end tests for it. Automation is great for reducing testing time, but it doesn’t act as an end-user the way a manual tester is trained to do. Automation scripts won’t seek out new bugs; see below. Once the UI of that feature is stable, it’s in a better state to be tested across browsers and when small code changes are made.

Exploratory testing

Automated tests follow a script, they don’t explore the software to experiment with an application and break features. Unfortunately, since users aren’t robotic they won’t always know to use software the way it’s intended. By exploratory testing, manual testers can go in and consider more creative possibilities of how a user will handle it and execute negative testing.

Tests to Prioritize Automating

Regression tests

Again, the most basic use of automation is for regression tests. Are there certain test scripts that you know you always execute with every deployment? Did you just get to a stable point in your integration tests of a new feature and want to speed testing. That’s where automation comes into greatly reduce the time it takes, thus effectively assisting the ability for quality deployments in a CI/CD environment. Regression testing allows you to build a test suite that ensures quality with every iteration and change to the code, pinpointing where possible new bugs may be when integration systems or deploying new features.

Unit tests

As the base of Mike Cohn’s Test Pyramid, unit tests are great for automation because they’re the smallest kind of test (Hint: you should have a lot of them). This also encompasses the motto of testing “early and often” that’s a foundation of any good testing strategy.

High traffic pages

You want to have a good reputation in the eyes of your users, and a bug or broken link on your homepage can be embarrassing. If your analytics show that your traffic is converting through your webinar section then you should prioritize that page before your blog, making sure customers are able to sign up and join them.

Impact on customers

This is similar but slightly different than targeting highly trafficked pages. Will your customers be more likely to leave your site if they don’t have a good experience during check out compared to browsing inventory pages? Consider pages with high bounces rate, or pages that have a high ROI – like your free trial or contact us page.

Data-driven tests

Manual testing is great for UI, but it’s not a very data-driven approach, more so relying on human intuition and curiosity. If you are looking for specific test inputs to tell you exactly where bugs lie automation will give you a more accurate portrayal.

Cross-browser testing

You’ve tested your web app, but if you only tested in one browser, you’re missing the mark for broader coverage. This is a simple way to make sure that your web application looks acceptable to a wide array of users on different devices, operating systems, and browsers, which is cross-browser testing. However, unless you want this to take hours or days of your valuable time, your best option is to automate parallel tests. To learn more cutting down test time, read our guide to parallel testing.

Conclusion

Deciding between manual and automation can be pretty straight forward. Manual testing is most often used for more detailed human observation, while automated tests are a faster and more data-driven approach for repeated tests.

In addition, sometimes the line can blur between which tests you should execute first in both situations. Our best advice is to always consider the end-user first. Think about which tests will best ensure usability, functionality, and performance for them specific to the stage of development your software is in.

By making both manual and automated testing part of your ongoing QA strategy, you can establish that your software continuously is known for optimal quality among your user base.

How do you decide what tests should be automated first? Tell us how you prioritize in the comments!

The Difference Between Manual and Automated Testing

March 8, 2017 By Alex McPeak Leave a Comment

Manual vs Automated Testing

When you begin web testing, one of the primary decisions you’ll have to make is whether you’re going to test manually or use automated testing.

While neither of these options are technically better or worse than the other, the size, budget and time allowance of a project will certainly be factors that affect which method will work best in your testing procedure.

Below, we’ll outline the best times to use manual and automated testing, and the advantages and disadvantages of each.

Manual Testing

Manual testing is essentially self-explanatory — testing of a web application is done manually, by human action. This means that someone actually goes on a device to evaluate numerous components including design, functionality, and performance by clicking through multiple elements or units of a web application without support from a tool or script.

Advantages

The method of manual testing is inherently valuable to consumer-facing websites because the tester is able to have the full user experience, making it easier to assess visual components such as text size and layout as well as UI.

Additionally, most QA processes will begin with some degree of manual testing because it’s an ideal way to perform one-off testing and exploratory test cases as well as usability and ad-hoc testing.

Disadvantages

The limitations of manual testing are that it’s not the best method for test cases that have frequent repetition. For code that’s updated by the day or hour, manual testing would take an exponential amount of time and human resources, and therefore, would not be an efficient way to approach agile CI.

Furthermore, because manual testing is performed by a human, there’s more room for human error as opposed to the accuracy of automated testing. It can also be a boring, time consuming and tedious task for developers to continually perform manual tests every time they update a code, rather than spending their time actually improving the code itself.

Manual Testing v Automated Testing for CI

Automated Testing

Automated testing uses the assistance of tools, scripts and software to perform test cases by repeating pre-defined actions. All automated testing, however, requires some manual testing at first to create initial testing scripts.

Advantages

The biggest draw of automated testing is that it’s noticeably faster and more accurate than manual testing. This is because automation allows multiple tests to be run in parallel, since the test cases can be distributed across different machines.

In comparison, manual testing has to be done sequentially, and so only one test can be run at a time. In this way, the accuracy of automated testing makes it ideal for finding bugs when running tests repeatedly over a long time period, or running multiple tests on different browsers.

Additionally, Automated testing is preferred for large projects that require regression, load and performance testing and repeated execution. Specifically, developers that use an agile CI methodology depend on automated testing to be able to retest scripts as they continually change code because manually testing every time a small fix was made would be exceptionally time consuming.

Disadvantages

The downfall of automated testing is that, unlike manual testing, there is little human observation involved. While automated testing is more efficient in many ways, developers will sometimes find that they need to execute manual testing in order to add a human element and meet specific consumer needs.

Often people will attribute automated testing as being more expensive because it requires a greater investment in tools. However, if your project calls for automated testing, this investment will most likely pay for itself and be more cost effective over time.

ROI of Test Automation

Complimenting Testing with Tools

Whether you’re deciding to pursue a manual or automated testing strategy, there are a number of ways to make the process faster, easier and more accurate for your QA team.

Tools such as CrossBrowserTesting offer solutions and features that complement manual and automated testing alike. These value-add tools including live testing, automated screenshots, visual comparisons, selenium testing and local testing all contribute to a well-rounded project execution.

However, the success of your testing process will first depend on your ability to measure which testing method is best for your project. By evaluating the pros and cons of manual testing and automated testing as well as their best uses and what you would like to accomplish through testing, you can launch a solid strategy that will continue to improve your web application and users’ experiences.

To learn more about beginning test automation, check out our eBook.

Read Now

Register for the free webinar here.

The ROI

Hiring Software Testers

The Tests Your Startup Should Be Doing

Exploratory Testing

Regression Testing

Performance Testing

Responsive Design

Conclusion

Pop-up Windows

Dynamic Content

Flakiness

Can’t Test Mobile Apps

Limited Reporting and Documentation

Multi-tab Testing

No Manual Testing

Scalability

Conclusion

The Problem with Regression Testing

Should I Automate All of My Tests?

Regression Testing Today

When Not to Automate

Usability Testing

One-time tests

Brand new features

Exploratory testing

Tests to Prioritize Automating

Regression tests

Unit tests

High traffic pages

Impact on customers

Data-driven tests

Cross-browser testing

Conclusion

Manual Testing

Advantages

Disadvantages

Automated Testing

Advantages

Disadvantages

Complimenting Testing with Tools

Join Over 200,000 Testers & Developers And Start Testing Today