CrossBrowserTesting.com

A Design, Development, Testing Blog

Discovering Exploratory Testing

By Leave a Comment

exploratory testing

Despite being fairly unstructured, scriptless, and independent of a pass/fail status, exploratory testing is a very important and strategic part of QA. In fact, its focus on discovery and learning in software is proven to be highly valuable when it comes to testing an application for the first time or looking into a new feature from the last integration.

By identifying problems, questions, ideas, and praise through this method of manual testing, exploratory testing helps start a conversation between internal teams and produce improved software.

What is Exploratory Testing?

Exploratory testing is a scriptless testing practice that emphasizes learning and discovery in software and aims to improve quality.

While test cases are not created in advance, exploratory testing still has a very strategic process and requires documentation. Test design and execution are done simultaneously in an ad-hoc manner, although exploratory testing and ad-hoc testing are not the same (we’ll get to that later).

Since exploratory testing requires a human to look at software, evaluate its functionality, and think of new ideas, it’s a manual practice rather than an automated one. It also depends on the skills and knowledge of an independent tester to be able to evaluate where software has issues or could use improvements.

Exploratory testing is often best for:

  • New websites
  • New feature
  • Unexplored areas
  • Different devices
  • UI testing

Afterwards, these insights are discussed with the rest of the testing and development team members to make changes, fix bugs, or add new features. Additionally, results can be used to raise questions about what’s working well and what’s not to implement in other areas of the software.

Session-Based Testing

Session-based testing is one way to provide more structure in exploratory testing. While you can’t go into it with a script or a definitive test plan, having certain elements in mind like objectives, heuristics and oracles, focus, risks, and follow-up will make sure the process is valuable.

Session-based testing will often include a technique called mind mapping to identify some of these elements in a visual layout. This also helps write better documentation to share with teams after testing is finished.

Establishing a Definition

James Bach and Michael Bolton are attributed as the testers responsible for calling attention to the value of exploratory testing.

However, Cem Kaner coined the term in 1983, defining it as “a style of software testing that emphasizes the personal freedom and responsibility of the individual tester to continually optimize the quality of his/her work by treating test-related learning, test design, test execution, and test result interpretation as mutually supportive activities that run in parallel throughout the project.”

Today, some people suggest that exploratory testing should just be called “testing” since all testing has an element of exploration to it, but we still think that it deserves a place as its own method according to the original definition.

Exploratory Testing vs. Other Testing

The parameters of exploratory testing are often confused or misunderstood. Comparing and contrasting exploratory testing to other types of testing can help clarify its purpose.

Scripted Testing – Scripted testing and exploratory testing are posed as opposites, which helps better understand their definitions. As mentioned, exploratory testing is scriptless and more random, whereas scripted testing follows a set and defined plan written by the tester. This also means that you can’t test something that isn’t planned ahead with scripted testing, whereas exploratory testing encourages testers to discover new regions as they go. This allows testers to act more as the end user and evaluate areas they come across rather than predefined spaces.

Ad-hoc Testing – Exploratory testing and ad-hoc testing are probably compared most often, but they’re two distinct methods of testing. For one, ad-hoc testing doesn’t require documentation, which is an essential part of exploratory testing. Similarly, it’s not well-integrated into team collaboration and has no set goals, objectives, or focus. Though exploratory testing is meant to be done in an ad-hoc style, it also has more set expectations than a being a random test. Additionally, many divert the two because exploratory testing could be said to require more skill, whereas anyone could perform an ad-hoc test.

Can You Automate Exploratory Testing?

Not really. Exploratory testing inherently depends on the individual tester to contribute their skills, knowledge, and insight to new or unevaluated components in an application — something that cannot be done with automation.

Thinking about this in terms of the testing vs checking debate, exploratory testing would be part of “testing” while automated testing is a method of “checking,” meaning that you could not automate it because you can only use automation to check — a tester has to test since a machine cannot provide independent thought.

Tips for Exploratory Testing

  • Have a loose plan – Going into a test with a focus is important in order to come out of the process with valuable feedback on certain objectives. At the same time, the nature of the exploratory test is to address certain issues as they arise and consider those as well. While testers should have a general plan, they should also allow room to improvise and consider unfamiliar elements.
  • Think like an end user – Exploratory testing is an ideal way to advocate as a customer that may be using a website. While developers are familiar with the code they build, it might be more difficult them to think outside the box and use software in a way it’s not optimized for. The job of the tester in this phase is to recognize these areas of weakness and bring them to the attention of the rest of the team.
  • Leverage your skills – While exploratory tests are designed and changed as the tester goes along, it doesn’t mean that just anyone can do it correctly. In fact, the results of the test will largely depend on a tester’s skill level. Leveraging a tester’s knowledge of software and previous testing practices will be the most efficient way to find and address problems.

Debunking 6 Automated Testing Myths

By Leave a Comment

Automated testing myths and misconceptions

Automated testing has been the answer for many teams who wanted to speed up and increase their testing efforts, but misconceptions surrounding automation mean that testers don’t always get the most out of their tools.

In an attempt to correct frequent misunderstandings, we’re busting common automated testing myths with the more accurate truths.

Myth: You should automate 100% of your tests.

Fact: You can’t test everything. Though the ability to automate testing is a great skill, it’s impossible to automate every test. And even if you could, it wouldn’t be an efficient testing solution. Automated tests are based off source code that’s built based on manual tests, and some tests are better off being executed manually anyway. Plus, anytime a new integration is added, a new test has to be done because automated tests will only check previous scripts. Experienced automation engineers know that prioritizing tests and knowing when to automate is just as important as knowing how to automate. However, though you’re not able to automate every test, automation will greatly increase the number of tests you’re able to run at once.

Myth: Developers should lead automated testing efforts.

Fact: Automation Engineers should drive automation. While it does take programming and coding knowledge, most of the time it’s the testers that are building automated test scripts. As the software industry begins to depend on new testing methods and development processes, test automation architects and engineers will be in high demand. In fact, most of the time having an automation engineer is preferred to a developer because testers have the mindset, training, and specialized skills for different automation tasks, which include much more than just coding.

Myth: Automation is more expensive than manual testing

Fact: If utilized correctly, automation can save money. While the initial investment in automated testing tools and hiring testers may seem more expensive, the ROI of test automation is often worth it since it saves time that would usually be spent on manual testing and increases productivity. For example, if you’re manually running tests that could be automated, it’s costing you in wasted time because your testers could be doing something more constructive. If you think about this scenario over weeks, months, or years, that’s a huge opportunity cost. Test automation is also a cost effective way to extend testing coverage, scale testing efforts, mitigate risks as your application grows.

Myth: Automated testing is better than manual testing

Fact: Neither automated or manual testing is better or worse than each other, they just have different uses. The most effective testing strategy will find a happy medium between automation and manual testing. Manual testing is better for one-off tests or tests that require human observation, so exploratory tests, smoke tests, and usability testing should all be manual. Automation is usually best for repeated tests or things that need to be frequently tested, such as regression tests. Familiarizing yourself with the different uses and applying them to the appropriate test cases is the best way to get the most out of your time, money, and testing efforts.

Myth: Automated Testing is an easy fix.

Fact: Automated testing isn’t a one-and-done solution. You can’t set up an automated test and expect that you’ll never have to look at it again. In fact, the source code of your automated tests takes a good amount of maintenance to check that they’re doing their job. Not to mention, that changing an organization’s testing and development processes will require new training and onboarding to familiarize with automation and write sophisticated tests. Though the ROI automation will likely be apparent over time, the transition to a mostly-automated process from a mostly-manual one will take commitment, patience, and financial investment.

Myth: Automation will solve all of your problems.

Fact: Automation is not a silver bullet. Automation engineers experience false positives and flaky builds that come from a lack of human observation, maintenance, and company-wide collaboration. For example, common challenges using Selenium for automated testing might surround coding for cases with more complicated elements like dynamic content, multiple tabs, and pop-up windows that are easier to do manually. Testing will still require the specialized skills of a talented testing team to troubleshoot problems, write scripts, manage tools, and maintain test cases. For some best practices of Selenium testing, check out this guide.

Why Your Startup Needs Software Testing

By Leave a Comment

Software testing for startups

The need for a software testing in large companies to protect security, functionality, and other components that can make or break their customers’ trust is obvious. Oftentimes, the focus is so large on testing that enterprises have entire teams of testers employed at different skill levels and specializing in different areas.

But for startups that have a minimal budget for development, nonetheless testing, why should budding businessmen invest in software testers? It may seem like a step that could be skipped without causing much impact, however, entrepreneurs that have been in similar situations know the truth isn’t that simple.

The ROI

Why does software testing matter to a startup? With so many other expenses being extended in sales, marketing, and day-to-day operations at a time where you’re probably just trying to break even, spending limited finances on another department may not seem very strategic at first.

However, there’s a big reason you’ll want a tester on your team, and that reason is quality.

While a few bugs early on in your software’s first years in the public eye might not seem like a huge deal, software defects are known to be costly when they make it past product release. In fact, it’s about 30 times more expensive to fix a bug after release than in design.

Debugging after a user has already noticed a problem is not just expensive, but it’ll cost you in productivity, reputation, and consumer trust as well, which is not something you can afford to lose with so few customers who are closely evaluating what your product has to offer them.

Instead, focusing on releasing high-quality software will create brand loyalty and help you stand out against competitors, making earning and retaining new customers much more attainable.

Hiring Software Testers

You may be asking that if you have a developer who clearly knows how to code and debug software, why do you need to hire a dedicated software tester? Why can’t your developer just test the software?

The biggest reason why having a developer test their own software is problematic is that they’re already so familiar with the application that it’s hard for them to act as an end-user and handle it in a way that it’s not made for.

On the other hand, the job of the tester is to test the limits of an application the way a user might. For example, since the developer built the application, they know how the checkout process is supposed to work and might not think to enter an invalid email like a tester would.

Since testers also have the technical knowledge and know the in’s and out’s of software development, their insight provides a highly accurate approach to the quality of software before it’s released. Developers and testers have two different skill sets, and each should be utilized for the job they’re best at in order to create a seamless software development life cycle.

Also, since developers obviously spend a lot of their time actually coding, it doesn’t provide them much time to be testing, too — or at least not as much as they should be. At the very least, there should always be one tester working alongside the developer to make sure appropriate testing can be done on the software at different stages of development and at every integration.

Additionally, if you’ve been hearing a lot about automated testing and think it may be a resource to replace actually hiring a human tester, you’d be sadly mistaken. Automated tests are great for following commands and pre-written tests, but won’t be able to find problems in unexplored regions of the software. This means that in order to do any testing, manual or automated, you’ll have to have a human that knows how to write tests.

The Tests Your Startup Should Be Doing

Below are the few of the different testing methods you’ll want to be doing, even at a startup:

Exploratory Testing

For a new application, exploratory testing is extremely important for laying a groundwork of quality. Exploratory testing is an ad-hoc process where the tester manually goes through the software in an attempt to find bugs.

Again, not just anyone can properly execute effective exploratory testing as there is usually a strict process and specific guidelines to follow. Exploratory testing is largely used to define and evaluate the usability of the app as a customer would experience it, and having an expert manually test the app is the best way to evaluate components like functionality and usability.

Regression Testing

Testing is not a one-off process — you can’t just run one test and be able to release new iterations whenever you want from then on. In fact, every time a change is made to the code, you’re going to want to run a regression test to make sure it hasn’t broken previously functioning features.

This is a great example of where you might have to utilize both manual and automated testing. Doing the same test over and over will take up a big chunk of time, so automation is usually a preferred way to compliment manual efforts. On the other hand, manually creating a test script and running it is necessary before automating regression tests. Either way, regression tests are important to make sure a bug doesn’t pop up after adding a new feature.

Performance Testing

When thinking about quality, fast performance is one of the best ways to establish a loyal user, but bad performance will just as quickly turn potential users away. When a 2-second delay in load time during a transaction results in abandonment rates of up to 87 percent, those are losses that you can’t afford.

You want to make sure your application loads quickly and can handle expected traffic. This is the best way to avoid frustrating your users and prevent them from leaving your website before they make it to check out.

Responsive Design

One of the most crucial details that can easily go un-examined without having a software tester that knows what to look for is the idea of responsive design.

Your website will most likely be accessed from an array of different devices and browsers — desktops, laptops, tablets, and smartphones on Chrome, Safari, or even the looming Internet Explorer. Gone untested, the user experience may be drastically different on each of these configurations, which isn’t a good thing.

One of the worst things you can do is go without optimizing your application for mobile devices — the web page simply won’t survive. Finding a tester who knows the importance of cross-browser testing and knows how to find inconsistencies in responsive design will be one of the most valuable assets your startup could invest in.

Conclusion

It’s pretty simple — when you’re a new player in the game, quality is what’s going to set your apart. Don’t make the mistake of skimping out of testing your product to save money because, in the end, it’ll probably cost you customers.

Hiring testers will ensure that someone is monitoring potential issues and catching bugs before release. While it’s nearly impossible to prevent every bug, having a testing strategy in place creates a safeguard to affect better software and happier customers.

The 4 Best Methods for Effective Usability Testing

By Leave a Comment

usability testing

Usability testing assesses how easy it is for a user to get from A to B. That’s to say, if someone is using your application, it should be simple for them to accomplish the action they’re looking to complete.

For example, if you’re using a website to buy a computer, how straightforward is the process from when you land on the website through checkout? Is navigation clear, does the application load quickly enough, do buttons do what they’re supposed to, are informational forms uncomplicated? Additionally, you have to consider visual design elements that can either make or break the user experience.

The usability of a web application embodies this entire user experience, and usability testing evaluates everything from performance and UI/UX to responsive design. It ensures that current and potential customers are happy with your application and will continue to come back.

Usability Testing Methods

While there are many components to consider when thinking about optimum usability, there are a few methods that will give you the best glimpse of your website and how well it functions.

User Testing

User testing requires having real people use your app to see how well it functions from a nonbiased and unfamiliar perspective. You can either recruit family and friends to try using the app or find focus groups that are willing to explore different features and functionalities.

While it should be manageable to find people who are willing to test your application and give feedback, there are also companies who are dedicated to setting up user testing services for you. For example, UserTesting provides videos of real users in your target audience navigating your web or mobile app and sharing their thoughts. In addition, the service provides metrics and written feedback so you can reference a real user analysis as you integrate changes.

No matter whether you seek out your own users or hire a company, watching how real people use and interact with software is highly beneficial to understanding user stories and insights.

Surveying

Getting feedback from users can be just as valuable after they’ve completed their task as during. By implementing customer satisfaction surveys after someone has purchased a project or completed a trial on your website, they can focus on fulfilling their task and evaluate the overall process once it’s finished.

SurveyMonkey is one of the top questionnaire development companies, and they even supply questions and templates specifically for website feedback. Rather than only looking at whether or not a user converted to a customer, finding out why they did not can help you target pain points and make improvements.

As customer support begins to take the forefront of successful business models, being able to address feedback straight from the source is invaluable. By actually listening to what users have to say about what they’re looking for in something and what it’s lacking, you can make changes that in turn will elicit customer loyalty and create a product they’re more likely to advocate for.

A/B Testing

A/B testing is where you share two versions of one product (version A and version B) to see which one performs better. These changes can be complete website redesigns or something as minimal as a button saying “Click here” v.s. “Sign up” or changing it between red and green. Almost anything can be A/B tested to find out what engages the most customers.

Using analytics, you then compare which one users had a better experience with. This can be based on the time on page, bounce rate, or most commonly, conversion rate. Additionally, you can also request customer feedback asking why users preferred one version over the other.

Similar to user testing and customer surveys, there are companies you can use for complete A/B testing such as Optimizely. By showing half of you traffic the original web page design and half the modified design and then analyzing responses through a statistical engine, Optimizely helps you make data-informed decisions about your web page to determine what the end-user prefers. Ultimately, A/B testing helps you make changes that contribute to the best version of your app.

Leave it to the Experts

While the recent testing trends have been revolving around automation, there’s still a critical place for manual testing, especially when it comes to usability.

Software testers are able to look at an app from different perspectives, handle it as a user would, and test its limits. While real user testing is undoubtedly beneficial to complement the work of a professional software tester, there’s no replacing their position when it comes to determining quality. Since they’re trained and knowledgeable about the way software works and removed from the development process, they’re in the best position to make decisions about what an application may need from a more technical perspective.

Manual testers are particularly skilled in this area and also have valuable proficiency with tools like CrossBrowserTesting screenshots to compare a website across different browsers and devices. Meanwhile, testers that are familiar with automation techniques can also prove their worth when it comes to usability testing as they know how to leverage a different set of tools, such as Selenium, which can be used to automate tests across different browsers and devices.

Having somebody that’s able to perform more complicated tasks like cross-browser testing to ensure that a website works for any user on any device means that you’re always addressing end-to-end usability without leaving any gaps in quality.

Conclusion

Usability testing is important because, often times, when developers and designers are working on a web application, it’s hard to remove themselves when testing and leave behind their pre-disposed knowledge of how it works to think as an end-user.

Recruiting real users, leveraging data, and employing software testers will give your website the competitive edge it needs in usability whether it’s a new page or a new iteration.

Prioritizing Tests When You Can’t Automate Everything

By Leave a Comment

Prioritizing tests automated toy robots in a line

Though automation is a great method for executing more frequent tests, teams that try to automate 100 percent of their tests will quickly find that it’s a herculean task. GUI tests should be carefully prioritized with regards to business critical processes, revenue generating flows, and common test cases.

As testers new to automation gain more experience, they should be able to prioritize tests, saving organizations time and money.

When Not to Automate

Usability Testing

User experience design (UX) is a human driven evaluation of the usability of a product, application, or feature. This means the UX can’t be evaluated by automation very accurately because that would take out the aspect of human observation. Using our Screenshot Engine, you can visually verify the UX across different devices and browsers, but we recommend serious Usability Testing be done manually.

One-time tests

This one is pretty self-explanatory, but if there’s a feature you only plan to test once or twice, there’s no reason to try and automate it. Automation is primarily used for regression and repeated tests, and best saved for more stable code bases. For example, an ad-hoc test may just be used to check something without the need of extensive documentation and a formal test structure.

Brand new features

Smoke testing and exploratory testing are better for testing a new feature than writing full end-to-end tests for it. Automation is great for reducing testing time, but it doesn’t act as an end-user the way a manual tester is trained to do. Automation scripts won’t seek out new bugs; see below. Once the UI of that feature is stable, it’s in a better state to be tested across browsers and when small code changes are made.

Exploratory testing

Automated tests follow a script, they don’t explore the software to experiment with an application and break features. Unfortunately, since users aren’t robotic they won’t always know to use software the way it’s intended. By exploratory testing, manual testers can go in and consider more creative possibilities of how a user will handle it and execute negative testing.

Tests to Prioritize Automating

Regression tests

Again, the most basic use of automation is for regression tests. Are there certain test scripts that you know you always execute with every deployment? Did you just get to a stable point in your integration tests of a new feature and want to speed testing. That’s where automation comes into greatly reduce the time it takes, thus effectively assisting the ability for quality deployments in a CI/CD environment. Regression testing allows you to build a test suite that ensures quality with every iteration and change to the code, pinpointing where possible new bugs may be when integration systems or deploying new features.

Unit tests

As the base of Mike Cohn’s Test Pyramid, unit tests are great for automation because they’re the smallest kind of test (Hint: you should have a lot of them). This also encompasses the motto of testing “early and often” that’s a foundation of any good testing strategy.

High traffic pages

You want to have a good reputation in the eyes of your users, and a bug or broken link on your homepage can be embarrassing. If your analytics show that your traffic is converting through your webinar section then you should prioritize that page before your blog, making sure customers are able to sign up and join them.

Impact on customers

This is similar but slightly different than targeting highly trafficked pages. Will your customers be more likely to leave your site if they don’t have a good experience during check out compared to browsing inventory pages? Consider pages with high bounces rate, or pages that have a high ROI – like your free trial or contact us page.

Data-driven tests

Manual testing is great for UI, but it’s not a very data-driven approach, more so relying on human intuition and curiosity. If you are looking for specific test inputs to tell you exactly where bugs lie automation will give you a more accurate portrayal.

Cross-browser testing

You’ve tested your web app, but if you only tested in one browser, you’re missing the mark for broader coverage. This is a simple way to make sure that your web application looks acceptable to a wide array of users on different devices, operating systems, and browsers, which is cross-browser testing. However, unless you want this to take hours or days of your valuable time, your best option is to automate parallel tests. To learn more cutting down test time, read our guide to parallel testing.

Conclusion

Deciding between manual and automation can be pretty straight forward. Manual testing is most often used for more detailed human observation, while automated tests are a faster and more data-driven approach for repeated tests.

In addition, sometimes the line can blur between which tests you should execute first in both situations. Our best advice is to always consider the end-user first. Think about which tests will best ensure usability, functionality, and performance for them specific to the stage of development your software is in.

By making both manual and automated testing part of your ongoing QA strategy, you can establish that your software continuously is known for optimal quality among your user base.

How do you decide what tests should be automated first? Tell us how you prioritize in the comments!

The Difference Between Manual and Automated Testing

By Leave a Comment

Manual vs Automated Testing

Manual vs Automated Testing

When you begin web testing, one of the primary decisions you’ll have to make is whether you’re going to test manually or use automated testing.

While neither of these options are technically better or worse than the other, the size, budget and time allowance of a project will certainly be factors that affect which method will work best in your testing procedure.

Below, we’ll outline the best times to use manual and automated testing, and the advantages and disadvantages of each.

Manual Testing

Manual testing is essentially self-explanatory — testing of a web application is done manually, by human action. This means that someone actually goes on a device to evaluate numerous components including design, functionality, and performance by clicking through multiple elements or units of a web application without support from a tool or script.

Advantages

The method of manual testing is inherently valuable to consumer-facing websites because the tester is able to have the full user experience, making it easier to assess visual components such as text size and layout as well as UI.

Additionally, most QA processes will begin with some degree of manual testing because it’s an ideal way to perform one-off testing and exploratory test cases as well as usability and ad-hoc testing.

Disadvantages

The limitations of manual testing are that it’s not the best method for test cases that have frequent repetition. For code that’s updated by the day or hour, manual testing would take an exponential amount of time and human resources, and therefore, would not be an efficient way to approach agile CI.

Furthermore, because manual testing is performed by a human, there’s more room for human error as opposed to the accuracy of automated testing. It can also be a boring, time consuming and tedious task for developers to continually perform manual tests every time they update a code, rather than spending their time actually improving the code itself.

Manual Testing v Automated Testing

Manual Testing v Automated Testing for CI

Automated Testing

Automated testing uses the assistance of tools, scripts and software to perform test cases by repeating pre-defined actions. All automated testing, however, requires some manual testing at first to create initial testing scripts.

Advantages

The biggest draw of automated testing is that it’s noticeably faster and more accurate than manual testing. This is because automation allows multiple tests to be run in parallel, since the test cases can be distributed across different machines.

In comparison, manual testing has to be done sequentially, and so only one test can be run at a time. In this way, the accuracy of automated testing makes it ideal for finding bugs when running tests repeatedly over a long time period, or running multiple tests on different browsers.

Additionally, Automated testing is preferred for large projects that require regression, load and performance testing and repeated execution. Specifically, developers that use an agile CI methodology depend on automated testing to be able to retest scripts as they continually change code because manually testing every time a small fix was made would be exceptionally time consuming.

Disadvantages

The downfall of automated testing is that, unlike manual testing, there is little human observation involved. While automated testing is more efficient in many ways, developers will sometimes find that they need to execute manual testing in order to add a human element and meet specific consumer needs.

Often people will attribute automated testing as being more expensive because it requires a greater investment in tools. However, if your project calls for automated testing, this investment will most likely pay for itself and be more cost effective over time.

automated testing vs manual testing

ROI of Test Automation

Complimenting Testing with Tools

Whether you’re deciding to pursue a manual or automated testing strategy, there are a number of ways to make the process faster, easier and more accurate for your QA team.

Tools such as CrossBrowserTesting offer solutions and features that complement manual and automated testing alike. These value-add tools including live testing, automated screenshots, visual comparisons, selenium testing and local testing all contribute to a well-rounded project execution.

However, the success of your testing process will first depend on your ability to measure which testing method is best for your project. By evaluating the pros and cons of manual testing and automated testing as well as their best uses and what you would like to accomplish through testing, you can launch a solid strategy that will continue to improve your web application and users’ experiences.

To learn more about beginning test automation, check out our eBook.

Read Now

Join Over 200,000 Testers & Developers And Start Testing Today

Learn more Free Trial