manual testing Archives | CrossBrowserTesting.com

Manual Cross-browser Testing for Beginners Part 2: Tools and How to

April 23, 2020 By Sarah Mischinger Leave a Comment

Welcome back to the second part of our manual cross-browser testing tutorial for beginners! In the last part of this series, we talked about what cross-browser tests are, why you need to conduct them, and what different types of tests we know. If you haven’t read the first part yet, we encourage you to do so now.

Today, we’re going to delve deeper into the How of cross-browser testing, and also take a look at some prerequisites and tools that help you test more efficiently. So, without further ado, let’s get started!

How to choose browsers for testing

In our previous blog post, we learned that cross-browser testing can help ensure that your web application looks and works perfectly on different browsers, browser versions, and devices. But let’s take a second to think about all the different combinations you could theoretically test. It’s hardly possible to test on every browser and device out there! Hence, you should spend some time figuring out which combinations to check.

First, you need some usage data to find out which browsers and browser versions are the most popular used in the market as well as the ones visiting your website.

You usually have two options here:

Analyze publicly available data on websites like this one
Analyze existing usage data of your web app (e.g., from Google Analytics)

Don’t forget to include mobile devices and their browsers in your research!

Use the data found to create a browser matrix that everyone on your team can understand and use when cross-browser testing. For this, create an Excel or Numbers spreadsheet that contains at least the following information:

Browser name
Browser version
OS
Device
Support (yes, no, or partially)

Depending on how many browsers you’ve listed for testing, you may need to prioritize them. Especially if you’re testing manually, you probably won’t be able to check every browser and device on your list when time is running out. Therefore, you should prioritize the entries in your browser matrix according to popularity, so that you can ensure that the majority of your users can enjoy a thoroughly tested web app.

Set up your ideal test infrastructure

After defining your browser matrix, you need to find a way to access all of these different browsers and devices. Here, you also have various options to choose from:

In-house device and browser lab

Your company may already have an internal lab that offers everything you need. If you are considering building a lab on-premises, first make sure that enough resources are available to purchase and maintain the necessary components of your infrastructure.

Emulators and simulators

To check your web app in mobile browsers, you can install and run emulators and simulators on your machine. These tools mimic real devices and are a great choice if you want to review your app during development. However, they are not the “real deal” and can’t completely replace testing on real devices. If you want to learn more about the pros and cons, read this blog post from our friends at Bitbar.

Virtual machines

You can install a virtual machine (VM for short) on your computer and use different images to run foreign operating systems. So, for example, if you are a Mac user and need to test your web app on Microsoft Edge, you can run Windows 10 in a Virtual Machine. Yet, VMs take up a lot of disk space – the more OS versions you need to run, the more images and space you need. Also, your computer may struggle with running another OS, which can cause severe lag, among other things. Therefore, this option may not be suitable for you.

Online services

The most promising option for most testers is to use an online-based test service that provides all the browsers, browser versions, and devices you need to test. Typically, these services are subscription-based but can save you a lot of resources and headaches since you don’t have to purchase and maintain the infrastructure and tools yourself.

How to conduct manual cross-browser tests

After you’ve set up a suitable infrastructure to access all of the combinations required for testing, it is time to think about the actual testing process. Let’s find out what usually works quite well:

Set a baseline for comparison

First of all, you should test your app in a modern, reliable and popular browser like the latest stable version of Google Chrome. This way, you can find and fix some of the more apparent bugs and get a clear idea of how the app should look and work. If you then test in other browsers, you can compare and evaluate the results using this baseline.

Create a cross-browser test strategy

You need a document that contains guidelines for cross-browser testing that everyone should work with. If you don’t have such a plan on hand, you may forget to test critical functions or follow different practices than your colleagues. As a result, you may invest a lot of time in tests that lead to unsatisfactory results.

In this test strategy document, describe what should be tested, how often it must be tested, and what should not be tested. Use your product’s test specification or create one if necessary. Additionally, you should include your browser matrix and add some information about the infrastructure used. Read our blog post “Real Device and Browser Testing Strategy for 2020” to learn more about what to include here.

Execution

After completing all of the above steps, you can finally get the show on the road and execute your tests manually according to your test strategy!

Tools that make manual cross-browser tests more efficient

Since manual testing can lead to bottlenecks that hinder the timely release of your web applications, we at CrossBrowserTesting have developed some tools that help you to test more efficiently:

Live testing

The Live Testing tool lets you perform interactive manual and exploratory tests on more than 2050 desktop and mobile browsers. You get access to browser extensions and inspection tools that you can use to debug your CSS, HTML, and JavaScript code.

Visual Testing

Don’t forget to check what your application looks like in addition to how it works – especially when it comes to your responsive website design. With our Visual testing tool, you can take full-page, chromeless, or above-the-fold screenshots of your web app’s layout on different devices at the same time. We also offer you an API that you can use to schedule screenshots.

Record and Replay

Would you like to get test automation into your test mix without having to write a single line of code? Then our Record and Replay tool is the perfect choice to eliminate the pain of manually executing repetitive test cases. Record your test flow and replay it in parallel on all desired browsers to shorten test execution time while increasing browser coverage.

Conclusion

In the second and last part about manual cross-browser testing for beginners, we gave you an idea of how to run cross-browser tests, what you need to do in advance, and which tools can make testing more efficient.

We hope that we’ve been able to provide you with some useful and valuable knowledge that will help you get a flying start with manual cross-browser testing!

Manual Cross-browser Testing for Beginners Part 1: Concepts and Types

March 26, 2020 By Sarah Mischinger Leave a Comment

You have probably noticed that automated testing receives a lot of love and hype right now. That’s because they can accelerate test execution times and make room for other, more pressing tasks. However, manual testing is still very much relevant as we cannot automate everything (not yet at least). For this reason, we want to take a closer look at the craft with a two-part series about manual cross-browser testing for beginners. In this first blog post, we’ll talk about what manual testing is, why and when you need it, and what types of cross-browser testing need to be done manually.

What is cross-browser testing?

When you develop a website or web app, you most likely use one or two browsers for development, such as Chrome and Firefox. Maybe you resize the browser window to see if the component you are creating adapts correctly to different screen sizes.

However, do all of your users browse the web with your development browsers? And do they solely use their laptop or PC to do so? As you probably know too well, the answer is no. There are many different browsers and versions out there, and your users often use their tablets or mobile devices to visit your website. In fact, the use of mobile internet has increased by 50 4% since 2011!

By conducting cross-browser tests, you can ensure your web app works and looks splendid on many different browsers, browser versions, and devices. The goal is to offer as many users as possible a consistent and positive experience with your online product.

Who performs these tests?

The answer to this question can vary from company to company since everyone has set up their own workflows with the available staff and resources. But let’s see who is most likely to run cross-browser tests:

Web Developers – Naturally, the people who implement the app under test are often best suited for the job. They already know what to look for, how to report and deal with errors, and are also capable of resolving any issues they encounter.
Quality Assurance teams (short QA) – Especially big companies have set up a QA department that helps make sure products meet specific criterias and quality standards. Usually, the staff is trained to conduct tests and report bugs back to the developers.
Web/UI/UX Designers – The designer of your web app should always ensure that the result matches their crafted design. However, since they check the overall look and feel of the web app, they don’t need to test on every browser and device and often limit their testing efforts on one of each.

Note: In general, anyone can help test your app. This often depends on the type of test you want to conduct. For example, if you need to verify your app’s intuitiveness, you can invite a group of external people to represent your target audience and watch them use your app.

Why should you test your website or web app?

In recent years, websites have become more and more complex. In fact, we mostly refer to them as web apps these days – a term which people hardly used ten years ago. Software that users previously had to install on their computers has been transformed into online services, and the web is no longer just a provider of information.

Imagine visiting your favorite blog and seeing that components such as the primary navigation don’t work in your browser. You may be a little annoyed, but things could be worse. However, if you need to fill out an essential form on your government’s web portal, and the same bug occurs, things aren’t looking so rosy anymore. For better or worse, web apps are an integral part of our daily lives we depend on.

Your web app users also expect your product to work flawlessly anytime, anywhere. That’s why you need to conduct thorough cross-browser tests! If you can’t provide users with a working app, they may be looking for an alternative. In the worst case, your company loses customers (and ultimately money).

When and what should you test manually?

Basically, we know two different ways to carry out cross-browser tests: manually and automatically. Since we’re covering manual testing in this blog series, we’re not going to talk too much about test automation, but include the basics to give you an idea of the differences.

Manual cross-browser testing

Here, a tester interacts personally with the web app on all browsers and devices. For this, they can follow a pre-written list of things to test and report any errors they encounter to a system or developer (depending on the company’s workflows).

Automatic cross-browser testing

Testers write test scripts and run them automatically on all required browsers and devices by using automation tools such as Selenium WebDriver. We recently published a blog post about when and what to test automatically, which you can read here.

Naturally, manual testing takes a lot of time – especially if your app is quite complex and must be checked in many different browser versions. Test automation helps reduce the needed time and allows you to focus more on analysis, troubleshooting, and other more pressing matters.

However, manual tests are still required! Some tests are too complicated or computationally intensive for automation, but testers can efficiently perform them manually. Most security components like Captchas cannot be tested automatically either, because it would make no sense if someone could bypass these security mechanisms through automation.

Besides, test automation is overkill for some projects. If you are working on a small and relatively simple website, writing test scripts can take more time than running the test manually. And if your company doesn’t have qualified staff to write these tests, you may also need to opt for manual testing until the situation changes.

What types of manual cross-browser tests are there?

Let’s complete this first blog post on manual cross-browser testing by taking a quick look at the types of tests that you can conduct manually.

Exploratory testing

This type of test is always done manually, as a human must freely discover the app to find UX vulnerabilities and bugs. Even if you could achieve 100% test automation, you would still miss errors that only occur under certain unpredictable circumstances. With exploratory testing, you can find out how users interact with your app and how it responds in different situations. Yet, since you don’t have a checklist to guide you through testing, it can be challenging to know where to start or what to look for. That’s why we have created a playbook to help you discover some of the types of personas you should test for.

Visual Testing

Visual tests ensure that your app looks and feels good in different browsers, browser versions, and devices. Of course, you need to check if your CSS animations and JavaScript-driven components work appropriately under different circumstances. Additionally, designers and QA teams can ensure that the web app meets the expected quality and that the implementation matches the design.

Functional Testing

Looks aren’t everything, and the most stunning web app is worthless if UI components don’t respond as expected. As the name suggests, functional tests help you make sure that users can actually use your app. For example, you can test whether your contact form checks the email field correctly and whether the user can submit forms if the required fields are empty.

Note: While you can’t automate exploratory tests, teams can save time by automating repetitive test cases and tests that can expect a specific result.

Conclusion

In this first part about manual cross-browser testing for beginners, we made sure to provide you with the basic knowledge of the craft. Now you should know about the different types of tests and who will perform them and when. We hope to see you again in the second part of this blog series, where we will talk about how to perform manual tests and what tools you can use to make the process more efficient.

Balancing Your Automation Strategy with Manual Testing

September 21, 2018 By Alex McPeak Leave a Comment

You’ve heard it a million times before — you can’t automate everything. But if your organization is just starting into the swing of automation, they’re probably thinking about all the ways they can automate. So how do you make sure you maintain the proper balance of manual testing and automation?

Whether you have an entire QA team whose roles are split between automated and manual testing, or there is one tester whose jobs is to do both, striking the right balance will be the difference between a testing strategy that provides value to your team and one that gives you more frustration than feedback.

By understanding how to best support your manual strategy with automation, you can ensure you’re getting the best of both world.

Know the Limitations of Automation

No matter how skilled you are in automation, how many Selenium commands you know, or what testing frameworks you use, some things simply cannot be automated.

For example, CAPTCHAs are often very popular with ecommerce sites. However, they’re inherently impossible to automate. This means that any test case that includes a CAPTCHA can’t be automated, and manual testing has to step in.

This, of course, isn’t the only case where you won’t be able to use automation. There are plenty of instances that it will be possible to automate, but it will be so complex to script, that you’re better off doing it manually.

Assess the limitations and challenges of test automation so you know where to step in with manual testing.

Solidify Your Regression Suite

The bottom line for automation is that you want to automate tests that are repetitive and time-consuming, so think about which tests you find yourself going through the most often. Solidifying your regression suite will help you determine which tests you want to cover after every integration or change to code.

Keep in mind that this will change over time as your application changes, but it’s helpful to have a suite that ensures basic functionality.

Additionally, deciding which environments you want to test on will be key to understanding which browsers, devices, and operating systems will be included in your regression suite.

Don’t Sweat the Small Stuff

Small test cases are best for automation and give you tests that are reusable and maintainable. The results of your automation are dependent on the state of your application, which means the smaller and more specific the test is, the more resistant it will be to changes in the UI. This is why unit and integration tests are ideal for automation.

But longer, more complex test cases and entire user journeys are often better left for manual testing. This gives you more room to explore the application and requires less time for scripting. Moreover, it means that if the UI of the application changes, you won’t have to go back into your script and figure out where it needs to be adjusted.

What’s the Risk?

When you’re thinking about whether or not to automate, risk should be a leading factor for consideration. However, while there are many methods for evaluating which tests are the most at risk, identifying these doesn’t mean you want to leave all other test cases that aren’t high risk out completely. In fact, these are perfect for manual testing.

When you considering risk, there are usually three tiers that you sort test cases into — high, medium, and low. As we discussed, most high-risk test cases will probably be included in your automation suite.

But, if you’re finding medium and low-risk cases that you still find valuable to test in order to provide feedback, then they might be something you want to run through manually.

Is It Providing Value?

Automation is a great way to speed up testing, but it also takes time for coding and maintenance, and that may not be worth it if you implement a test that isn’t adding value.

Say you run a test suite every day, and it passes 100 percent of the time. Is it worth it to keep automating if your results never change?

Just because a test was once automated doesn’t mean it should stay that way. Automation requires making changes, updates, and edits as the application code changes.

If a test is no longer providing value through automation, it might be better to take it back to manual testing to see if there are new parts to explore.

Communicate With Developers

Developers can’t tell you what to automate and what not to automate, but they can give you insight into the state of the application.

Keep open lines of communication with developers and other team members so that you are clear on what’s being changed in the application during development cycles. Depending on whether integrations are minor changes to the code, completely new features, or a revamped UI will make a difference in whether the test that follows will be part of your automation suite or should be tested manually.

To get additional insight, it may be helpful to pair with developers when going through the application to get a more in-depth look at what has changed and what areas or functions need to be tested.

Exploratory Testing

Exploratory testing is crucial to visually and functionally verifying new features and changes to the UI. Automation frees you up to test parts that you wouldn’t normally get to, so take it as a challenge to find a new bug or test for a different persona.

Human observation, curiosity, and creativity are paramount to being a tester. Automation can only take you so far when it comes to finding new bugs. That’s why manual testing, and especially exploratory testing, will always have a place in the QA process.

Visual Testing

Functional automated testing is not the only thing you have to worry about. Oftentimes, a functional test will pass because your application is working correctly, but it could still be visually inconsistent across browsers. Not only could this affect your layout and design, it could also affect the way your users are able to interact with your application.

In addition to functional testing with Selenium, you should incorporate automated screenshot testing into your workflow. This will help you account for visual cross-compatibility issues that you may have missed with functional testing by allowing you to compare screenshots of your web application side-by-side in different browsers.

Increasing Test Coverage

Automation will take care of repeated sequences, but in order to increase test coverage, you want to use manual testing to look at areas that have not yet been tested.

In this way, automation can be used as a tool to gradually help you increase test coverage over time. Test high-level functionality with automation, then fill in the gaps with manual testing. Look at edge cases and try negative testing to go beyond pass and fail results to gain deeper insight into your application.

This is also a good time to figure out what new tests you would like to add into your automation suite, so you can continue to increase coverage. Evaluate manual test cases to determine whether they would be valuable in regression, and assess which tests may be missing from the suite already.

Considering ROI

At the end of the day, it’s best to consider return on investment when deciding whether or not to automate. Automating takes time to analyze, code, and maintain.

If the time it takes to automate isn’t worth the results that you’d get from the test, it may be something you want to do manually, or not at all.

What Robotic Restaurants Can Teach Us About Test Automation

August 28, 2018 By Alex McPeak Leave a Comment

To most people, going out to eat doesn’t evoke the image of robots. But to the four founders of this restaurant, robot chefs are the only viable solution.

In this automated kitchen, robots begin cooking your meal in temperature controlled pans as soon as you submit your order. The bots a programmed to spin the ingredients and spices around until everything’s perfectly cooked. Then, it dumps your meal into a compostable bowl.

While the restaurant may seem overrun by restaurants when you look around, there are more humans involved than you think.

Once in the bowl, a person takes it and adds yogurts, sauces, and dry toppings, then hands you your finished product. Behind the scenes, the Culinary Director and Executive Chef are the masterminds behind the delicious recipes, making sure the finished meals have the right balance of flavor. Additionally, there are people prepping the ingredients. Of course, there are also the four engineers behind the robots making sure that everything runs smoothly.

So what does this have to do with software testing?

In the shift from manual to automated testing, we often get ahead of ourselves in trying to automate as much as possible, or even attempting to automate everything.

But just like a fast-casual restaurant cannot operate on robotics alone, neither can we rely on automation to fulfill 100 percent of our testing. There are certain qualities that testers possess that we can’t automate.

So as we continue to explore the ways that automation can aid in our testing efforts, here are a few takeaways we can gain from robotic restaurants:

A Good Recipe is Key – Just as we need a chef to create the recipes for the robotic sous chef to follow, there needs to be the first test for automation to follow. Whether this is done through record & replay or by writing out a script, the quality of the first test is crucial to designing a suite that will provide insightful results. Just like a robot can’t follow a recipe it isn’t yet programmed to make and can’t create one on its own, a test automation tool can only test what it is told. Additionally, every bowl is only as good as the recipe, just as every regression test is only as good as the first test or script.
Robots Don’t Have an Eye for Detail – Just like humans are the final step in garnishing your meal at a robotic restaurant, human testers have an eye for detail that is critical in testing. There are many soft skills that software testers have — empathy, creativity, curiosity, and knowledge of the software, for example — that are impossible to teach to an automation system. This is why it’s important to have real people that can perform exploratory tests on new features, layouts, or other noticeable application updates, so they can bring unique perception and understanding to provide valuable insight. Not to mention the customer service aspect — communication is important that there’s someone on the team advocating for testing as well as the end users and who’s able to talk to other stakeholders.
There is Risk Associated with Automation – Automation usually accounts for “normal” situations but doesn’t consider edge cases or stress. What happens if a customer has an issue with their bowl? What happens if a concert gets out, and the influx of customers is 10 times what it normally is? What happens if you run out of utensils? These are the instances where automation can no longer solve the problem, and a human needs to step in. Automation can’t go outside of what it’s told, which means there will always be risk involved when expected situations arise.
Automation is Best for Repeated Actions – A McKinsey report said the food industry is most susceptible to automation because there are so many predictable physical activities involved. Similarly, during code changes and new integrations, it’s important to repeat some of the same tests run previously to make sure they work. This is where automation becomes a good candidate for testing — regressions that have programmable repetition that helps to reduce human error. Rather than have someone perform the same actions over and over, having a machine take over the tester has more time to add new tests is what will result faster productivity in less time.
Cut Costs Without Cutting Jobs – Because of reduced labor costs, the robotic kitchen’s bowls start at $7.50 — around $3 less than comparable options. If you’re getting to the point where your solo tester doesn’t have time to run all the regression tests they need across browsers in addition to new exploratory tests, introducing an automation solution can alleviate some of those time-consuming tasks and give you a higher ROI. However, if you implement automation, there needs to be a human behind that as well. It’s also important to have someone who understands automation and the tools in order to properly execute and maintain tests.

The Biggest Takeaway

You can’t automate everything. Though you can add significant automation to a project, there will always be a place for human testers. Additionally, automation should be used as a tool to support manual work rather than replace it.

Interested in hearing more about how test automation is like a robot restaurant. Listen to Episode 6, Manual vs Automated…Restaurants? of SmartBear’s podcast series “The Good, the Bad, and the Buggy” with an interview with Paul Grizzaffi.

Exploratory Testing Techniques for Mobile Devices

July 26, 2018 By Alex McPeak Leave a Comment

For those of us who use our devices for everything, from buying coffee to connecting with friends, the mobile experience holds high priority despite smaller screens and limited capabilities.

Due to the sheer popularity of smartphones and tablets, the importance of mobile device testing doesn’t come as much of a surprise. As mobile testing becomes the industry standard, testers must increasingly incorporate test cases that consider the needs of users that are on the go.

However, while you may be familiar with which tests to perform with a keyboard and mouse, you have to come up with a new strategy for mobile devices, especially during exploratory testing.

Here are a few ways to test mobile devices when manual testing:

Touch Screen – Scrolling, clicking, and swiping are different when you’re using your fingers compared to a computer mouse. You want to perform a variety of these actions throughout your application to ensure that you can do everything on mobile that you’re able to do on a desktop and that it functions with the same accuracy. Are buttons big enough to click? Is it easy to scroll from top to bottom and back? How does it look when you zoom in or out? Does swiping work without too much lag? These considerations will be critical in assessing usability.
Keypad – Rather than they keyboard we’re all used to, mobile devices have much smaller on-screen versions that also use touchscreen technology, which can pose a difficulty to users. Because they’re generally less user-friendly, you want to look at instances that might use require the keypad such as search boxes. Pay special attention to form fields to make sure they’re not too narrow or hard to use. You don’t want your users becoming frustrated because of a keypad that makes using the app impossible.
Navigation – Many times certain navigations will be different for mobile compared to desktop because of the smaller screen space. For example, where a drop-down menu might normally be labeled “Menu” on the web application, it might turn into a hamburger menu on a phone screen. Ensure that these icons and elements are organized in a way that makes their function apparent to the user. Because some of these navigation items may be less obvious, you want to go through and make sure that you’re still able to accomplish the same journey.
Portrait and Landscape – When you’re testing on a desktop, the web page will always be horizontal. Mobile phones are most often in portrait mode, but they can also be turned to view in landscape. Visual testing will allow you to evaluate and compare responsive layouts to make sure images, text, media, and other content is digestible in both portrait and landscape modes on mobile.
Devices – It’s not enough to test on one mobile device — in fact, you should probably be testing a selection of devices. Because of mobile fragmentation, different models, brands, screen sizes, and resolutions mean different mobile experiences. Additionally, you want to test on both iOS and Android operating systems to get a better understanding of how it differs for each.
Performance – Performance may be in good shape on your desktop, but that doesn’t mean it’s consistent for mobile. When 85% of mobile users expect pages to load as fast or faster than on the desktop, testing for performance is not something you want to skip. Test out different pages and record their load time. You also might want to test out different networks, locations, and even battery charges to see how it affects speed.
Popups and Alerts – Checking pages that have popups and alerts is important because while it may look fine on a desktop, it could be intrusive on mobile. Alerts that are too large and unresponsive or popups that are layered over each other may even make the app unusable. Additionally, it’s a good idea to see how phone calls, notifications, and other interruptions affect the functionality application.

Once you know which devices you want to test on, you can build out a more comprehensive mobile testing strategy. Keep in mind the ways that devices differ, both from each other (tablet vs Android vs iPhone) and from desktop, to determine which test cases you should focus on during exploratory testing.

Your users aren’t ditching their devices any time soon. Exploratory testing on mobile devices means that you can better understand how users interact with your application.

Pair Testing: Working Together to Define Quality

June 22, 2018 By Alex McPeak Leave a Comment

When you think of software testing, you probably think of an individual sitting at their computer performing manual tests or maybe writing up a Selenium script. In this scenario, they are also the sole person providing feedback on the test case and recording any notes or documentation.

However, pair testing is a little different, and it’s gaining popularity among the testing community.

So what is pair testing, why would you want to do it, and how do you do it right? A

What is Pair Testing?

It takes two to tango, or pair test. Pair testing is usually done with a tester and a developer or two testers, but the tester can pair with anyone they think would make a good partner such as customer support, product management, or even marketing.

While testers don’t usually want people peeking over their shoulder while they do their job, pair testing encourages two people to work together towards share objectives in testing.

The two will work at the same machine with one person controlling the test through the mouse and keyboard and one person giving feedback, asking questions, taking notes, and making observations. Both parties should be taking part in a collaborative discussion about the test scenario.

Pair testing is similar to the idea of pair programming where two developers work together, and one writes the code while the other reviews, ask questions, or makes suggestions. In pair testing, however, both participants are going through the functionality, usability, or visual representation of that code whether it be a new feature or a completely new UI.

In Lisa Crispin’s pair testing blog, she says her partner has likened pair testing to riding in the passenger seat versus driving the car — “You have so much more chance to look around.”

Basically, pair testing is just two people testing the same scenario together instead of one, using their unique skills to provide additional insight to the rest of their team.

Benefits of Pair Testing

Cross-Team Collaboration – By combining the unique skills of different team members, you can gain more insight into what is needed for the specific scenario to be successful. Pairing with a developer, for example, means less back and forth between both roles when new code needs to be tested. Working with another tester or a product manager would give another insight. Each of these roles can then work together to decide what is working well and what may need to be changed or fixed.
Two Perspectives – When testing, you’re always limited to the fact that not every user journey will be accounted for. The reality is, it’s impossible to think of every possible combination of actions that could be performed. But by having two participants think about a case, it means there’s another set of eyes and brains to consider new possibilities or strategies.
Breaking Down Silos – When things are moving fast in Agile development teams, testers can often end up in siloed roles — especially if they’re working on a small or single team. Pair testing helps break down barriers, work with new people, and get testers’ constructive feedback ideas out in the open so each role can better understand where the other fits in and how it leads to quality.
Exploratory Testing – Because of the nature of pair testing, it’s usually an exploratory effort. Where exploratory testing is so important for finding bugs and providing feedback on new features, it still doesn’t always get the dedication it deserves. Pair testing encourages teams to make more time for exploratory testing to collect and deliver that valuable feedback.
Having a Witness – Sometimes it can be hard to recreate bugs, and then when that happens, make a case that they need to be fixed. But the best way to solve any mystery case is with a witness. By having someone to see bugs as they come up and help determine whether or not it should be considered a bug in the first place, it is easier to pass to the development team with certainty for fixing.
Training Opportunity – Not only is pair testing a great way to help the team deliver with an extra measure quality, it’s also a good way for each participant to learn from each other. For newer testers, matching up with a mentor for pair testing is a fun and productive way to teach them what goes into a successful test session. For other roles that may be interested in testing more, this can teach them what to look at. No matter what level or expertise, there’s always something to be learned from your partner in pair testing.

How to do Pair Testing

Pair testing really isn’t that different from other methods of software testing, but there are a few things you may want to do in preparation.

Following these steps will ensure that it’ll be a productive use of both your time.

Identify the test case you want to work on and why you think pair testing is a solution.
Identify who you want to work with and what each person’s role will be. What do you hope to gain by working with this person specifically? Will one person be operating the machine the entire time or will you switch?
Determine an objective from your pair testing session. Do you want to find bugs or suggest new features? Maybe you want to define tests that should be automated — maybe all of the above. Understand who you will be bringing your results to if it’s different than your normal report.
Create a timeline that gives yourselves enough room to explore and meet these objectives while staying focused on the tasks at hand.
Come up with a few specific cases you want to be included in the pair testing. Katrina Clokie suggests writing down a simple checklist of ideas.
Speak your mind and ask questions. Mike Talks says that “pairing is an interactive partnership,” which means both parties are equally contributing thoughts, ideas, and suggestions regularly throughout the test.
Be open to differences of opinion, styles, and techniques. While this can lead to conflicting points of view, it’s always important to respect the other person’s feedback and try to learn from it and accommodate it.
Keep notes and documentation that you can use to share results with other stakeholders.

This, of course, is a very basic process for pair testing, but in real life, the test case, your partner, and objectives will all affect how you go about pair testing. These should give you a good foundation to get started and ask the right questions.

Two is Company

Pair testing is one kind of what Malcolm Isaacs calls “social software testing”. When following an agile methodology and shifting left, it’s crucial for testers, developers, product owners, and other participants in the SDLC to come together for collaboration. Pair testing is just way one to do this.

While it may not be as deeply technical as, say, test automation might be, there are clearly many advantages to having two team members get together to discuss the quality of the application under test — what’s working, what’s not, and what needs to change?

How to Apply User Personas to Improve Exploratory Testing

February 12, 2018 By Alex McPeak Leave a Comment

As a software tester, your job is to advocate for the end user. However, when there are thousands or even millions of end users, how can we be sure that we’re keeping in mind their differences?

While advocating for each and every user may be an impossibility, adopting persona-based testing may be the key to deriving more accurate insights for your exploratory tests. By paying attention to the details that give different end users unique experiences, you can curate a more intelligent approach to testing for them.

Sometimes personas are supplemented with pictures of the person and detailed descriptions of everything from hobbies to the number of pets they have. While these can help to paint a complete picture, you may want to start with some of the essentials.

In order to begin building your user personas, consider the following when it comes to determining how visitors are using your web application:

Age – Does your product or service target a mix of ages, or is it geared toward users that are older or younger? Younger users are more technically savvy — many have grown up using the internet, so navigating the web comes naturally to them from social media to online shopping. However, while they may be more equipped at handling complex features, often their standards are higher and patience is thinner. They will not stick around for complicated forms or slow load times but will leave your site for a sleeker user experience. While older generations may be less impatient, on the other hand, they also may be less likely to get around more intricate components. They also may be on older browsers and operating systems, which could render your application useless if not accounted for. When it comes to making their experience better, you want to think simple and straightforward. Taking into account the range of user behavior solely based on age, it’s easy to see why it’s an important element to consider when thinking of personas.
Job/Role – What does the person do, and what are they using it for that will influence their decision? For example, if you own an office supplies website, you may have two 30-year-old women visiting your website in the span of an hour — does that mean they should be tested for the same way? What if one woman is coming to the site as a freelance writer at a small magazine and one is coming in as an office manager for a major enterprise? They’re going to be looking for two different outcomes and have two different experiences. The freelancer may be looking for a few single items such as a pack of pens, a notebook, and a binder. The manager may be looking to bulk order supplies — what happens when she puts in an order for 300 office chairs and 600 monitors? Can the application handle these requests in the same way as a single item? That may be just one difference the two women encounter going through these process, and you want to assess those possibilities.
Accessibility – Accessibility can be an overlooked part of software testing, but considering different challenges people may have when it comes to user personas can be helpful. For example, many men are prone to color blindness, which means something as simple as a color choice could make or break the experience for someone. Users that are hard of hearing may rely on image attributes and labeled form fields. Even just having a slight mobility impairment or vision loss could affect the way an application is experienced, especially when it comes to responsive design and mobile device usage. Once you realize these obstacles are common for a variety of people, it becomes important to make accessibility part of your persona-based testing.
Geolocation – Looking at demographics is a great way to understand personas, but it’s not just limited to age and gender; it also includes looking at where people are in the world. In fact, where someone’s looking at your site from has a big influence on how they’ll use it. Keep in mind Hofstede’s cultural dimension theory to determine what different cultures prioritize in order to apply that to your personas and application testing. Look at how McDonald’s changes its web design to accommodate a cross-cultural experience to understand how geolocations can apply to persona-based testing. Depending on where you have customers coming from, these preferences may change how you want to relay your website. Even something like network latency will affect how people from different countries experience your application, so you may want to see what it looks like through the eyes of a different country.
Behavior – Some personas can’t be pinned down by hard facts — sometimes you have to get creative imagining the ways different people might use your application. For example, knowing someone’s job title won’t tell you that they have a tendency to open multiple tabs and windows at once, keeping a shopping car that’s ready to check out open on their computer for days at a time. It won’t tell you that someone may enter illegal inputs or never remember their password. As critical as it is to collect data on your customers and create personas that accurately represent your users, part of being a great tester is also thinking outside the box to give those personas depth that includes quirky behavior. Oftentimes, it may seem useless to test an action that you wouldn’t expect anyone to perform, but that sequence could be the one that breaks the application…like the computer glitch that let someone buy nothing from McDonald’s for 99 cents.

Conclusion

Persona-based testing helps you step out of your preconceived notions about how people use an application to think about how they could use it.

Recently, we made an infographic about the different personas you should use in exploratory testing. While specific attributes may differ from team to team in order to paint an accurate representation of users, we believe these personas will give you a better understanding of all the ways that people can push an application to its limits.

Check out Exploratory Testing Playbook: 7 Personas Your Team Should Be Testing For and join us on a visual safari to determine who you should be testing for and how to test for them.

Exploratory Testing Techniques for Your E-Commerce Site

December 13, 2017 By Alex McPeak Leave a Comment

Though test automation has been a saving grace for when checking previously written code coverage, we know that there is no substitute for manual testing. Whether you follow the traditional waterfall method or are practicing agile development with your team, exploratory testing plays a critical role.

Anyone who owns or manages an e-commerce website especially knows this to be true. What’s the use of automating a script you derived out of thin air? Instead, exploratory testing with real, human testers provides insight into functionality, performance, and usability of the application.

In order to better understand some of the most important areas to cover when conducting exploratory tests on an e-commerce site, we put together a few steps to consider when live testing.

What happens when someone tries to log in? Is the user directed to the correct page after they logged in and does it load properly? What happens if they put in the wrong information — do they have to re-enter email and password every time or does it stay in the field? What do any error messages say and is it easy to retrieve a forgotten password or use existing login data? Whether your web application prompts a login as soon as someone lands on the home page or just in order to log into their account at payment, you want this to be a seamless experience because if someone has trouble logging in, there’s a good chance they won’t end up purchasing.
How user-friendly are the form fields? If your website is good enough that a customer can easily make it to check out, you don’t want to trip them up on the last few steps of their purchase as the fill out form fields like name, address, and payment information. Unfortunately, however, this is still something many websites have yet to master. Are you asking too much information or not enough? How long does it take to get one from their shopping cart to a finished checkout? Make sure that promotion boxes and purchase total calculations function correctly. Additionally, you’ll want to make sure that all payment processing works, so that customers who are trying to use PayPal won’t encounter an issue that someone charging to their credit card wouldn’t get.
What’s the experience like on mobile? Speaking of form fields, they can pretty much make or break the mobile user experience. You want to make sure that each field takes up most of the screen when a user goes to fill it in, or someone going through checkout might become frustrated at how hard it is to see and type their information to make the purchase. You’ll generally want to make sure the website design is responsive, which may mean implementing different types of features and designs for users that do not access the site on a desktop in order to make it easier to use and look at. Consider all the ways that the buying process differs with a touch screen and a scaled down keyboard. Of course, it’s best to test for mobile on a few real devices to truly understand how these different functions come into play.
How long do different pages take to load? First of all, you want your homepage to load quickly because this is where new and returning traffic will start, and you want to make a good first impression. However, you also have to consider other areas of your application that don’t get the same level of traffic but are equally as important. Does the last product page load as fast as the first or second one? Do some categories perform better than other? What happens in the shopping cart — is the site laggy when it tries to process a user’s input information? Basically, where does the site trip up that could affect the buyer’s journey and deter them from fulfilling their purchase? Don’t forget about load and performance testing.
What happens when you abandon the site and come back? How much progress does the page save and for how long? What about if the page is reloaded? Will someone be logged out or have their cart emptied if they leave for a few minutes? You have to consider the fact that not every user will have the same consistent experience with your application from beginning to end. They might view a product on multiple devices or keep the tab open while they check your competition, and you would hate to lose them when they come back to buy because you haven’t made it easy enough to pick up where they left off.
What could be better from a usability perspective? You know what they say — if it’s not broken, don’t fix it. Except, this isn’t necessarily true when you’re talking about developing and testing a web application. Just because you don’t find a bug in your exploratory tests, doesn’t mean there’s no feedback to give the developers. Consider not just how to fix the application, but what would make it more intuitive for your users. Could the navigation be in a better place or the contact information easier to find? Are there design components that could be more appealing and is the site accessible to a broad range of users? Is there a to make searching for specific products easier such as different a filter? Usability testing will assure your website remains a popular destination.
How functional are different elements? It takes a testers eye to look beyond just the surface of the website to assess whether it really works the way it’s supposed to. Test search algorithms and results. Experiment with different filter combinations. Check calculations and confirm results. Any time a user has to input information or interact with the website, you should be testing whether or not that the application is responding correctly.
What happens after a customer has completed their purchase? You’ve made it through and impressed (or avoided-annoying) a customer long enough. But you also want to make sure their experience is excellent from beginning to end so that they come back next time. Make sure that the page redirects to something thoughtful so that the customer understands their order is being processed. Give them options to review and cancel their order, contact customer support, and access tracking information. Each of these components needs to be tested to make sure that data is accurate and the functions work.

From 0 to 100: How to Go From Manual to Automated Testing

October 16, 2017 By Alex McPeak Leave a Comment

Test automation is all the rage these days, but breaking into the role of an automation engineer is challenging, and sometimes it might seem like more trouble than it’s worth. When there’s still a clear need for manual testing, what’s the point of you uprooting your current ways just to follow the trend?

Though going from manual automated testing may seem daunting, it’s about much more than adding a skill to your LinkedIn profile. In fact, your future career could depend on your automated testing strategy.

Why Do I Need Automated Testing?

Faster testing – Your current process might be working for you now, but as a tester, you should be striving for more ways to get faster and more effective at your job. And besides, manually testing repeat test cases must get pretty boring. Learning automation lets you get regression testing out of the way so you can focus on the fun stuff, while simultaneously allowing you to test more software in each sprint.
More and more teams are automating – Automation has been steadily adopted by companies to improve test efficiency, and today 44 percent of tests are automated on average, according to the SmartBear State of Testing survey. Not only that, but testers broadly believe it will become even more popular the coming years as more organizations shift to Agile.
High in demand – Because more teams are getting on board with the practice, test automation skills are becoming increasingly valuable. Testers that know how to code and are familiar with tools such as Selenium and Appium are the ones that will stand out when it comes time to make a career move. If you find yourself in the position where you’re looking for a new career, it’d be beneficial to have at least a base knowledge of automation and the corresponding technologies that these companies are embracing.

4 Steps for Moving to Test Automation

Learn a programming language – As mentioned, learning a programming language is becoming a valuable skill for testers to have. It’s never too late to learn, but it’ll also be the biggest adjustment when adopting automation practices since it wasn’t previously needed for manual testing. In fact, learning to code will probably be one of the most difficult parts of learning test automation, as well. However, test automation is impossible without coding knowledge, so it will be something you need to know. We usually suggest starting with Python since it’s often easiest to pick up for beginners and there are countless resources for learning such as Treehouse or Codecademy. Additionally, other skills like XML, HTML, CSS, XPATH, and SOAP are often preferred.
Adopt the right tools and frameworks – You’ll make the process a lot easier on yourself if you use a combination of frameworks, open-source tools, and third-party tools for automation. Sorry to break it to you, but you can no longer solely rely on your mouse and keyboard. Selenium WebDriver will probably be the foundation of your automated testing approach as it’s by far the most popular, which is convenient because you can use any of the new languages you learned. You can learn more about getting started with Selenium here. You’ll also want to familiarize yourself with testing frameworks like Nightwatch.JS, Cucumber, and Protractor, libraries like Capybara and Watir, and Continuous Integration tools like Jenkins CI and TeamCity to get the most out of automated testing. On top of that, paid tools like CrossBrowserTesting will make your life a lot easier when it comes to run test across different browsers. Alternatively, if you haven’t learned to write scripts you can use Record & Replay for lightweight, codeless automated testing.
Start small – The smaller your test cases are, the easier they will be to debug, maintain, and reuse. You don’t want to go crazy with automation and write heavy test cases, or you’ll just make your life harder in the end. Keep them separate and labeled so they’re easy to identify and share in reporting. On that note, you’ll also have to prioritize which tests you want to automate in general. Just because you have this newfound knowledge of automated testing doesn’t mean you should apply it to everything — in fact, it’s virtually impossible to automate everything, and many things are better off being done manually. Polish off those exploratory testing skills when a new feature is released and run risk analysis for which parts of your application should be automated. In addition, you’ll want to create a list of the browsers and devices that are going to be essential to your particular test automation plan.
Back to the basics – Keep practicing, learning, and using your resources. Automated testing might seem intimidating when you’re first getting into it, but all it really takes is time and patience to learn the craft. Also, no matter how good an automation engineer you are, there’s always more to learn, so take comfort in knowing even the experts don’t know it all. Luckily, there’s also a lot of resources at your disposal. Between online guides, forums, conferences, meetups, communities, social media, and even Slack channels, there is no lack of help available at any time when you have a question or are just looking for some guidance. The testing community is robust, active online and offline, with many people who are happy to provide guidance and mentorship.

Additional Resource for Learning Automated Testing

Of course, this is a broad overview of what it takes to go from a manual to an automated tester, and there’s a lot more to actually learning automation than four steps. However, it’s likely that you’ll find the change worth it, both to heighten your career and enhance your abilities.

Discovering Exploratory Testing

August 22, 2017 By Alex McPeak Leave a Comment

Despite being fairly unstructured, scriptless, and independent of a pass/fail status, exploratory testing is a very important and strategic part of QA. In fact, its focus on discovery and learning in software is proven to be highly valuable when it comes to testing an application for the first time or looking into a new feature from the last integration.

By identifying problems, questions, ideas, and praise through this method of manual testing, exploratory testing helps start a conversation between internal teams and produce improved software.

What is Exploratory Testing?

Exploratory testing is a scriptless testing practice that emphasizes learning and discovery in software and aims to improve quality.

While test cases are not created in advance, exploratory testing still has a very strategic process and requires documentation. Test design and execution are done simultaneously in an ad-hoc manner, although exploratory testing and ad-hoc testing are not the same (we’ll get to that later).

Since exploratory testing requires a human to look at software, evaluate its functionality, and think of new ideas, it’s a manual practice rather than an automated one. It also depends on the skills and knowledge of an independent tester to be able to evaluate where software has issues or could use improvements.

However, just because it’s a manual practice doesn’t mean it can be left behind in the industry move to Agile and CI/CD. Even as teams shift left and transition from manual testing to automation, exploratory testing continues to play a critical role in Agile development.

Exploratory testing is often best for:

New websites
New feature
Unexplored areas
Different devices
UI testing

Afterwards, these insights are discussed with the rest of the testing and development team members to make changes, fix bugs, or add new features. Additionally, results can be used to raise questions about what’s working well and what’s not to implement in other areas of the software.

Session-Based Testing

Session-based testing is one way to provide more structure in exploratory testing. While you can’t go into it with a script or a definitive test plan, having certain elements in mind like objectives, heuristics and oracles, focus, risks, and follow-up will make sure the process is valuable.

Session-based testing will often include a technique called mind mapping to identify some of these elements in a visual layout. This also helps write better documentation to share with teams after testing is finished.

Establishing a Definition

James Bach and Michael Bolton are attributed as the testers responsible for calling attention to the value of exploratory testing.

However, Cem Kaner coined the term in 1983, defining it as “a style of software testing that emphasizes the personal freedom and responsibility of the individual tester to continually optimize the quality of his/her work by treating test-related learning, test design, test execution, and test result interpretation as mutually supportive activities that run in parallel throughout the project.”

Today, some people suggest that exploratory testing should just be called “testing” since all testing has an element of exploration to it, but we still think that it deserves a place as its own method according to the original definition.

Exploratory Testing vs. Other Testing

The parameters of exploratory testing are often confused or misunderstood. Comparing and contrasting exploratory testing to other types of testing can help clarify its purpose.

Scripted Testing – Scripted testing and exploratory testing are posed as opposites, which helps better understand their definitions. As mentioned, exploratory testing is scriptless and more random, whereas scripted testing follows a set and defined plan written by the tester. This also means that you can’t test something that isn’t planned ahead with scripted testing, whereas exploratory testing encourages testers to discover new regions as they go. This allows testers to act more as the end user and evaluate areas they come across rather than predefined spaces.

Ad-hoc Testing – Exploratory testing and ad-hoc testing are probably compared most often, but they’re two distinct methods of testing. For one, ad-hoc testing doesn’t require documentation, which is an essential part of exploratory testing. Similarly, it’s not well-integrated into team collaboration and has no set goals, objectives, or focus. Though exploratory testing is meant to be done in an ad-hoc style, it also has more set expectations than a being a random test. Additionally, many divert the two because exploratory testing could be said to require more skill, whereas anyone could perform an ad-hoc test.

Can You Automate Exploratory Testing?

Not really. Exploratory testing inherently depends on the individual tester to contribute their skills, knowledge, and insight to new or unevaluated components in an application — something that cannot be done with automation.

Thinking about this in terms of the testing vs checking debate, exploratory testing would be part of “testing” while automated testing is a method of “checking,” meaning that you could not automate it because you can only use automation to check — a tester has to test since a machine cannot provide independent thought.

Tips for Exploratory Testing

Have a loose plan – Going into a test with a focus is important in order to come out of the process with valuable feedback on certain objectives. At the same time, the nature of the exploratory test is to address certain issues as they arise and consider those as well. While testers should have a general plan, they should also allow room to improvise and consider unfamiliar elements.
Think like an end user – Exploratory testing is an ideal way to advocate for a customer that may be using a website. While developers are familiar with the code they build, it might be more difficult them to think outside the box and use software in a way it’s not optimized for. The job of the tester in this phase is to recognize these areas of weakness and bring them to the attention of the rest of the team. Check out our Exploratory Testing Playbook for ideas on how to test for user personas.
Leverage your skills – While exploratory tests are designed and changed as the tester goes along, it doesn’t mean that just anyone can do it correctly. In fact, the results of the test will largely depend on a tester’s skill level. Leveraging a tester’s knowledge of software and previous testing practices will be the most efficient way to find and address problems.