CrossBrowserTesting.com

A Design, Development, Testing Blog

Pair Testing: Working Together to Define Quality

By Leave a Comment

pair testing

When you think of software testing, you probably think of an individual sitting at their computer performing manual tests or maybe writing up a Selenium script. In this scenario, they are also the sole person providing feedback on the test case and recording any notes or documentation.

However, pair testing is a little different, and it’s gaining popularity among the testing community.

So what is pair testing, why would you want to do it, and how do you do it right? A

What is Pair Testing?

It takes two to tango, or pair test. Pair testing is usually done with a tester and a developer or two testers, but the tester can pair with anyone they think would make a good partner such as customer support, product management, or even marketing.

While testers don’t usually want people peeking over their shoulder while they do their job, pair testing encourages two people to work together towards share objectives in testing.

The two will work at the same machine with one person controlling the test through the mouse and keyboard and one person giving feedback, asking questions, taking notes, and making observations. Both parties should be taking part in a collaborative discussion about the test scenario.

Pair testing is similar to the idea of pair programming where two developers work together, and one writes the code while the other reviews, ask questions, or makes suggestions. In pair testing, however, both participants are going through the functionality, usability, or visual representation of that code whether it be a new feature or a completely new UI.

In Lisa Crispin’s pair testing blog, she says her partner has likened pair testing to riding in the passenger seat versus driving the car — “You have so much more chance to look around.”

Basically, pair testing is just two people testing the same scenario together instead of one, using their unique skills to provide additional insight to the rest of their team.

Benefits of Pair Testing

  • Cross-Team Collaboration – By combining the unique skills of different team members, you can gain more insight into what is needed for the specific scenario to be successful. Pairing with a developer, for example, means less back and forth between both roles when new code needs to be tested. Working with another tester or a product manager would give another insight. Each of these roles can then work together to decide what is working well and what may need to be changed or fixed.
  • Two Perspectives – When testing, you’re always limited to the fact that not every user journey will be accounted for. The reality is, it’s impossible to think of every possible combination of actions that could be performed. But by having two participants think about a case, it means there’s another set of eyes and brains to consider new possibilities or strategies.
  • Breaking Down Silos – When things are moving fast in Agile development teams, testers can often end up in siloed roles — especially if they’re working on a small or single team. Pair testing helps break down barriers, work with new people, and get testers’ constructive feedback ideas out in the open so each role can better understand where the other fits in and how it leads to quality.
  • Exploratory Testing – Because of the nature of pair testing, it’s usually an exploratory effort. Where exploratory testing is so important for finding bugs and providing feedback on new features, it still doesn’t always get the dedication it deserves. Pair testing encourages teams to make more time for exploratory testing to collect and deliver that valuable feedback.
  • Having a Witness – Sometimes it can be hard to recreate bugs, and then when that happens, make a case that they need to be fixed. But the best way to solve any mystery case is with a witness. By having someone to see bugs as they come up and help determine whether or not it should be considered a bug in the first place, it is easier to pass to the development team with certainty for fixing.
  • Training Opportunity – Not only is pair testing a great way to help the team deliver with an extra measure quality, it’s also a good way for each participant to learn from each other. For newer testers, matching up with a mentor for pair testing is a fun and productive way to teach them what goes into a successful test session. For other roles that may be interested in testing more, this can teach them what to look at. No matter what level or expertise, there’s always something to be learned from your partner in pair testing.

How to do Pair Testing

Pair testing really isn’t that different from other methods of software testing, but there are a few things you may want to do in preparation.

Following these steps will ensure that it’ll be a productive use of both your time.

  1. Identify the test case you want to work on and why you think pair testing is a solution.
  2. Identify who you want to work with and what each person’s role will be. What do you hope to gain by working with this person specifically? Will one person be operating the machine the entire time or will you switch?
  3. Determine an objective from your pair testing session. Do you want to find bugs or suggest new features? Maybe you want to define tests that should be automated — maybe all of the above. Understand who you will be bringing your results to if it’s different than your normal report.
  4. Create a timeline that gives yourselves enough room to explore and meet these objectives while staying focused on the tasks at hand.
  5. Come up with a few specific cases you want to be included in the pair testing. Katrina Clokie suggests writing down a simple checklist of ideas.
  6. Speak your mind and ask questions. Mike Talks says that “pairing is an interactive partnership,” which means both parties are equally contributing thoughts, ideas, and suggestions regularly throughout the test.
  7. Be open to differences of opinion, styles, and techniques. While this can lead to conflicting points of view, it’s always important to respect the other person’s feedback and try to learn from it and accommodate it.
  8. Keep notes and documentation that you can use to share results with other stakeholders.

This, of course, is a very basic process for pair testing, but in real life, the test case, your partner, and objectives will all affect how you go about pair testing. These should give you a good foundation to get started and ask the right questions.

Two is Company

Pair testing is one kind of what Malcolm Isaacs calls “social software testing”. When following an agile methodology and shifting left, it’s crucial for testers, developers, product owners, and other participants in the SDLC to come together for collaboration. Pair testing is just way one to do this.

While it may not be as deeply technical as, say, test automation might be, there are clearly many advantages to having two team members get together to discuss the quality of the application under test — what’s working, what’s not, and what needs to change?

 

 

How to Apply User Personas to Improve Exploratory Testing

By Leave a Comment

persona based testing exploratory testing user personas

As a software tester, your job is to advocate for the end user. However, when there are thousands or even millions of end users, how can we be sure that we’re keeping in mind their differences?

While advocating for each and every user may be an impossibility, adopting persona-based testing may be the key to deriving more accurate insights for your exploratory tests. By paying attention to the details that give different end users unique experiences, you can curate a more intelligent approach to testing for them.

Sometimes personas are supplemented with pictures of the person and detailed descriptions of everything from hobbies to the number of pets they have. While these can help to paint a complete picture, you may want to start with some of the essentials.

In order to begin building your user personas, consider the following when it comes to determining how visitors are using your web application:

  • Age – Does your product or service target a mix of ages, or is it geared toward users that are older or younger? Younger users are more technically savvy — many have grown up using the internet, so navigating the web comes naturally to them from social media to online shopping. However, while they may be more equipped at handling complex features, often their standards are higher and patience is thinner. They will not stick around for complicated forms or slow load times but will leave your site for a sleeker user experience. While older generations may be less impatient, on the other hand, they also may be less likely to get around more intricate components. They also may be on older browsers and operating systems, which could render your application useless if not accounted for. When it comes to making their experience better, you want to think simple and straightforward. Taking into account the range of user behavior solely based on age, it’s easy to see why it’s an important element to consider when thinking of personas.
  • Job/Role – What does the person do, and what are they using it for that will influence their decision? For example, if you own an office supplies website, you may have two 30-year-old women visiting your website in the span of an hour — does that mean they should be tested for the same way? What if one woman is coming to the site as a freelance writer at a small magazine and one is coming in as an office manager for a major enterprise? They’re going to be looking for two different outcomes and have two different experiences. The freelancer may be looking for a few single items such as a pack of pens, a notebook, and a binder. The manager may be looking to bulk order supplies — what happens when she puts in an order for 300 office chairs and 600 monitors? Can the application handle these requests in the same way as a single item? That may be just one difference the two women encounter going through these process, and you want to assess those possibilities.
  • Accessibility – Accessibility can be an overlooked part of software testing, but considering different challenges people may have when it comes to user personas can be helpful. For example, many men are prone to color blindness, which means something as simple as a color choice could make or break the experience for someone. Users that are hard of hearing may rely on image attributes and labeled form fields. Even just having a slight mobility impairment or vision loss could affect the way an application is experienced, especially when it comes to responsive design and mobile device usage. Once you realize these obstacles are common for a variety of people, it becomes important to make accessibility part of your persona-based testing.
  • Geolocation – Looking at demographics is a great way to understand personas, but it’s not just limited to age and gender; it also includes looking at where people are in the world. In fact, where someone’s looking at your site from has a big influence on how they’ll use it. Keep in mind Hofstede’s cultural dimension theory to determine what different cultures prioritize in order to apply that to your personas and application testing. Look at how McDonald’s changes its web design to accommodate a cross-cultural experience to understand how geolocations can apply to persona-based testing. Depending on where you have customers coming from, these preferences may change how you want to relay your website. Even something like network latency will affect how people from different countries experience your application, so you may want to see what it looks like through the eyes of a different country.
  • Behavior – Some personas can’t be pinned down by hard facts — sometimes you have to get creative imagining the ways different people might use your application. For example, knowing someone’s job title won’t tell you that they have a tendency to open multiple tabs and windows at once, keeping a shopping car that’s ready to check out open on their computer for days at a time. It won’t tell you that someone may enter illegal inputs or never remember their password. As critical as it is to collect data on your customers and create personas that accurately represent your users, part of being a great tester is also thinking outside the box to give those personas depth that includes quirky behavior. Oftentimes, it may seem useless to test an action that you wouldn’t expect anyone to perform, but that sequence could be the one that breaks the application…like the computer glitch that let someone buy nothing from McDonald’s for 99 cents.

Conclusion

Persona-based testing helps you step out of your preconceived notions about how people use an application to think about how they could use it.

Recently, we made an infographic about the different personas you should use in exploratory testing. While specific attributes may differ from team to team in order to paint an accurate representation of users, we believe these personas will give you a better understanding of all the ways that people can push an application to its limits.

Check out Exploratory Testing Playbook: 7 Personas Your Team Should Be Testing For and join us on a visual safari to determine who you should be testing for and how to test for them.

Exploratory Testing Techniques for Your E-Commerce Site

By Leave a Comment

exploratory testing techniques ecommerce website testing

Though test automation has been a saving grace for when checking previously written code coverage, we know that there is no substitute for manual testing. Whether you follow the traditional waterfall method or are practicing agile development with your team, exploratory testing plays a critical role.

Anyone who owns or manages an e-commerce website especially knows this to be true. What’s the use of automating a script you derived out of thin air? Instead, exploratory testing with real, human testers provides insight into functionality, performance, and usability of the application.

In order to better understand some of the most important areas to cover when conducting exploratory tests on an e-commerce site, we put together a few steps to consider when live testing.

  1. What happens when someone tries to log in? Is the user directed to the correct page after they logged in and does it load properly? What happens if they put in the wrong information — do they have to re-enter email and password every time or does it stay in the field? What do any error messages say and is it easy to retrieve a forgotten password or use existing login data? Whether your web application prompts a login as soon as someone lands on the home page or just in order to log into their account at payment, you want this to be a seamless experience because if someone has trouble logging in, there’s a good chance they won’t end up purchasing.
  2. How user-friendly are the form fields? If your website is good enough that a customer can easily make it to check out, you don’t want to trip them up on the last few steps of their purchase as the fill out form fields like name, address, and payment information. Unfortunately, however, this is still something many websites have yet to master. Are you asking too much information or not enough? How long does it take to get one from their shopping cart to a finished checkout? Make sure that promotion boxes and purchase total calculations function correctly. Additionally, you’ll want to make sure that all payment processing works, so that customers who are trying to use PayPal won’t encounter an issue that someone charging to their credit card wouldn’t get.
  3. What’s the experience like on mobile? Speaking of form fields, they can pretty much make or break the mobile user experience. You want to make sure that each field takes up most of the screen when a user goes to fill it in, or someone going through checkout might become frustrated at how hard it is to see and type their information to make the purchase. You’ll generally want to make sure the website design is responsive, which may mean implementing different types of features and designs for users that do not access the site on a desktop in order to make it easier to use and look at. Consider all the ways that the buying process differs with a touch screen and a scaled down keyboard. Of course, it’s best to test for mobile on a few real devices to truly understand how these different functions come into play.
  4. How long do different pages take to load? First of all, you want your homepage to load quickly because this is where new and returning traffic will start, and you want to make a good first impression. However, you also have to consider other areas of your application that don’t get the same level of traffic but are equally as important. Does the last product page load as fast as the first or second one? Do some categories perform better than other? What happens in the shopping cart — is the site laggy when it tries to process a user’s input information? Basically, where does the site trip up that could affect the buyer’s journey and deter them from fulfilling their purchase? Don’t forget about load and performance testing.
  5. What happens when you abandon the site and come back? How much progress does the page save and for how long? What about if the page is reloaded? Will someone be logged out or have their cart emptied if they leave for a few minutes? You have to consider the fact that not every user will have the same consistent experience with your application from beginning to end. They might view a product on multiple devices or keep the tab open while they check your competition, and you would hate to lose them when they come back to buy because you haven’t made it easy enough to pick up where they left off.
  6. What could be better from a usability perspective? You know what they say — if it’s not broken, don’t fix it. Except, this isn’t necessarily true when you’re talking about developing and testing a web application. Just because you don’t find a bug in your exploratory tests, doesn’t mean there’s no feedback to give the developers. Consider not just how to fix the application, but what would make it more intuitive for your users. Could the navigation be in a better place or the contact information easier to find? Are there design components that could be more appealing and is the site accessible to a broad range of users? Is there a to make searching for specific products easier such as different a filter? Usability testing will assure your website remains a popular destination.
  7. How functional are different elements? It takes a testers eye to look beyond just the surface of the website to assess whether it really works the way it’s supposed to. Test search algorithms and results. Experiment with different filter combinations. Check calculations and confirm results. Any time a user has to input information or interact with the website, you should be testing whether or not that the application is responding correctly.
  8. What happens after a customer has completed their purchase? You’ve made it through and impressed (or avoided-annoying) a customer long enough. But you also want to make sure their experience is excellent from beginning to end so that they come back next time. Make sure that the page redirects to something thoughtful so that the customer understands their order is being processed. Give them options to review and cancel their order, contact customer support, and access tracking information. Each of these components needs to be tested to make sure that data is accurate and the functions work.

From 0 to 100: How to Go From Manual to Automated Testing

By Leave a Comment

Manual tester to automated testing speedometer

Test automation is all the rage these days, but breaking into the role of an automation engineer is challenging, and sometimes it might seem like more trouble than it’s worth. When there’s still a clear need for manual testing, what’s the point of you uprooting your current ways just to follow the trend?

Though going from manual automated testing may seem daunting, it’s about much more than adding a skill to your LinkedIn profile. In fact, your future career could depend on your automated testing skills.

Why Do I Need Automated Testing?

  • Faster testing – Your current process might be working for you now, but as a tester, you should be striving for more ways to get faster and more effective at your job. And besides, manually testing repeat test cases must get pretty boring. Learning automation lets you get regression testing out of the way so you can focus on the fun stuff, while simultaneously allowing you to test more software in each sprint.
  • More and more teams are automating – Automation has been steadily adopted by companies to improve test efficiency, and today 44 percent of tests are automated on average, according to the SmartBear State of Testing survey. Not only that, but testers broadly believe it will become even more popular the coming years as more organizations shift to Agile.
  • High in demand – Because more teams are getting on board with the practice, test automation skills are becoming increasingly valuable. Testers that know how to code and are familiar with tools such as Selenium and Appium are the ones that will stand out when it comes time to make a career move. If you find yourself in the position where you’re looking for a new career, it’d be beneficial to have at least a base knowledge of automation and the corresponding technologies that these companies are embracing.

4 Steps for Moving to Test Automation

  1. Learn a programming language – As mentioned, learning a programming language is becoming a valuable skill for testers to have. It’s never too late to learn, but it’ll also be the biggest adjustment when adopting automation practices since it wasn’t previously needed for manual testing. In fact, learning to code will probably be one of the most difficult parts of learning test automation, as well. However, test automation is impossible without coding knowledge, so it will be something you need to know. We usually suggest starting with Python since it’s often easiest to pick up for beginners and there are countless resources for learning such as Treehouse or Codecademy. Additionally, other skills like XML, HTML, CSS, XPATH, and SOAP are often preferred.
  2. Adopt the right tools and frameworks – You’ll make the process a lot easier on yourself if you use a combination of frameworks, open-source tools, and third-party tools for automation. Sorry to break it to you, but you can no longer solely rely on your mouse and keyboard. Selenium WebDriver will probably be the foundation of your automated testing efforts as it’s by far the most popular, which is convenient because you can use any of the new languages you learned. You can learn more about getting started with Selenium here. You’ll also want to familiarize yourself with testing frameworks like Nightwatch.JS, Cucumber, and Protractor, libraries like Capybara and Watir, and Continuous Integration tools like Jenkins CI and TeamCity to get the most out of automated testing. On top of that, paid tools like CrossBrowserTesting will make your life a lot easier when it comes to run test across different browsers. Alternatively, you have the option of using tools like Record & Replay if you haven’t learned to write scripts in order to do some lightweight, codeless automated testing.
  3. Start small – The smaller your test cases are, the easier they will be to debug, maintain, and reuse. You don’t want to go crazy with automation and write heavy test cases, or you’ll just make your life harder in the end. Keep them separate and labeled so they’re easy to identify and share in reporting. On that note, you’ll also have to prioritize which tests you want to automate in general. Just because you have this newfound knowledge of automated testing doesn’t mean you should apply it to everything — in fact, it’s virtually impossible to automate everything, and many things are better off being done manually. Polish off those exploratory testing skills when a new feature is released and run risk analysis for which parts of your application should be automated. In addition, you’ll want to create a list of the browsers and devices that are going to be essential to your particular test automation plan.
  4. Back to the basics – Keep practicing, learning, and using your resources. Automated testing might seem intimidating when you’re first getting into it, but all it really takes is time and patience to learn the craft. Also, no matter how good an automation engineer you are, there’s always more to learn, so take comfort in knowing even the experts don’t know it all. Luckily, there’s also a lot of resources at your disposal. Between online guides, forums, conferences, meetups, communities, social media, and even Slack channels, there is no lack of help available at any time when you have a question or are just looking for some guidance. The testing community is robust, active online and offline, with many people who are happy to provide guidance and mentorship.

Additional Resource for Learning Automated Testing

Of course, this is a broad overview of what it takes to go from a manual to an automated tester, and there’s a lot more to actually learning automation than four steps. However, it’s likely that you’ll find the change worth it, both to heighten your career and enhance your abilities.

Discovering Exploratory Testing

By Leave a Comment

exploratory testing

Despite being fairly unstructured, scriptless, and independent of a pass/fail status, exploratory testing is a very important and strategic part of QA. In fact, its focus on discovery and learning in software is proven to be highly valuable when it comes to testing an application for the first time or looking into a new feature from the last integration.

By identifying problems, questions, ideas, and praise through this method of manual testing, exploratory testing helps start a conversation between internal teams and produce improved software.

What is Exploratory Testing?

Exploratory testing is a scriptless testing practice that emphasizes learning and discovery in software and aims to improve quality.

While test cases are not created in advance, exploratory testing still has a very strategic process and requires documentation. Test design and execution are done simultaneously in an ad-hoc manner, although exploratory testing and ad-hoc testing are not the same (we’ll get to that later).

Since exploratory testing requires a human to look at software, evaluate its functionality, and think of new ideas, it’s a manual practice rather than an automated one. It also depends on the skills and knowledge of an independent tester to be able to evaluate where software has issues or could use improvements.

However, just because it’s a manual practice doesn’t mean it can be left behind in the industry move to Agile and CI/CD. Even as teams shift left and transition from manual testing to automation, exploratory testing continues to play a critical role in Agile development.

Exploratory testing is often best for:

  • New websites
  • New feature
  • Unexplored areas
  • Different devices
  • UI testing

Afterwards, these insights are discussed with the rest of the testing and development team members to make changes, fix bugs, or add new features. Additionally, results can be used to raise questions about what’s working well and what’s not to implement in other areas of the software.

Session-Based Testing

Session-based testing is one way to provide more structure in exploratory testing. While you can’t go into it with a script or a definitive test plan, having certain elements in mind like objectives, heuristics and oracles, focus, risks, and follow-up will make sure the process is valuable.

Session-based testing will often include a technique called mind mapping to identify some of these elements in a visual layout. This also helps write better documentation to share with teams after testing is finished.

Establishing a Definition

James Bach and Michael Bolton are attributed as the testers responsible for calling attention to the value of exploratory testing.

However, Cem Kaner coined the term in 1983, defining it as “a style of software testing that emphasizes the personal freedom and responsibility of the individual tester to continually optimize the quality of his/her work by treating test-related learning, test design, test execution, and test result interpretation as mutually supportive activities that run in parallel throughout the project.”

Today, some people suggest that exploratory testing should just be called “testing” since all testing has an element of exploration to it, but we still think that it deserves a place as its own method according to the original definition.

Exploratory Testing vs. Other Testing

The parameters of exploratory testing are often confused or misunderstood. Comparing and contrasting exploratory testing to other types of testing can help clarify its purpose.

Scripted Testing – Scripted testing and exploratory testing are posed as opposites, which helps better understand their definitions. As mentioned, exploratory testing is scriptless and more random, whereas scripted testing follows a set and defined plan written by the tester. This also means that you can’t test something that isn’t planned ahead with scripted testing, whereas exploratory testing encourages testers to discover new regions as they go. This allows testers to act more as the end user and evaluate areas they come across rather than predefined spaces.

Ad-hoc Testing – Exploratory testing and ad-hoc testing are probably compared most often, but they’re two distinct methods of testing. For one, ad-hoc testing doesn’t require documentation, which is an essential part of exploratory testing. Similarly, it’s not well-integrated into team collaboration and has no set goals, objectives, or focus. Though exploratory testing is meant to be done in an ad-hoc style, it also has more set expectations than a being a random test. Additionally, many divert the two because exploratory testing could be said to require more skill, whereas anyone could perform an ad-hoc test.

Can You Automate Exploratory Testing?

Not really. Exploratory testing inherently depends on the individual tester to contribute their skills, knowledge, and insight to new or unevaluated components in an application — something that cannot be done with automation.

Thinking about this in terms of the testing vs checking debate, exploratory testing would be part of “testing” while automated testing is a method of “checking,” meaning that you could not automate it because you can only use automation to check — a tester has to test since a machine cannot provide independent thought.

Tips for Exploratory Testing

  • Have a loose plan – Going into a test with a focus is important in order to come out of the process with valuable feedback on certain objectives. At the same time, the nature of the exploratory test is to address certain issues as they arise and consider those as well. While testers should have a general plan, they should also allow room to improvise and consider unfamiliar elements.
  • Think like an end user – Exploratory testing is an ideal way to advocate for a customer that may be using a website. While developers are familiar with the code they build, it might be more difficult them to think outside the box and use software in a way it’s not optimized for. The job of the tester in this phase is to recognize these areas of weakness and bring them to the attention of the rest of the team. Check out our Exploratory Testing Playbook for ideas on how to test for user personas.
  • Leverage your skills – While exploratory tests are designed and changed as the tester goes along, it doesn’t mean that just anyone can do it correctly. In fact, the results of the test will largely depend on a tester’s skill level. Leveraging a tester’s knowledge of software and previous testing practices will be the most efficient way to find and address problems.

Debunking 6 Automated Testing Myths

By Leave a Comment

Automated testing myths and misconceptions

Automated testing has been the answer for many teams who wanted to speed up and increase their testing efforts, but misconceptions surrounding automation mean that testers don’t always get the most out of their tools.

In an attempt to correct frequent misunderstandings, we’re busting common automated testing myths to give you the truth.

Myth: You should automate 100% of your tests.

Fact: You can’t test everything. Though the ability to automate testing is a great skill, it’s impossible to automate every test. And even if you could, it wouldn’t be an efficient testing solution. Automated tests are based off source code that’s built based on manual tests. Meanwhile, some tests are better off being executed manually. Anytime a new integration is added, a new test has to be done because automated tests will only check previous scripts. Experienced automation engineers know that prioritizing tests and knowing when to automate is just as important as knowing how to automate. However, though you’re not able to automate every test, automation will greatly increase the number of tests you’re able to run.

Myth: Developers should lead automated testing efforts.

Fact: Automation Engineers should drive automation. While it does take programming and coding knowledge, most of the time it’s the testers that are building automated test scripts. As the software industry begins to depend on new testing methods and development processes, test automation architects and engineers will be in high demand. In fact, most of the time having an automation engineer is preferred to a developer because testers have the mindset, training, and specialized skills for different automation tasks, which include much more than just coding.

Myth: Automation is more expensive than manual testing

Fact: If utilized correctly, automation can save money. While the initial investment in automated testing tools and hiring testers may seem more expensive, the ROI of test automation is often worth it since it saves time that would usually be spent on manual testing and increases productivity. For example, if you’re manually running tests that could be automated, it’s costing you in wasted time because your testers could be doing something more constructive. If you think about this scenario over weeks, months, or years, that’s a huge opportunity cost. Test automation is also a cost effective way to extend testing coverage, scale testing efforts, and mitigate risks as your application grows.

Myth: Automated testing is better than manual testing

Fact: Neither automated or manual testing is better or worse than each other, they just have different uses. The most effective testing strategy will find a happy medium between automation and manual testing. Manual testing is better for one-off tests or tests that require human observation, so exploratory tests, smoke tests, and usability testing should all be manual. Automation is usually best for repeated tests or things that need to be frequently tested, such as regression tests. Familiarizing yourself with the different uses and applying them to the appropriate test cases is the best way to get the most out of your time, money, and testing efforts.

Myth: Automated Testing is an easy fix.

Fact: Automated testing isn’t a one-and-done solution. You can’t set up an automated test and expect that you’ll never have to look at it again. In fact, the source code of your automated tests takes a good amount of maintenance to check that it’s doing its job. Not to mention that changing an organization’s testing and development processes will require new training and onboarding to familiarize stakeholders with automation and writing sophisticated tests. Though the ROI of test automation will likely be apparent over time, the transition to a mostly-automated process from a mostly-manual one will take commitment, patience, and financial investment.

Myth: Automation will solve all of your problems.

Fact: Automation is not a silver bullet. Automation engineers experience false positives and flaky builds that come from a lack of human observation, maintenance, and company-wide collaboration. For example, common challenges using Selenium for automated testing might include coding for cases with more complicated elements like dynamic content, multiple tabs, and pop-up windows that are easier to test manually. Testing will still require the specialized skills of a talented testing team to troubleshoot problems, write scripts, manage tools, and maintain test cases. For some best practices of Selenium testing, check out this guide.

Why Your Startup Needs Software Testing

By Leave a Comment

Software testing for startups

The need for a software testing in large companies to protect security, functionality, and other components that can make or break their customers’ trust is obvious. Oftentimes, the focus is so large on testing that enterprises have entire teams of testers employed at different skill levels and specializing in different areas.

But for startups that have a minimal budget for development, nonetheless testing, why should budding businessmen invest in software testers? It may seem like a step that could be skipped without causing much impact, however, entrepreneurs that have been in similar situations know the truth isn’t that simple.

The ROI

Why does software testing matter to a startup? With so many other expenses being extended in sales, marketing, and day-to-day operations at a time where you’re probably just trying to break even, spending limited finances on another department may not seem very strategic at first.

However, there’s a big reason you’ll want a tester on your team, and that reason is quality.

While a few bugs early on in your software’s first years in the public eye might not seem like a huge deal, software defects are known to be costly when they make it past product release. In fact, it’s about 30 times more expensive to fix a bug after release than in design.

Debugging after a user has already noticed a problem is not just expensive, but it’ll cost you in productivity, reputation, and consumer trust as well, which is not something you can afford to lose with so few customers who are closely evaluating what your product has to offer them.

Instead, focusing on releasing high-quality software will create brand loyalty and help you stand out against competitors, making earning and retaining new customers much more attainable.

Hiring Software Testers

You may be asking that if you have a developer who clearly knows how to code and debug software, why do you need to hire a dedicated software tester? Why can’t your developer just test the software?

The biggest reason why having a developer test their own software is problematic is that they’re already so familiar with the application that it’s hard for them to act as an end-user and handle it in a way that it’s not made for.

On the other hand, the job of the tester is to test the limits of an application the way a user might. For example, since the developer built the application, they know how the checkout process is supposed to work and might not think to enter an invalid email like a tester would.

Since testers also have the technical knowledge and know the in’s and out’s of software development, their insight provides a highly accurate approach to the quality of software before it’s released. Developers and testers have two different skill sets, and each should be utilized for the job they’re best at in order to create a seamless software development life cycle.

Also, since developers obviously spend a lot of their time actually coding, it doesn’t provide them much time to be testing, too — or at least not as much as they should be. At the very least, there should always be one tester working alongside the developer to make sure appropriate testing can be done on the software at different stages of development and at every integration.

Additionally, if you’ve been hearing a lot about automated testing and think it may be a resource to replace actually hiring a human tester, you’d be sadly mistaken. Automated tests are great for following commands and pre-written tests, but won’t be able to find problems in unexplored regions of the software. This means that in order to do any testing, manual or automated, you’ll have to have a human that knows how to write tests.

The Tests Your Startup Should Be Doing

Below are the few of the different testing methods you’ll want to be doing, even at a startup:

Exploratory Testing

For a new application, exploratory testing is extremely important for laying a groundwork of quality. Exploratory testing is an ad-hoc process where the tester manually goes through the software in an attempt to find bugs.

Again, not just anyone can properly execute effective exploratory testing as there is usually a strict process and specific guidelines to follow. Exploratory testing is largely used to define and evaluate the usability of the app as a customer would experience it, and having an expert manually test the app is the best way to evaluate components like functionality and usability.

Regression Testing

Testing is not a one-off process — you can’t just run one test and be able to release new iterations whenever you want from then on. In fact, every time a change is made to the code, you’re going to want to run a regression test to make sure it hasn’t broken previously functioning features.

This is a great example of where you might have to utilize both manual and automated testing. Doing the same test over and over will take up a big chunk of time, so automation is usually a preferred way to compliment manual efforts. On the other hand, manually creating a test script and running it is necessary before automating regression tests. Either way, regression tests are important to make sure a bug doesn’t pop up after adding a new feature.

Performance Testing

When thinking about quality, fast performance is one of the best ways to establish a loyal user, but bad performance will just as quickly turn potential users away. When a 2-second delay in load time during a transaction results in abandonment rates of up to 87 percent, those are losses that you can’t afford.

You want to make sure your application loads quickly and can handle expected traffic. This is the best way to avoid frustrating your users and prevent them from leaving your website before they make it to check out.

Responsive Design

One of the most crucial details that can easily go un-examined without having a software tester that knows what to look for is the idea of responsive design.

Your website will most likely be accessed from an array of different devices and browsers — desktops, laptops, tablets, and smartphones on Chrome, Safari, or even the looming Internet Explorer. Gone untested, the user experience may be drastically different on each of these configurations, which isn’t a good thing.

One of the worst things you can do is go without optimizing your application for mobile devices — the web page simply won’t survive. Finding a tester who knows the importance of cross-browser testing and knows how to find inconsistencies in responsive design will be one of the most valuable assets your startup could invest in.

Conclusion

It’s pretty simple — when you’re a new player in the game, quality is what’s going to set your apart. Don’t make the mistake of skimping out of testing your product to save money because, in the end, it’ll probably cost you customers.

Hiring testers will ensure that someone is monitoring potential issues and catching bugs before release. While it’s nearly impossible to prevent every bug, having a testing strategy in place creates a safeguard to affect better software and happier customers.

The 4 Best Usability Testing Methods

By Leave a Comment

usability testing

Usability testing assesses how easy it is for a user to get from A to B. That’s to say, if someone is using your application, it should be simple for them to accomplish the action they’re looking to complete.

For example, if you’re using a website to buy a computer, how straightforward is the process from when you land on the website through checkout? Is navigation clear, does the application load quickly enough, do buttons do what they’re supposed to, are informational forms uncomplicated? Additionally, you have to consider visual design elements that can either make or break the user experience.

The usability of a web application embodies this entire user experience, and usability testing evaluates everything from performance and UI/UX to responsive design. It ensures that current and potential customers are happy with your application and will continue to come back.

Usability Testing Methods

While there are many components to consider when thinking about optimum usability, there are a few methods that will give you the best glimpse of your website and how well it functions.

User Testing

User testing requires having real people use your app to see how well it functions from a nonbiased and unfamiliar perspective. You can either recruit family and friends to try using the app or find focus groups that are willing to explore different features and functionalities.

While it should be manageable to find people who are willing to test your application and give feedback, there are also companies who are dedicated to setting up user testing services for you. For example, UserTesting provides videos of real users in your target audience navigating your web or mobile app and sharing their thoughts. In addition, the service provides metrics and written feedback so you can reference a real user analysis as you integrate changes.

No matter whether you seek out your own users or hire a company, watching how real people use and interact with software is highly beneficial to understanding user stories and insights.

Surveying

Getting feedback from users can be just as valuable after they’ve completed their task as during. By implementing customer satisfaction surveys after someone has purchased a project or completed a trial on your website, they can focus on fulfilling their task and evaluate the overall process once it’s finished.

SurveyMonkey is one of the top questionnaire development companies, and they even supply questions and templates specifically for website feedback. Rather than only looking at whether or not a user converted to a customer, finding out why they did not can help you target pain points and make improvements.

As customer support begins to take the forefront of successful business models, being able to address feedback straight from the source is invaluable. By actually listening to what users have to say about what they’re looking for in something and what it’s lacking, you can make changes that in turn will elicit customer loyalty and create a product they’re more likely to advocate for.

A/B Testing

A/B testing is where you share two versions of one product (version A and version B) to see which one performs better. These changes can be complete website redesigns or something as minimal as a button saying “Click here” v.s. “Sign up” or changing it between red and green. Almost anything can be A/B tested to find out what engages the most customers.

Using analytics, you then compare which one users had a better experience with. This can be based on the time on page, bounce rate, or most commonly, conversion rate. Additionally, you can also request customer feedback asking why users preferred one version over the other.

Similar to user testing and customer surveys, there are companies you can use for complete A/B testing such as Optimizely. By showing half of you traffic the original web page design and half the modified design and then analyzing responses through a statistical engine, Optimizely helps you make data-informed decisions about your web page to determine what the end-user prefers. Ultimately, A/B testing helps you make changes that contribute to the best version of your app.

Leave it to the Experts

While the recent testing trends have been revolving around automation, there’s still a critical place for manual testing, especially when it comes to usability.

Software testers are able to look at an app from different perspectives, handle it as a user would, and test its limits. While real user testing is undoubtedly beneficial to complement the work of a professional software tester, there’s no replacing their position when it comes to determining quality. Since they’re trained and knowledgeable about the way software works and removed from the development process, they’re in the best position to make decisions about what an application may need from a more technical perspective.

Manual testers are particularly skilled in this area and also have valuable proficiency with tools like CrossBrowserTesting screenshots to compare a website across different browsers and devices. Meanwhile, testers that are familiar with automation techniques can also prove their worth when it comes to usability testing as they know how to leverage a different set of tools, such as Selenium, which can be used to automate tests across different browsers and devices.

Having somebody that’s able to perform more complicated tasks like cross-browser testing to ensure that a website works for any user on any device means that you’re always addressing end-to-end usability without leaving any gaps in quality.

Conclusion

Usability testing is important because, often times, when developers and designers are working on a web application, it’s hard to remove themselves when testing and leave behind their pre-disposed knowledge of how it works to think as an end-user.

Recruiting real users, leveraging data, and employing software testers will give your website the competitive edge it needs in usability whether it’s a new page or a new iteration.

Deciding What to Automate When You Can’t Test Everything

By Leave a Comment

 

prioritizing when you can't automate everything

Though automation is a great method for executing more frequent tests, teams that try to automate everything will quickly find that it’s a herculean task. GUI tests should be carefully prioritized with business-critical critical processes, revenue-generating flows, and common test cases.

As testers new to automation gain more experience, they should be able to prioritize tests, saving organizations time and money.

When Not to Automate

Usability Testing

User experience design (UX) is a human-driven evaluation of the usability of a product, application, or feature. This means the UX can’t be evaluated by automation very accurately because that would take out the aspect of human observation. Using our Screenshot Engine, you can visually verify the UX across different devices and browsers, but we recommend serious Usability Testing be done manually.

One-time tests

This one is pretty self-explanatory, but if there’s a feature you only plan to test once or twice, there’s no reason to try and automate it. Automation is primarily used for regression and repeated tests, and best saved for more stable code bases. For example, an ad-hoc test may just be used to check something without the need for extensive documentation and a formal test structure.

Brand new features

Smoke testing and exploratory testing are better for testing a new feature than writing full end-to-end tests for it. Automation is great for reducing testing time, but it doesn’t act as an end-user the way a manual tester is trained to do. Automation scripts won’t seek out new bugs; see below. Once the UI of that feature is stable, it’s in a better state to be tested across browsers and when small code changes are made.

Exploratory testing

Automated tests follow a script, they don’t explore the software to experiment with an application and break features. Unfortunately, since users aren’t robotic they won’t always know to use software the way it’s intended. By exploratory testing, manual testers can go in and consider more creative possibilities of how a user will handle it and execute negative testing.

What Test Cases to Automate

Regression tests

Again, the most basic use of automation is for regression tests. Are there certain test scripts that you know you always execute with every deployment? Did you just get to a stable point in your integration tests of a new feature and want to speed testing. That’s where automation comes into greatly reduce the time it takes, thus effectively assisting the ability for quality deployments in a CI/CD environment. Regression testing allows you to build a test suite that ensures quality with every iteration and change to the code, pinpointing where possible new bugs may be when integration systems or deploying new features.

Unit tests

As the base of Mike Cohn’s Test Pyramid, unit tests are great for automation because they’re the smallest kind of test (Hint: you should have a lot of them). This also encompasses the motto of testing “early and often” that’s a foundation of any good testing strategy.

High traffic pages

You want to have a good reputation in the eyes of your users, and a bug or broken link on your homepage can be embarrassing. If your analytics show that your traffic is converting through your webinar section then you should prioritize that page before your blog, making sure customers are able to sign up and join them.

Impact on customers

This is similar but slightly different than targeting highly trafficked pages. Will your customers be more likely to leave your site if they don’t have a good experience during check out compared to browsing inventory pages? Consider pages with high bounces rate, or pages that have a high ROI – like your free trial or contact us page.

Data-driven tests

Manual testing is great for UI, but it’s not a very data-driven approach, more so relying on human intuition and curiosity. If you are looking for specific test inputs to tell you exactly where bugs lie automation will give you a more accurate portrayal.

Cross-browser testing

You’ve tested your web app, but if you only tested in one browser, you’re missing the mark for broader coverage. This is a simple way to make sure that your web application looks acceptable to a wide array of users on different devices, operating systems, and browsers, which is cross-browser testing. However, unless you want this to take hours or days of your valuable time, your best option is to automate parallel tests. To learn more cutting down test time, read our guide to parallel testing.

Stop Trying to Automate Everything

Deciding between manual and automation can be pretty straightforward. Manual testing is most often used for more detailed human observation, while automated tests are a faster and more data-driven approach for repeated tests.

In addition, sometimes the line can blur between which tests you should execute first in both situations. Our best advice is to always consider the end-user first. Think about which tests will best ensure usability, functionality, and performance for them specific to the stage of development your software is in.

By making both manual and automated testing part of your ongoing QA strategy, you can establish that your software continuously is known for optimal quality among your user base.

How do you decide what tests should be automated first? Tell us how you prioritize in the comments!

Getting Started with Manual Testing

By Leave a Comment

Manual Testing

Manual testing is a must

What exactly is Manual Testing?

When any tester, developer, or QA team begins a functional or performance testing process for the first time, they’ll likely start with manual testing. Regardless of the size of the company or web application, manual testing is an essential component of any testing strategy because it gives deep insight into the user experience.

This is because manual testing is done completely by human hand, without the assistance of automation tools or scripts of any kind. For this reason, manual testing is the ideal way to assess the details that are relevant to the end-user including how the application functions, how well visual elements render, and other GUI qualities.

Manual testing is specifically valuable in exploratory testing or other test cases that are only run once or twice and ensures that you discover bugs and other issues early on in the development cycle. These tests are an optimal environment for incorporating human observation without an urgent need for the speed and in-depth accuracy that’s characteristic of automated testing.

Manual testing also allows testers to build test cases, which are needed for automation so that you can repeat those actions. No matter how much time you’re willing to put into manual testing, it’s the first step to creating a script that can be run in parallel in other browsers, or many times over.

For this reason, it’s crucial to understand some of the stages and techniques before executing manual tests.

Types of Software Tests

These stages of testing are sequential, starting with the smallest component of the application and progressing to make sure that all parts are working together.

Unit

Unit testing is at the bottom of the testing pyramid because, as the smallest testable part of an application, it focuses on one individual function within the entire application. Although this type of test is better performed in an automated manner – it’s still the base of our pyramid and most plentiful type of tests out there.

Clicking a button to make sure it takes you to the correct destination would be a unit test. For example, making sure that the “Next page” button will actually take to you the next page.

Integration

Integration testing is the next step after unit testing and is the process of integrating multiple units to test them together. This lets testers observes how different components of the software harmonize to ensure functionality.

An example of an integrated test could be your applications log-in process. Now, you’re testing whether entering a username and password and clicking login will bring you to the Welcome screen. These types of tests match two or more unit tests together.

It is important to have an inverse amount of tests as you go down the testing pyramid. This allows for easy debugging and error locating, speeding up both testing and development. Have unit tests along with integration and other test lets you diagnose the exact location of bugs.

System

A system test refers to testing completely integrated software in order to evaluate that system meets the requirements of the other hardware or software systems that it’ll come in contact with.

This is actually a series of multiple tests including end-to-end scenario testing, verifying inputs and output and testing the user experience. There are also multiple types of system testing such as stress, regression and functional testing, the uses of which will differ from tester to tester depending on available time and resources.

Again, all the components that would be tested in a system test have already passed integration testing. System testing after is necessary for testing the design and behavior of the software.

Acceptance

Acceptance testing or User Acceptance testing is the beta testing of the software performed by actual end users. This, of course, is to bring the product into a real life environment to see how well it measures up to a user’s standards in case the tester misses an essential user function, specification or request.

As you can see, manual testing requires a thorough and sequential completion of these stages in order to properly measure a software’s functionality at different points in the testing process in order to target or fix various areas.

Black Box vs White Box Testing

Black box testing and white box testing are two techniques of testing that can be applied to the above stages.

Black box testing is where the internal structure of the software being tested isn’t known to the test, while the internal structure is known within white box testing.

Black box testing is the most common for testers because they don’t have to be expert in coding, making it easier and simpler to execute. It’s also quicker to do because testers only have to look at the GUIs that users are interacting with and looking at.

However, black box testing has restricted coverage in this way and can only evaluate a percentage of every possible test case or scenario, rather it just considers inputs and outputs.

In comparison, white box testing is a little more in-depth and looks at the programmatic structure of the system. In this way, it allows the developer or tester to spot errors and bugs quicker and more accurately and provides a more thorough insight.

Keeping Manual in Your Testing Strategy

While the testing environment is largely moving to automated testing as today’s developers focus on a strategy that embodies agile CI, there will always be a very specific and important place for manual testing. Those hoping to move towards 100 percent automation will find that their results will fall short from complete and efficient web development.

Instead, getting to know your code through the methods and techniques of manual testing will prove to be invaluable when it comes to present and future web testing practices.

Join Over 200,000 Testers & Developers And Start Testing Today

Learn more Free Trial