CrossBrowserTesting.com

A Design, Development, Testing Blog

Balancing Your Automation Strategy with Manual Testing

By Leave a Comment

balance between manual and automationbalance between manual and automation

You’ve heard it a million times before — you can’t automate everything. But if your organization is just starting into the swing of automation, they’re probably thinking about all the ways they can automate. So how do you make sure you maintain the proper balance of manual testing and automation?

Whether you have an entire QA team whose roles are split between automated and manual testing, or there is one tester whose jobs is to do both, striking the right balance will be the difference between a testing strategy that provides value to your team and one that gives you more frustration than feedback.

By understanding how to best support your manual strategy with automation, you can ensure you’re getting the best of both world.

Know the Limitations of Automation

No matter how skilled you are in automation, how many Selenium commands you know, or what testing frameworks you use, some things simply cannot be automated.

For example, CAPTCHAs are often very popular with ecommerce sites. However, they’re inherently impossible to automate. This means that any test case that includes a CAPTCHA can’t be automated, and manual testing has to step in.

This, of course, isn’t the only case where you won’t be able to use automation. There are plenty of instances that it will be possible to automate, but it will be so complex to script, that you’re better off doing it manually.

Assess the limitations and challenges of test automation so you know where to step in with manual testing.

Solidify Your Regression Suite

The bottom line for automation is that you want to automate tests that are repetitive and time-consuming, so think about which tests you find yourself going through the most often. Solidifying your regression suite will help you determine which tests you want to cover after every integration or change to code.

Keep in mind that this will change over time as your application changes, but it’s helpful to have a suite that ensures basic functionality.

Additionally, deciding which environments you want to test on will be key to understanding which browsers, devices, and operating systems will be included in your regression suite.

Don’t Sweat the Small Stuff

Small test cases are best for automation and give you tests that are reusable and maintainable. The results of your automation are dependent on the state of your application, which means the smaller and more specific the test is, the more resistant it will be to changes in the UI. This is why unit and integration tests are ideal for automation.

But longer, more complex test cases and entire user journeys are often better left for manual testing. This gives you more room to explore the application and requires less time for scripting. Moreover, it means that if the UI of the application changes, you won’t have to go back into your script and figure out where it needs to be adjusted.

What’s the Risk?

When you’re thinking about whether or not to automate, risk should be a leading factor for consideration. However, while there are many methods for evaluating which tests are the most at risk, identifying these doesn’t mean you want to leave all other test cases that aren’t high risk out completely. In fact, these are perfect for manual testing.

When you considering risk, there are usually three tiers that you sort test cases into — high, medium, and low. As we discussed, most high-risk test cases will probably be included in your automation suite.

But, if you’re finding medium and low-risk cases that you still find valuable to test in order to provide feedback, then they might be something you want to run through manually.

Is It Providing Value?

Automation is a great way to speed up testing, but it also takes time for coding and maintenance, and that may not be worth it if you implement a test that isn’t adding value.

Say you run a test suite every day, and it passes 100 percent of the time. Is it worth it to keep automating if your results never change?

Just because a test was once automated doesn’t mean it should stay that way. Automation requires making changes, updates, and edits as the application code changes.

If a test is no longer providing value through automation, it might be better to take it back to manual testing to see if there are new parts to explore.

Communicate With Developers

Developers can’t tell you what to automate and what not to automate, but they can give you insight into the state of the application.

Keep open lines of communication with developers and other team members so that you are clear on what’s being changed in the application during development cycles. Depending on whether integrations are minor changes to the code, completely new features, or a revamped UI will make a difference in whether the test that follows will be part of your automation suite or should be tested manually.

To get additional insight, it may be helpful to pair with developers when going through the application to get a more in-depth look at what has changed and what areas or functions need to be tested.

Exploratory Testing

Exploratory testing is crucial to visually and functionally verifying new features and changes to the UI. Automation frees you up to test parts that you wouldn’t normally get to, so take it as a challenge to find a new bug or test for a different persona.

Human observation, curiosity, and creativity are paramount to being a tester. Automation can only take you so far when it comes to finding new bugs. That’s why manual testing, and especially exploratory testing, will always have a place in the QA process.

Visual Testing

Functional automated testing is not the only thing you have to worry about. Oftentimes, a functional test will pass because your application is working correctly, but it could still be visually inconsistent across browsers. Not only could this affect your layout and design, it could also affect the way your users are able to interact with your application.

In addition to functional testing with Selenium, you should incorporate automated screenshot testing into your workflow. This will help you account for visual cross-compatibility issues that you may have missed with functional testing by allowing you to compare screenshots of your web application side-by-side in different browsers.

Increasing Test Coverage

Automation will take care of repeated sequences, but in order to increase test coverage, you want to use manual testing to look at areas that have not yet been tested.

In this way, automation can be used as a tool to gradually help you increase test coverage over time. Test high-level functionality with automation, then fill in the gaps with manual testing. Look at edge cases and try negative testing to go beyond pass and fail results to gain deeper insight into your application.

This is also a good time to figure out what new tests you would like to add into your automation suite, so you can continue to increase coverage. Evaluate manual test cases to determine whether they would be valuable in regression, and assess which tests may be missing from the suite already.

Considering ROI

At the end of the day, it’s best to consider return on investment when deciding whether or not to automate. Automating takes time to analyze, code, and maintain.

If the time it takes to automate isn’t worth the results that you’d get from the test, it may be something you want to do manually, or not at all.

 

What Robotic Restaurants Can Teach Us About Test Automation

By Leave a Comment

robotic restaurants test automation

To most people, going out to eat doesn’t evoke the image of robots. But to the four founders of this restaurant, robot chefs are the only viable solution.

In this automated kitchen, robots begin cooking your meal in temperature controlled pans as soon as you submit your order. The bots a programmed to spin the ingredients and spices around until everything’s perfectly cooked. Then, it dumps your meal into a compostable bowl.

While the restaurant may seem overrun by restaurants when you look around, there are more humans involved than you think.

Once in the bowl, a person takes it and adds yogurts, sauces, and dry toppings, then hands you your finished product. Behind the scenes, the Culinary Director and Executive Chef are the masterminds behind the delicious recipes, making sure the finished meals have the right balance of flavor. Additionally, there are people prepping the ingredients. Of course, there are also the four engineers behind the robots making sure that everything runs smoothly.

So what does this have to do with software testing?

In the shift from manual to automated testing, we often get ahead of ourselves in trying to automate as much as possible, or even attempting to automate everything.

But just like a fast-casual restaurant cannot operate on robotics alone, neither can we rely on automation to fulfill 100 percent of our testing. There are certain qualities that testers possess that we can’t automate.

So as we continue to explore the ways that automation can aid in our testing efforts, here are a few takeaways we can gain from robotic restaurants:

  • A Good Recipe is Key – Just as we need a chef to create the recipes for the robotic sous chef to follow, there needs to be the first test for automation to follow. Whether this is done through record & replay or by writing out a script, the quality of the first test is crucial to designing a suite that will provide insightful results. Just like a robot can’t follow a recipe it isn’t yet programmed to make and can’t create one on its own, a test automation tool can only test what it is told. Additionally, every bowl is only as good as the recipe, just as every regression test is only as good as the first test or script.
  • Robots Don’t Have an Eye for Detail – Just like humans are the final step in garnishing your meal at a robotic restaurant, human testers have an eye for detail that is critical in testing. There are many soft skills that software testers have — empathy, creativity, curiosity, and knowledge of the software, for example — that are impossible to teach to an automation system. This is why it’s important to have real people that can perform exploratory tests on new features, layouts, or other noticeable application updates, so they can bring unique perception and understanding to provide valuable insight. Not to mention the customer service aspect — communication is important that there’s someone on the team advocating for testing as well as the end users and who’s able to talk to other stakeholders.
  • There is Risk Associated with Automation – Automation usually accounts for “normal” situations but doesn’t consider edge cases or stress. What happens if a customer has an issue with their bowl? What happens if a concert gets out, and the influx of customers is 10 times what it normally is? What happens if you run out of utensils? These are the instances where automation can no longer solve the problem, and a human needs to step in. Automation can’t go outside of what it’s told, which means there will always be risk involved when expected situations arise.
  • Automation is Best for Repeated Actions – A McKinsey report said the food industry is most susceptible to automation because there are so many predictable physical activities involved. Similarly, during code changes and new integrations, it’s important to repeat some of the same tests run previously to make sure they work. This is where automation becomes a good candidate for testing — regressions that have programmable repetition that helps to reduce human error. Rather than have someone perform the same actions over and over, having a machine take over the tester has more time to add new tests is what will result faster productivity in less time.
  • Cut Costs Without Cutting Jobs – Because of reduced labor costs, the robotic kitchen’s bowls start at $7.50 — around $3 less than comparable options. If you’re getting to the point where your solo tester doesn’t have time to run all the regression tests they need across browsers in addition to new exploratory tests, introducing an automation solution can alleviate some of those time-consuming tasks and give you a higher ROI. However, if you implement automation, there needs to be a human behind that as well. It’s also important to have someone who understands automation and the tools in order to properly execute and maintain tests.

The Biggest Takeaway

You can’t automate everything. Though you can add significant automation to a project, there will always be a place for human testers. Additionally, automation should be used as a tool to support manual work rather than replace it.

Interested in hearing more about how test automation is like a robot restaurant. Listen to Episode 6, Manual vs Automated…Restaurants? of SmartBear’s podcast series “The Good, the Bad, and the Buggy” with an interview with Paul Grizzaffi.

 

Exploratory Testing Techniques for Mobile Devices

By Leave a Comment

Exploratory testing mobile devices

For those of us who use our devices for everything, from buying coffee to connecting with friends, the mobile experience holds high priority despite smaller screens and limited capabilities.

Due to the sheer popularity of smartphones and tablets, the importance of mobile device testing doesn’t come as much of a surprise. As mobile testing becomes the industry standard, testers must increasingly incorporate test cases that consider the needs of users that are on the go.

However, while you may be familiar with which tests to perform with a keyboard and mouse, you have to come up with a new strategy for mobile devices, especially during exploratory testing.

Here are a few ways to test mobile devices when manual testing:

  • Touch Screen – Scrolling, clicking, and swiping are different when you’re using your fingers compared to a computer mouse. You want to perform a variety of these actions throughout your application to ensure that you can do everything on mobile that you’re able to do on a desktop and that it functions with the same accuracy. Are buttons big enough to click? Is it easy to scroll from top to bottom and back? How does it look when you zoom in or out? Does swiping work without too much lag? These considerations will be critical in assessing usability.
  • Keypad – Rather than they keyboard we’re all used to, mobile devices have much smaller on-screen versions that also use touchscreen technology, which can pose a difficulty to users. Because they’re generally less user-friendly, you want to look at instances that might use require the keypad such as search boxes. Pay special attention to form fields to make sure they’re not too narrow or hard to use. You don’t want your users becoming frustrated because of a keypad that makes using the app impossible.
  • Navigation – Many times certain navigations will be different for mobile compared to desktop because of the smaller screen space. For example, where a drop-down menu might normally be labeled “Menu” on the web application, it might turn into a hamburger menu on a phone screen. Ensure that these icons and elements are organized in a way that makes their function apparent to the user. Because some of these navigation items may be less obvious, you want to go through and make sure that you’re still able to accomplish the same journey.
  • Portrait and Landscape – When you’re testing on a desktop, the web page will always be horizontal. Mobile phones are most often in portrait mode, but they can also be turned to view in landscape. Visual testing will allow you to evaluate and compare responsive layouts to make sure images, text, media, and other content is digestible in both portrait and landscape modes on mobile.
  • Devices – It’s not enough to test on one mobile device — in fact, you should probably be testing a selection of devices. Because of mobile fragmentation, different models, brands, screen sizes, and resolutions mean different mobile experiences. Additionally, you want to test on both iOS and Android operating systems to get a better understanding of how it differs for each.
  • Performance – Performance may be in good shape on your desktop, but that doesn’t mean it’s consistent for mobile. When 85% of mobile users expect pages to load as fast or faster than on the desktop, testing for performance is not something you want to skip. Test out different pages and record their load time. You also might want to test out different networks, locations, and even battery charges to see how it affects speed.
  • Popups and Alerts – Checking pages that have popups and alerts is important because while it may look fine on a desktop, it could be intrusive on mobile. Alerts that are too large and unresponsive or popups that are layered over each other may even make the app unusable. Additionally, it’s a good idea to see how phone calls, notifications, and other interruptions affect the functionality application.

Once you know which devices you want to test on, you can build out a more comprehensive mobile testing strategy. Keep in mind the ways that devices differ, both from each other (tablet vs Android vs iPhone) and from desktop, to determine which test cases you should focus on during exploratory testing.

Your users aren’t ditching their devices any time soon. Exploratory testing on mobile devices means that you can better understand how users interact with your application.

 

Pair Testing: Working Together to Define Quality

By Leave a Comment

pair testing

When you think of software testing, you probably think of an individual sitting at their computer performing manual tests or maybe writing up a Selenium script. In this scenario, they are also the sole person providing feedback on the test case and recording any notes or documentation.

However, pair testing is a little different, and it’s gaining popularity among the testing community.

So what is pair testing, why would you want to do it, and how do you do it right? A

What is Pair Testing?

It takes two to tango, or pair test. Pair testing is usually done with a tester and a developer or two testers, but the tester can pair with anyone they think would make a good partner such as customer support, product management, or even marketing.

While testers don’t usually want people peeking over their shoulder while they do their job, pair testing encourages two people to work together towards share objectives in testing.

The two will work at the same machine with one person controlling the test through the mouse and keyboard and one person giving feedback, asking questions, taking notes, and making observations. Both parties should be taking part in a collaborative discussion about the test scenario.

Pair testing is similar to the idea of pair programming where two developers work together, and one writes the code while the other reviews, ask questions, or makes suggestions. In pair testing, however, both participants are going through the functionality, usability, or visual representation of that code whether it be a new feature or a completely new UI.

In Lisa Crispin’s pair testing blog, she says her partner has likened pair testing to riding in the passenger seat versus driving the car — “You have so much more chance to look around.”

Basically, pair testing is just two people testing the same scenario together instead of one, using their unique skills to provide additional insight to the rest of their team.

Benefits of Pair Testing

  • Cross-Team Collaboration – By combining the unique skills of different team members, you can gain more insight into what is needed for the specific scenario to be successful. Pairing with a developer, for example, means less back and forth between both roles when new code needs to be tested. Working with another tester or a product manager would give another insight. Each of these roles can then work together to decide what is working well and what may need to be changed or fixed.
  • Two Perspectives – When testing, you’re always limited to the fact that not every user journey will be accounted for. The reality is, it’s impossible to think of every possible combination of actions that could be performed. But by having two participants think about a case, it means there’s another set of eyes and brains to consider new possibilities or strategies.
  • Breaking Down Silos – When things are moving fast in Agile development teams, testers can often end up in siloed roles — especially if they’re working on a small or single team. Pair testing helps break down barriers, work with new people, and get testers’ constructive feedback ideas out in the open so each role can better understand where the other fits in and how it leads to quality.
  • Exploratory Testing – Because of the nature of pair testing, it’s usually an exploratory effort. Where exploratory testing is so important for finding bugs and providing feedback on new features, it still doesn’t always get the dedication it deserves. Pair testing encourages teams to make more time for exploratory testing to collect and deliver that valuable feedback.
  • Having a Witness – Sometimes it can be hard to recreate bugs, and then when that happens, make a case that they need to be fixed. But the best way to solve any mystery case is with a witness. By having someone to see bugs as they come up and help determine whether or not it should be considered a bug in the first place, it is easier to pass to the development team with certainty for fixing.
  • Training Opportunity – Not only is pair testing a great way to help the team deliver with an extra measure quality, it’s also a good way for each participant to learn from each other. For newer testers, matching up with a mentor for pair testing is a fun and productive way to teach them what goes into a successful test session. For other roles that may be interested in testing more, this can teach them what to look at. No matter what level or expertise, there’s always something to be learned from your partner in pair testing.

How to do Pair Testing

Pair testing really isn’t that different from other methods of software testing, but there are a few things you may want to do in preparation.

Following these steps will ensure that it’ll be a productive use of both your time.

  1. Identify the test case you want to work on and why you think pair testing is a solution.
  2. Identify who you want to work with and what each person’s role will be. What do you hope to gain by working with this person specifically? Will one person be operating the machine the entire time or will you switch?
  3. Determine an objective from your pair testing session. Do you want to find bugs or suggest new features? Maybe you want to define tests that should be automated — maybe all of the above. Understand who you will be bringing your results to if it’s different than your normal report.
  4. Create a timeline that gives yourselves enough room to explore and meet these objectives while staying focused on the tasks at hand.
  5. Come up with a few specific cases you want to be included in the pair testing. Katrina Clokie suggests writing down a simple checklist of ideas.
  6. Speak your mind and ask questions. Mike Talks says that “pairing is an interactive partnership,” which means both parties are equally contributing thoughts, ideas, and suggestions regularly throughout the test.
  7. Be open to differences of opinion, styles, and techniques. While this can lead to conflicting points of view, it’s always important to respect the other person’s feedback and try to learn from it and accommodate it.
  8. Keep notes and documentation that you can use to share results with other stakeholders.

This, of course, is a very basic process for pair testing, but in real life, the test case, your partner, and objectives will all affect how you go about pair testing. These should give you a good foundation to get started and ask the right questions.

Two is Company

Pair testing is one kind of what Malcolm Isaacs calls “social software testing”. When following an agile methodology and shifting left, it’s crucial for testers, developers, product owners, and other participants in the SDLC to come together for collaboration. Pair testing is just way one to do this.

While it may not be as deeply technical as, say, test automation might be, there are clearly many advantages to having two team members get together to discuss the quality of the application under test — what’s working, what’s not, and what needs to change?

 

 

How to Apply User Personas to Improve Exploratory Testing

By Leave a Comment

persona based testing exploratory testing user personas

As a software tester, your job is to advocate for the end user. However, when there are thousands or even millions of end users, how can we be sure that we’re keeping in mind their differences?

While advocating for each and every user may be an impossibility, adopting persona-based testing may be the key to deriving more accurate insights for your exploratory tests. By paying attention to the details that give different end users unique experiences, you can curate a more intelligent approach to testing for them.

Sometimes personas are supplemented with pictures of the person and detailed descriptions of everything from hobbies to the number of pets they have. While these can help to paint a complete picture, you may want to start with some of the essentials.

In order to begin building your user personas, consider the following when it comes to determining how visitors are using your web application:

  • Age – Does your product or service target a mix of ages, or is it geared toward users that are older or younger? Younger users are more technically savvy — many have grown up using the internet, so navigating the web comes naturally to them from social media to online shopping. However, while they may be more equipped at handling complex features, often their standards are higher and patience is thinner. They will not stick around for complicated forms or slow load times but will leave your site for a sleeker user experience. While older generations may be less impatient, on the other hand, they also may be less likely to get around more intricate components. They also may be on older browsers and operating systems, which could render your application useless if not accounted for. When it comes to making their experience better, you want to think simple and straightforward. Taking into account the range of user behavior solely based on age, it’s easy to see why it’s an important element to consider when thinking of personas.
  • Job/Role – What does the person do, and what are they using it for that will influence their decision? For example, if you own an office supplies website, you may have two 30-year-old women visiting your website in the span of an hour — does that mean they should be tested for the same way? What if one woman is coming to the site as a freelance writer at a small magazine and one is coming in as an office manager for a major enterprise? They’re going to be looking for two different outcomes and have two different experiences. The freelancer may be looking for a few single items such as a pack of pens, a notebook, and a binder. The manager may be looking to bulk order supplies — what happens when she puts in an order for 300 office chairs and 600 monitors? Can the application handle these requests in the same way as a single item? That may be just one difference the two women encounter going through these process, and you want to assess those possibilities.
  • Accessibility – Accessibility can be an overlooked part of software testing, but considering different challenges people may have when it comes to user personas can be helpful. For example, many men are prone to color blindness, which means something as simple as a color choice could make or break the experience for someone. Users that are hard of hearing may rely on image attributes and labeled form fields. Even just having a slight mobility impairment or vision loss could affect the way an application is experienced, especially when it comes to responsive design and mobile device usage. Once you realize these obstacles are common for a variety of people, it becomes important to make accessibility part of your persona-based testing.
  • Geolocation – Looking at demographics is a great way to understand personas, but it’s not just limited to age and gender; it also includes looking at where people are in the world. In fact, where someone’s looking at your site from has a big influence on how they’ll use it. Keep in mind Hofstede’s cultural dimension theory to determine what different cultures prioritize in order to apply that to your personas and application testing. Look at how McDonald’s changes its web design to accommodate a cross-cultural experience to understand how geolocations can apply to persona-based testing. Depending on where you have customers coming from, these preferences may change how you want to relay your website. Even something like network latency will affect how people from different countries experience your application, so you may want to see what it looks like through the eyes of a different country.
  • Behavior – Some personas can’t be pinned down by hard facts — sometimes you have to get creative imagining the ways different people might use your application. For example, knowing someone’s job title won’t tell you that they have a tendency to open multiple tabs and windows at once, keeping a shopping car that’s ready to check out open on their computer for days at a time. It won’t tell you that someone may enter illegal inputs or never remember their password. As critical as it is to collect data on your customers and create personas that accurately represent your users, part of being a great tester is also thinking outside the box to give those personas depth that includes quirky behavior. Oftentimes, it may seem useless to test an action that you wouldn’t expect anyone to perform, but that sequence could be the one that breaks the application…like the computer glitch that let someone buy nothing from McDonald’s for 99 cents.

Conclusion

Persona-based testing helps you step out of your preconceived notions about how people use an application to think about how they could use it.

Recently, we made an infographic about the different personas you should use in exploratory testing. While specific attributes may differ from team to team in order to paint an accurate representation of users, we believe these personas will give you a better understanding of all the ways that people can push an application to its limits.

Check out Exploratory Testing Playbook: 7 Personas Your Team Should Be Testing For and join us on a visual safari to determine who you should be testing for and how to test for them.

Exploratory Testing Techniques for Your E-Commerce Site

By Leave a Comment

exploratory testing techniques ecommerce website testing

Though test automation has been a saving grace for when checking previously written code coverage, we know that there is no substitute for manual testing. Whether you follow the traditional waterfall method or are practicing agile development with your team, exploratory testing plays a critical role.

Anyone who owns or manages an e-commerce website especially knows this to be true. What’s the use of automating a script you derived out of thin air? Instead, exploratory testing with real, human testers provides insight into functionality, performance, and usability of the application.

In order to better understand some of the most important areas to cover when conducting exploratory tests on an e-commerce site, we put together a few steps to consider when live testing.

  1. What happens when someone tries to log in? Is the user directed to the correct page after they logged in and does it load properly? What happens if they put in the wrong information — do they have to re-enter email and password every time or does it stay in the field? What do any error messages say and is it easy to retrieve a forgotten password or use existing login data? Whether your web application prompts a login as soon as someone lands on the home page or just in order to log into their account at payment, you want this to be a seamless experience because if someone has trouble logging in, there’s a good chance they won’t end up purchasing.
  2. How user-friendly are the form fields? If your website is good enough that a customer can easily make it to check out, you don’t want to trip them up on the last few steps of their purchase as the fill out form fields like name, address, and payment information. Unfortunately, however, this is still something many websites have yet to master. Are you asking too much information or not enough? How long does it take to get one from their shopping cart to a finished checkout? Make sure that promotion boxes and purchase total calculations function correctly. Additionally, you’ll want to make sure that all payment processing works, so that customers who are trying to use PayPal won’t encounter an issue that someone charging to their credit card wouldn’t get.
  3. What’s the experience like on mobile? Speaking of form fields, they can pretty much make or break the mobile user experience. You want to make sure that each field takes up most of the screen when a user goes to fill it in, or someone going through checkout might become frustrated at how hard it is to see and type their information to make the purchase. You’ll generally want to make sure the website design is responsive, which may mean implementing different types of features and designs for users that do not access the site on a desktop in order to make it easier to use and look at. Consider all the ways that the buying process differs with a touch screen and a scaled down keyboard. Of course, it’s best to test for mobile on a few real devices to truly understand how these different functions come into play.
  4. How long do different pages take to load? First of all, you want your homepage to load quickly because this is where new and returning traffic will start, and you want to make a good first impression. However, you also have to consider other areas of your application that don’t get the same level of traffic but are equally as important. Does the last product page load as fast as the first or second one? Do some categories perform better than other? What happens in the shopping cart — is the site laggy when it tries to process a user’s input information? Basically, where does the site trip up that could affect the buyer’s journey and deter them from fulfilling their purchase? Don’t forget about load and performance testing.
  5. What happens when you abandon the site and come back? How much progress does the page save and for how long? What about if the page is reloaded? Will someone be logged out or have their cart emptied if they leave for a few minutes? You have to consider the fact that not every user will have the same consistent experience with your application from beginning to end. They might view a product on multiple devices or keep the tab open while they check your competition, and you would hate to lose them when they come back to buy because you haven’t made it easy enough to pick up where they left off.
  6. What could be better from a usability perspective? You know what they say — if it’s not broken, don’t fix it. Except, this isn’t necessarily true when you’re talking about developing and testing a web application. Just because you don’t find a bug in your exploratory tests, doesn’t mean there’s no feedback to give the developers. Consider not just how to fix the application, but what would make it more intuitive for your users. Could the navigation be in a better place or the contact information easier to find? Are there design components that could be more appealing and is the site accessible to a broad range of users? Is there a to make searching for specific products easier such as different a filter? Usability testing will assure your website remains a popular destination.
  7. How functional are different elements? It takes a testers eye to look beyond just the surface of the website to assess whether it really works the way it’s supposed to. Test search algorithms and results. Experiment with different filter combinations. Check calculations and confirm results. Any time a user has to input information or interact with the website, you should be testing whether or not that the application is responding correctly.
  8. What happens after a customer has completed their purchase? You’ve made it through and impressed (or avoided-annoying) a customer long enough. But you also want to make sure their experience is excellent from beginning to end so that they come back next time. Make sure that the page redirects to something thoughtful so that the customer understands their order is being processed. Give them options to review and cancel their order, contact customer support, and access tracking information. Each of these components needs to be tested to make sure that data is accurate and the functions work.

From 0 to 100: How to Go From Manual to Automated Testing

By Leave a Comment

Manual tester to automated testing speedometer

Test automation is all the rage these days, but breaking into the role of an automation engineer is challenging, and sometimes it might seem like more trouble than it’s worth. When there’s still a clear need for manual testing, what’s the point of you uprooting your current ways just to follow the trend?

Though going from manual automated testing may seem daunting, it’s about much more than adding a skill to your LinkedIn profile. In fact, your future career could depend on your automated testing strategy.

Why Do I Need Automated Testing?

  • Faster testing – Your current process might be working for you now, but as a tester, you should be striving for more ways to get faster and more effective at your job. And besides, manually testing repeat test cases must get pretty boring. Learning automation lets you get regression testing out of the way so you can focus on the fun stuff, while simultaneously allowing you to test more software in each sprint.
  • More and more teams are automating – Automation has been steadily adopted by companies to improve test efficiency, and today 44 percent of tests are automated on average, according to the SmartBear State of Testing survey. Not only that, but testers broadly believe it will become even more popular the coming years as more organizations shift to Agile.
  • High in demand – Because more teams are getting on board with the practice, test automation skills are becoming increasingly valuable. Testers that know how to code and are familiar with tools such as Selenium and Appium are the ones that will stand out when it comes time to make a career move. If you find yourself in the position where you’re looking for a new career, it’d be beneficial to have at least a base knowledge of automation and the corresponding technologies that these companies are embracing.

4 Steps for Moving to Test Automation

  1. Learn a programming language – As mentioned, learning a programming language is becoming a valuable skill for testers to have. It’s never too late to learn, but it’ll also be the biggest adjustment when adopting automation practices since it wasn’t previously needed for manual testing. In fact, learning to code will probably be one of the most difficult parts of learning test automation, as well. However, test automation is impossible without coding knowledge, so it will be something you need to know. We usually suggest starting with Python since it’s often easiest to pick up for beginners and there are countless resources for learning such as Treehouse or Codecademy. Additionally, other skills like XML, HTML, CSS, XPATH, and SOAP are often preferred.
  2. Adopt the right tools and frameworks – You’ll make the process a lot easier on yourself if you use a combination of frameworks, open-source tools, and third-party tools for automation. Sorry to break it to you, but you can no longer solely rely on your mouse and keyboard. Selenium WebDriver will probably be the foundation of your automated testing approach as it’s by far the most popular, which is convenient because you can use any of the new languages you learned. You can learn more about getting started with Selenium here. You’ll also want to familiarize yourself with testing frameworks like Nightwatch.JS, Cucumber, and Protractor, libraries like Capybara and Watir, and Continuous Integration tools like Jenkins CI and TeamCity to get the most out of automated testing. On top of that, paid tools like CrossBrowserTesting will make your life a lot easier when it comes to run test across different browsers. Alternatively, if you haven’t learned to write scripts you can use Record & Replay for lightweight, codeless automated testing.
  3. Start small – The smaller your test cases are, the easier they will be to debug, maintain, and reuse. You don’t want to go crazy with automation and write heavy test cases, or you’ll just make your life harder in the end. Keep them separate and labeled so they’re easy to identify and share in reporting. On that note, you’ll also have to prioritize which tests you want to automate in general. Just because you have this newfound knowledge of automated testing doesn’t mean you should apply it to everything — in fact, it’s virtually impossible to automate everything, and many things are better off being done manually. Polish off those exploratory testing skills when a new feature is released and run risk analysis for which parts of your application should be automated. In addition, you’ll want to create a list of the browsers and devices that are going to be essential to your particular test automation plan.
  4. Back to the basics – Keep practicing, learning, and using your resources. Automated testing might seem intimidating when you’re first getting into it, but all it really takes is time and patience to learn the craft. Also, no matter how good an automation engineer you are, there’s always more to learn, so take comfort in knowing even the experts don’t know it all. Luckily, there’s also a lot of resources at your disposal. Between online guides, forums, conferences, meetups, communities, social media, and even Slack channels, there is no lack of help available at any time when you have a question or are just looking for some guidance. The testing community is robust, active online and offline, with many people who are happy to provide guidance and mentorship.

Additional Resource for Learning Automated Testing

Of course, this is a broad overview of what it takes to go from a manual to an automated tester, and there’s a lot more to actually learning automation than four steps. However, it’s likely that you’ll find the change worth it, both to heighten your career and enhance your abilities.

Discovering Exploratory Testing

By Leave a Comment

exploratory testing

Despite being fairly unstructured, scriptless, and independent of a pass/fail status, exploratory testing is a very important and strategic part of QA. In fact, its focus on discovery and learning in software is proven to be highly valuable when it comes to testing an application for the first time or looking into a new feature from the last integration.

By identifying problems, questions, ideas, and praise through this method of manual testing, exploratory testing helps start a conversation between internal teams and produce improved software.

What is Exploratory Testing?

Exploratory testing is a scriptless testing practice that emphasizes learning and discovery in software and aims to improve quality.

While test cases are not created in advance, exploratory testing still has a very strategic process and requires documentation. Test design and execution are done simultaneously in an ad-hoc manner, although exploratory testing and ad-hoc testing are not the same (we’ll get to that later).

Since exploratory testing requires a human to look at software, evaluate its functionality, and think of new ideas, it’s a manual practice rather than an automated one. It also depends on the skills and knowledge of an independent tester to be able to evaluate where software has issues or could use improvements.

However, just because it’s a manual practice doesn’t mean it can be left behind in the industry move to Agile and CI/CD. Even as teams shift left and transition from manual testing to automation, exploratory testing continues to play a critical role in Agile development.

Exploratory testing is often best for:

  • New websites
  • New feature
  • Unexplored areas
  • Different devices
  • UI testing

Afterwards, these insights are discussed with the rest of the testing and development team members to make changes, fix bugs, or add new features. Additionally, results can be used to raise questions about what’s working well and what’s not to implement in other areas of the software.

Session-Based Testing

Session-based testing is one way to provide more structure in exploratory testing. While you can’t go into it with a script or a definitive test plan, having certain elements in mind like objectives, heuristics and oracles, focus, risks, and follow-up will make sure the process is valuable.

Session-based testing will often include a technique called mind mapping to identify some of these elements in a visual layout. This also helps write better documentation to share with teams after testing is finished.

Establishing a Definition

James Bach and Michael Bolton are attributed as the testers responsible for calling attention to the value of exploratory testing.

However, Cem Kaner coined the term in 1983, defining it as “a style of software testing that emphasizes the personal freedom and responsibility of the individual tester to continually optimize the quality of his/her work by treating test-related learning, test design, test execution, and test result interpretation as mutually supportive activities that run in parallel throughout the project.”

Today, some people suggest that exploratory testing should just be called “testing” since all testing has an element of exploration to it, but we still think that it deserves a place as its own method according to the original definition.

Exploratory Testing vs. Other Testing

The parameters of exploratory testing are often confused or misunderstood. Comparing and contrasting exploratory testing to other types of testing can help clarify its purpose.

Scripted Testing – Scripted testing and exploratory testing are posed as opposites, which helps better understand their definitions. As mentioned, exploratory testing is scriptless and more random, whereas scripted testing follows a set and defined plan written by the tester. This also means that you can’t test something that isn’t planned ahead with scripted testing, whereas exploratory testing encourages testers to discover new regions as they go. This allows testers to act more as the end user and evaluate areas they come across rather than predefined spaces.

Ad-hoc Testing – Exploratory testing and ad-hoc testing are probably compared most often, but they’re two distinct methods of testing. For one, ad-hoc testing doesn’t require documentation, which is an essential part of exploratory testing. Similarly, it’s not well-integrated into team collaboration and has no set goals, objectives, or focus. Though exploratory testing is meant to be done in an ad-hoc style, it also has more set expectations than a being a random test. Additionally, many divert the two because exploratory testing could be said to require more skill, whereas anyone could perform an ad-hoc test.

Can You Automate Exploratory Testing?

Not really. Exploratory testing inherently depends on the individual tester to contribute their skills, knowledge, and insight to new or unevaluated components in an application — something that cannot be done with automation.

Thinking about this in terms of the testing vs checking debate, exploratory testing would be part of “testing” while automated testing is a method of “checking,” meaning that you could not automate it because you can only use automation to check — a tester has to test since a machine cannot provide independent thought.

Tips for Exploratory Testing

  • Have a loose plan – Going into a test with a focus is important in order to come out of the process with valuable feedback on certain objectives. At the same time, the nature of the exploratory test is to address certain issues as they arise and consider those as well. While testers should have a general plan, they should also allow room to improvise and consider unfamiliar elements.
  • Think like an end user – Exploratory testing is an ideal way to advocate for a customer that may be using a website. While developers are familiar with the code they build, it might be more difficult them to think outside the box and use software in a way it’s not optimized for. The job of the tester in this phase is to recognize these areas of weakness and bring them to the attention of the rest of the team. Check out our Exploratory Testing Playbook for ideas on how to test for user personas.
  • Leverage your skills – While exploratory tests are designed and changed as the tester goes along, it doesn’t mean that just anyone can do it correctly. In fact, the results of the test will largely depend on a tester’s skill level. Leveraging a tester’s knowledge of software and previous testing practices will be the most efficient way to find and address problems.

Debunking 6 Automated Testing Myths

By Leave a Comment

Automated testing myths and misconceptions

Automated testing has been the answer for many teams who wanted to speed up and increase their testing efforts, but misconceptions surrounding automation mean that testers don’t always get the most out of their tools.

In an attempt to correct frequent misunderstandings, we’re busting common automated testing myths to give you the truth.

Myth: You should automate 100% of your tests.

Fact: You can’t test everything. Though the ability to perform agile automated tests is a great skill, it’s impossible to automate every test. And even if you could, it wouldn’t be an efficient testing solution. Automated tests are based off source code that’s built based on manual tests. Meanwhile, some tests are better off being executed manually. Anytime a new integration is added, a new test has to be done because automated tests will only check previous scripts. Experienced automation engineers know that prioritizing tests and knowing when to automate is just as important as knowing how to automate. However, though you’re not able to automate every test, automation will greatly increase the number of tests you’re able to run.

Myth: Developers should lead automated testing efforts.

Fact: Automation Engineers should drive automation. While it does take programming and coding knowledge, most of the time it’s the testers that are building automated test scripts. As the software industry begins to depend on new testing methods and development processes, test automation architects and engineers will be in high demand. In fact, most of the time having an automation engineer is preferred to a developer because testers have the mindset, training, and specialized skills for different automation tasks, which include much more than just coding.

Myth: Automation is more expensive than manual testing

Fact: If utilized correctly, automation can save money. While the initial investment in automated testing tools and hiring testers may seem more expensive, the ROI of test automation is often worth it since it saves time that would usually be spent on manual testing and increases productivity. For example, if you’re manually running tests that could be automated, it’s costing you in wasted time because your testers could be doing something more constructive. If you think about this scenario over weeks, months, or years, that’s a huge opportunity cost. Test automation is also a cost effective way to extend testing coverage, scale testing efforts, and mitigate risks as your application grows.

Myth: Automated testing is better than manual testing

Fact: Neither automated or manual testing is better or worse than each other, they just have different uses. The most effective testing strategy will find a happy medium between automation and manual testing. Manual testing is better for one-off tests or tests that require human observation, so exploratory tests, smoke tests, and usability testing should all be manual. Automation is usually best for repeated tests or things that need to be frequently tested, such as regression tests. Familiarizing yourself with the different uses and applying them to the appropriate test cases is the best way to get the most out of your time, money, and testing efforts.

Myth: Automated Testing is an easy fix.

Fact: Automated testing isn’t a one-and-done solution. You can’t set up an automated test and expect that you’ll never have to look at it again. In fact, the source code of your automated tests takes a good amount of maintenance to check that it’s doing its job. Not to mention that changing an organization’s testing and development processes will require new training and onboarding to familiarize stakeholders with automation and writing sophisticated tests. Though the ROI of test automation will likely be apparent over time, the transition to a mostly-automated process from a mostly-manual one will take commitment, patience, and financial investment.

Myth: Automation will solve all of your problems.

Fact: Automation is not a silver bullet. Automation engineers experience false positives and flaky builds that come from a lack of human observation, maintenance, and company-wide collaboration. For example, common challenges using Selenium for automated website testing might include coding for cases with more complicated elements like dynamic content, multiple tabs, and pop-up windows that are easier to test manually. Testing will still require the specialized skills of a talented testing team to troubleshoot problems, write scripts, manage tools, and maintain test cases. For some best practices of Selenium testing, check out this guide.

Why Your Startup Needs Software Testing

By Leave a Comment

Software testing for startups

The need for a software testing in large companies to protect security, functionality, and other components that can make or break their customers’ trust is obvious. Oftentimes, the focus is so large on testing that enterprises have entire teams of testers employed at different skill levels and specializing in different areas.

But for startups that have a minimal budget for development, nonetheless testing, why should budding businessmen invest in software testers? It may seem like a step that could be skipped without causing much impact, however, entrepreneurs that have been in similar situations know the truth isn’t that simple.

The ROI

Why does software testing matter to a startup? With so many other expenses being extended in sales, marketing, and day-to-day operations at a time where you’re probably just trying to break even, spending limited finances on another department may not seem very strategic at first.

However, there’s a big reason you’ll want a tester on your team, and that reason is quality.

While a few bugs early on in your software’s first years in the public eye might not seem like a huge deal, software defects are known to be costly when they make it past product release. In fact, it’s about 30 times more expensive to fix a bug after release than in design.

Debugging after a user has already noticed a problem is not just expensive, but it’ll cost you in productivity, reputation, and consumer trust as well, which is not something you can afford to lose with so few customers who are closely evaluating what your product has to offer them.

Instead, focusing on releasing high-quality software will create brand loyalty and help you stand out against competitors, making earning and retaining new customers much more attainable.

Hiring Software Testers

You may be asking that if you have a developer who clearly knows how to code and debug software, why do you need to hire a dedicated software tester? Why can’t your developer just test the software?

The biggest reason why having a developer test their own software is problematic is that they’re already so familiar with the application that it’s hard for them to act as an end-user and handle it in a way that it’s not made for.

On the other hand, the job of the tester is to test the limits of an application the way a user might. For example, since the developer built the application, they know how the checkout process is supposed to work and might not think to enter an invalid email like a tester would.

Since testers also have the technical knowledge and know the in’s and out’s of software development, their insight provides a highly accurate approach to the quality of software before it’s released. Developers and testers have two different skill sets, and each should be utilized for the job they’re best at in order to create a seamless software development life cycle.

Also, since developers obviously spend a lot of their time actually coding, it doesn’t provide them much time to be testing, too — or at least not as much as they should be. At the very least, there should always be one tester working alongside the developer to make sure appropriate testing can be done on the software at different stages of development and at every integration.

Additionally, if you’ve been hearing a lot about automated testing and think it may be a resource to replace actually hiring a human tester, you’d be sadly mistaken. Automated tests are great for following commands and pre-written tests, but won’t be able to find problems in unexplored regions of the software. This means that in order to do any testing, manual or automated, you’ll have to have a human that knows how to write tests.

The Tests Your Startup Should Be Doing

Below are the few of the different testing methods you’ll want to be doing, even at a startup:

Exploratory Testing

For a new application, exploratory testing is extremely important for laying a groundwork of quality. Exploratory testing is an ad-hoc process where the tester manually goes through the software in an attempt to find bugs.

Again, not just anyone can properly execute effective exploratory testing as there is usually a strict process and specific guidelines to follow. Exploratory testing is largely used to define and evaluate the usability of the app as a customer would experience it, and having an expert manually test the app is the best way to evaluate components like functionality and usability.

Regression Testing

Testing is not a one-off process — you can’t just run one test and be able to release new iterations whenever you want from then on. In fact, every time a change is made to the code, you’re going to want to run a regression test to make sure it hasn’t broken previously functioning features.

This is a great example of where you might have to utilize both manual and automated testing. Doing the same test over and over will take up a big chunk of time, so automation is usually a preferred way to compliment manual efforts. On the other hand, manually creating a test script and running it is necessary before automating regression tests. Either way, regression tests are important to make sure a bug doesn’t pop up after adding a new feature.

Performance Testing

When thinking about quality, fast performance is one of the best ways to establish a loyal user, but bad performance will just as quickly turn potential users away. When a 2-second delay in load time during a transaction results in abandonment rates of up to 87 percent, those are losses that you can’t afford.

You want to make sure your application loads quickly and can handle expected traffic. This is the best way to avoid frustrating your users and prevent them from leaving your website before they make it to check out.

Responsive Design

One of the most crucial details that can easily go un-examined without having a software tester that knows what to look for is the idea of responsive design.

Your website will most likely be accessed from an array of different devices and browsers — desktops, laptops, tablets, and smartphones on Chrome, Safari, or even the looming Internet Explorer. Gone untested, the user experience may be drastically different on each of these configurations, which isn’t a good thing.

One of the worst things you can do is go without optimizing your application for mobile devices — the web page simply won’t survive. Finding a tester who knows the importance of cross-browser testing and knows how to find inconsistencies in responsive design will be one of the most valuable assets your startup could invest in.

Conclusion

It’s pretty simple — when you’re a new player in the game, quality is what’s going to set your apart. Don’t make the mistake of skimping out of testing your product to save money because, in the end, it’ll probably cost you customers.

Hiring testers will ensure that someone is monitoring potential issues and catching bugs before release. While it’s nearly impossible to prevent every bug, having a testing strategy in place creates a safeguard to affect better software and happier customers.

Join Over 600,000 Testers & Developers And Start Testing Today

Learn more Free Trial