CrossBrowserTesting.com

A Design, Development, Testing Blog

The Top 5 Things We Learned in Our Webinar with Bas Dijkstra

By Leave a Comment

Recently, Bas Dijsktra of On Test Automation joined the CrossBrowserTesting team in the webinar “Lighten Up: Methods for Making Your Automation Suite Leaner ”.

Bas’s presentation was jam-packed with important lessons and actionable takeaways. To get the most out of it, you should take the time to watch the full webinar at the bottom of this post.

But we’re also rounding up a few of the most important lessons we learned from Bas. Here are our top five:

  1. There’s such thing as too much automation – With the introduction of tools like Selenium that allow us to automate at the UI level across different browsers, it can be tempting to automate anything we know how to write a test for. However, automation isn’t always the answer, and if you’re automating too much, it may be more of a problem than a solution when your automation suite becomes bloated. This can mean slow tests, shallow feedback, and high maintenance costs that reverse the benefits of automation. Additionally, when your suite is too heavy with automation, it no longer helps to shorten your feedback loop, and will likely add time instead. In order to get the most value out of your automation suite, Bas says that it’s important to understand where you should be testing at the UI level, when you should be testing at a lower level, and when to skip automation altogether.
  2. We have some unrealistic expectations around automation – Automation should never be the end goal of our testing. Bas advocates that we stop trying to automate everything and start focusing on automating the right things. Prioritize the business value rather than automating just to automate. Understand that not everything testers do can be automated, and not all test should be automated. Know the limitations of automation and your testing tool, and familiarize yourself with the tests that could be taken out of your suite to lighten your load to improve your feedback loop.
  3. Automation requires resources – One of the reasons why it may not be worth it to automate is return on investment, or ROI. Automation requires resources to write, maintain, run, and analyze tests. Bas says that this is too often forgotten when teams strive for “complete automation”. You have to decide whether it’s worth it to expend this energy and resources on automation, or if it would be best spent somewhere else. For example, maybe it would be more valuable to do an exploratory test of your application and do a deeper dive into areas you can’t assess with automation. Ask, “Are we comfortable not having the information provided by this test?” If you are, it might be a reason to redirect your resources away from automation.
  4. Not everything should be tested from the end user perspective – UI testing is great to test the end user perspective, but it’s not always the best method. You should automate as close to the implementation as possible and only do user interface-driven end-to-end automation when verifying end-user journeys. Ask, “where is the logic that we’re verifying actually implemented?” Often times you can run a test in the UI, but it won’t give you the feedback you’re looking for. In his example, Bas shows there are instances where testing at a lower level will give you more accurate test results.
  5. You might not need to run your automation suite as often as you think – UI tests can take a long time to run, and with multiple browsers and operating systems, it can take even longer. While tools like CrossBrowserTesting allow you access to a seemingly unlimited amount of browsers, that doesn’t mean you have to test in them all every time. Bas encourages you to decide which browsers you need to test and how often by looking at which your users are on and which are most problematic. If you think there’s value in testing in a browser or operating system but not adding it to your regression suite, CrossBrowserTesting lets you look at those same configurations in on-demand so you can test manually, as well. Bas also suggests moving tests out of the build pipeline and into an overnight job, so you’re not spending as much time running your suite.

Conclusion

As teams become better acquainted with automation, they have certain choices to make about what to automate, how to automate it, and why it should be automated. Unfortunately, sometimes we don’t ask these questions first and end up with a bloated test automation suite. Bas Dijkstra gives valuable advice for considering will actually set you up for success and give you more valuable test results.

To watch “Lighten Up: Methods for Making Your Automation Suite Leaner ” play the video below.

3 Easy Tutorials for Getting Started with Selenium

By Leave a Comment

selenium tutorial easy beginners webdriver with examplesselenium tutorial easy beginners webdriver with examples

You’ve learned your first programming language. You’ve familiarized yourself with Selenium. You’ve got a great Selenium testing tool and a great team to support you. You know what to automate and what not to. But where do you start with your first test?

No matter how prepared you are, getting started with Selenium can be intimidating. But by starting small and getting some of those repetitive tests out of the way, you can get your bearings while building a maintainable and reusable automation suite.

If you work in ecommerce, there are a few test cases you might come across regularly: sign-up, login, and sign out. The good news is that these user stories are fairly easy to automate, and once you understand how to set them up, it’ll be easier to start building your own tests in the future.

SignUp

The signup process will differ from site to site, but there are a few patterns that will tend to repeat themselves. For example, many will have multiple pages with “next” buttons, dropdown menus, and form fields to fill out.

Let’s go through what a test case would look like if you were automating a signup for Gmail.

First, we want to import WebDriver as well as the “Select” command we’ll be using:

from selenium import webdriver
from selenium.webdriver.support.select import Select
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait

Next, we need to instantiate an instance of ChromeDriver, the guy who will be driving our browser:

driver = webdriver.Chrome()

Now we want to define a method, so you can reuse this test, and navigate to Gmail.com where we’ll click “Create an Account” using the element id:

def site_signup():
driver.get(“https://gmail.com”)
driver.find_element_by_id(“Get Gmail”).click()

Clicking should take us to a new tab, then we’ll want wait for the page to load completely so that the form to create your Google Account appears:

WebDriverWait(driver, 10).until(EC.title_contains("Create Your Account"))

Here you find multiple form fields including First name, Last name, Username, Password, and Confirm password. This script will fill all these fields and click next:

driver.find_element_by_id (“firstName”).send_keys(“Mark”)
driver.find_element_by_id (“lastName”).send_keys(“Zuckerberg”)
driver.find_element_by_id (“username”).send_keys(“MZuckerberg”)
driver.find_element_by_id (“password”).send_keys(“Harvard2004”)
driver.find_element_by_id (“confirmpassword”).send_keys(“Harvard2004”)
driver.find_element_by_id (“Next”).click()

Again, this will take to to a new page where you’ll want to wait until it shows all the form fields for “Create Your Account”. It will then request you fill out more form fields. These will have drop down boxes:

def droppable()


select_fr = Select(driver.find_element_by_id("Month")) select_fr.select_by_index(May)
select_fr = Select(driver.find_element_by_id("Day")) select_fr.select_by_index(14)
select_fr = Select(driver.find_element_by_id("Year")) select_fr.select_by_index(1984)
select_fr = Select(driver.find_element_by_id("Gender")) select_fr.select_by_index(Male)

driver.find_element_by_id(“Next”).click()

From here, you agree to terms and conditions, which means you have to scroll down until you see the “I agree” button:

element = driver.find_element_by_id('I agree') element.location_once_scrolled_into_view

driver.find_element_by_id(“I agree”).click()

That’s it! If you follow these steps, it should create a Gmail account.

Log In

Once you get some of those trickier commands out of the way with sign up, these test cases should be easy. As we mentioned, you’ll see how you build on your previous learning to create new tests, such as with this login that uses find element by id, send keys, and click that we just used.

This is what a login with Facebook might look like (we hope that Mark doesn’t actually use the same password for Gmail and Facebook…and we also hope it’s more secure than Harvard2004):

def site_login():
driver.get (“facebook.com”)
driver.find_element_by_id(“ID”).send_keys(“MZuckerberg”)
driver.find_element_by_id (“ID”).send_keys(“Harvard2004”)
driver.find_element_by_id(“submit”).click()

LogOut

Logouts are usually pretty simple as well, but you will probably have to find settings or menu through a drop down. This is what the test would look like if you wanted to log out of Facebook:

def site_logout():
driver.get (“facebook.com”)
driver.find_element_by_id(“userNavigationLabel”).click()
driver.find_element_by_partial_link_text(“Log Out”).click()

You can see here that we switched up the way we located the element (by name instead of id) because of the nature of the dropdown window. Keep in mind there are multiple ways to locate an element if one doesn’t work out, including:

  • ID
  • Name
  • XPath
  • Link Text
  • Partial Link Text
  • Tag Name
  • Class Name
  • CSS Selector

Conclusion

Even if you don’t own a site that includes sign up, login, or log out, practicing tests similar to these will teach you some of the foundations of Selenium, popular commands, and good practices so you can continue to build and improve your Selenium test automation strategy.

To find more common commands that you may run into when building out your Selenium tests, check out our full Selenium toolkit, which includes more Selenium tutorials.

Balancing Your Automation Strategy with Manual Testing

By Leave a Comment

balance between manual and automationbalance between manual and automation

You’ve heard it a million times before — you can’t automate everything. But if your organization is just starting into the swing of automation, they’re probably thinking about all the ways they can automate. So how do you make sure you maintain the proper balance of manual testing and automation?

Whether you have an entire QA team whose roles are split between automated and manual testing, or there is one tester whose jobs is to do both, striking the right balance will be the difference between a testing strategy that provides value to your team and one that gives you more frustration than feedback.

By understanding how to best support your manual strategy with automation, you can ensure you’re getting the best of both world.

Know the Limitations of Automation

No matter how skilled you are in automation, how many Selenium commands you know, or what testing frameworks you use, some things simply cannot be automated.

For example, CAPTCHAs are often very popular with ecommerce sites. However, they’re inherently impossible to automate. This means that any test case that includes a CAPTCHA can’t be automated, and manual testing has to step in.

This, of course, isn’t the only case where you won’t be able to use automation. There are plenty of instances that it will be possible to automate, but it will be so complex to script, that you’re better off doing it manually.

Assess the limitations and challenges of test automation so you know where to step in with manual testing.

Solidify Your Regression Suite

The bottom line for automation is that you want to automate tests that are repetitive and time-consuming, so think about which tests you find yourself going through the most often. Solidifying your regression suite will help you determine which tests you want to cover after every integration or change to code.

Keep in mind that this will change over time as your application changes, but it’s helpful to have a suite that ensures basic functionality.

Additionally, deciding which environments you want to test on will be key to understanding which browsers, devices, and operating systems will be included in your regression suite.

Don’t Sweat the Small Stuff

Small test cases are best for automation and give you tests that are reusable and maintainable. The results of your automation are dependent on the state of your application, which means the smaller and more specific the test is, the more resistant it will be to changes in the UI. This is why unit and integration tests are ideal for automation.

But longer, more complex test cases and entire user journeys are often better left for manual testing. This gives you more room to explore the application and requires less time for scripting. Moreover, it means that if the UI of the application changes, you won’t have to go back into your script and figure out where it needs to be adjusted.

What’s the Risk?

When you’re thinking about whether or not to automate, risk should be a leading factor for consideration. However, while there are many methods for evaluating which tests are the most at risk, identifying these doesn’t mean you want to leave all other test cases that aren’t high risk out completely. In fact, these are perfect for manual testing.

When you considering risk, there are usually three tiers that you sort test cases into — high, medium, and low. As we discussed, most high-risk test cases will probably be included in your automation suite.

But, if you’re finding medium and low-risk cases that you still find valuable to test in order to provide feedback, then they might be something you want to run through manually.

Is It Providing Value?

Automation is a great way to speed up testing, but it also takes time for coding and maintenance, and that may not be worth it if you implement a test that isn’t adding value.

Say you run a test suite every day, and it passes 100 percent of the time. Is it worth it to keep automating if your results never change?

Just because a test was once automated doesn’t mean it should stay that way. Automation requires making changes, updates, and edits as the application code changes.

If a test is no longer providing value through automation, it might be better to take it back to manual testing to see if there are new parts to explore.

Communicate With Developers

Developers can’t tell you what to automate and what not to automate, but they can give you insight into the state of the application.

Keep open lines of communication with developers and other team members so that you are clear on what’s being changed in the application during development cycles. Depending on whether integrations are minor changes to the code, completely new features, or a revamped UI will make a difference in whether the test that follows will be part of your automation suite or should be tested manually.

To get additional insight, it may be helpful to pair with developers when going through the application to get a more in-depth look at what has changed and what areas or functions need to be tested.

Exploratory Testing

Exploratory testing is crucial to visually and functionally verifying new features and changes to the UI. Automation frees you up to test parts that you wouldn’t normally get to, so take it as a challenge to find a new bug or test for a different persona.

Human observation, curiosity, and creativity are paramount to being a tester. Automation can only take you so far when it comes to finding new bugs. That’s why manual testing, and especially exploratory testing, will always have a place in the QA process.

Visual Testing

Functional automated testing is not the only thing you have to worry about. Oftentimes, a functional test will pass because your application is working correctly, but it could still be visually inconsistent across browsers. Not only could this affect your layout and design, it could also affect the way your users are able to interact with your application.

In addition to functional testing with Selenium, you should incorporate automated screenshot testing into your workflow. This will help you account for visual cross-compatibility issues that you may have missed with functional testing by allowing you to compare screenshots of your web application side-by-side in different browsers.

Increasing Test Coverage

Automation will take care of repeated sequences, but in order to increase test coverage, you want to use manual testing to look at areas that have not yet been tested.

In this way, automation can be used as a tool to gradually help you increase test coverage over time. Test high-level functionality with automation, then fill in the gaps with manual testing. Look at edge cases and try negative testing to go beyond pass and fail results to gain deeper insight into your application.

This is also a good time to figure out what new tests you would like to add into your automation suite, so you can continue to increase coverage. Evaluate manual test cases to determine whether they would be valuable in regression, and assess which tests may be missing from the suite already.

Considering ROI

At the end of the day, it’s best to consider return on investment when deciding whether or not to automate. Automating takes time to analyze, code, and maintain.

If the time it takes to automate isn’t worth the results that you’d get from the test, it may be something you want to do manually, or not at all.

 

Webinar | Lighten Up: Methods for Making Your Automation Suite Leaner

By Leave a Comment

Bas Dijkstra webinar leaner test automation

Selenium has been the testers’ tool of choice for automated functional UI testing, but it’s too often misused. This results in inaccurate results, slow tests, and shallow feedback — and no one wants that.

If you’re struggling to write Selenium tests that meet your goals, it might be time to take a step back and get a better understand of the test cases that are best for Selenium as well as those that are better suited for testing at a lower level, or perhaps not automated at all.

Test Consultant Bas Dijkstra of On Test Automation joins CrossBrowserTesting to provide his expertise on how to de-bloat you UI tests and get your automation suite into shape. In this webinar, Bas will arm you with solutions to tighten up your end-to-end automation and streamline your Selenium strategy.

In this webinar you will learn:

  • What makes a good Selenium test, and what doesn’t
  • What is better tested at the API level, or not automated at all
  • How to future-proof your Selenium tests for better end-to-end testing
  • Live Q&A with Bas

Register Now

Bas Dijkstra

More about Bas Dijkstra: Bas Dijkstra a Test Automation and Service Virtualization Consultant at On Test Automation who’s always looking for more intelligent ways to use tools to improve test processes and software quality. He’s been in the field for over 10 years, designing and developing test automation and service virtualization solutions that enhance and improve test teams and test processes. You can read more about him on LinkedIn or Twitter.

Selenium Interview Questions You Should be Prepared to Answer

By Leave a Comment

Selenium interview questions

Software development teams are constantly shifting, which means testers’ roles have to evolve along with them. As we’ve seen more teams go towards Agile, DevOps, and CI/CD, testers are encouraged to concurrently take on skills that better support these workflows and help with faster testing.

One of those skills is Selenium. As more software testers take on test automation and begin to shift left, Selenium testing is one of the most popular strategies, and a comprehensive knowledge of the tool is foundational to creating tests that provide the team with productive feedback and insight. While Selenium is a tool that can benefit from continuous learning, there are certain things that interviewers will want to make sure you know when you’re applying for you’re next role in automation.

To better prepare for that upcoming interview, here are the top Selenium interview questions you can expect to be asked.

Explain the different Selenium components.

It’s important to be able to prove you know the full breadth of Selenium beyond writing a script. Selenium has an entire suite of tools that are used for a different capability. Review the history of Selenium to fully understand the differences between IDE, RC, WebDriver, and Grid.

Do you know the difference between Selenium 1 and 2? How about the programming languages and browsers it supports? Knowing the tool and rounding out your understanding of how it works will be something that employers value.

What’s your experience with Selenium?

Be able to talk about the different test cases that you use Selenium for. What worked well in your last role, and how do you identify a good case for automation? Is there a time that you automated something you shouldn’t have, and what did you learn from it? What automated tests have you cut out of your suite because they weren’t valuable? How many tests did you automate? Do you have experience using a tool to run Selenium in the cloud?

Giving an in-depth look into the breadth of your experience will help your interviewer understand your qualifications.

What are the advantages of Selenium? What are the most common Selenium challenges?

There are plenty of reasons that people gravitate towards Selenium. Have an answer for why people would choose it over other test automation tools and what it can help accomplish.

However, Selenium also has its challenges. Flakiness, dynamic content, reporting, mobile testing, and captcha are a few things that may come to mind when you think of these limitations. Know which ones have solutions and how to implement them, and know which mean testing is better of done manually.

What should Selenium be used for? What shouldn’t it be used for?

Going off of the last question, you should have a comprehensive knowledge of which tests should be automated and which are better off done manually.

This is not necessarily subject to Selenium as you should be able to give an overview of the right times to automate, how to decide by creating a risk analysis, and how to prioritize your automation.

When do you use Waits and Sleeps?

Knowing the proper use of explicit wait, implicit waits, and sleeps is key to reducing flakiness in your tests, which is undoubtedly something your future employer will care about — no one wants false positives and negatives.

Be able to talk about the difference between implicit wait and explicit wait and sleep, when to use each one, and how.

How do you handle common commands?

To verify your basic knowledge of Selenium commands, it’s expected an interviewer might ask you how to:

  • Launch a browser
  • Click
  • Scroll
  • Drag and drop
  • Send keys
  • Find an element
  • Close and quit
  • Handle pop-ups
  • Navigate back and forth
  • Switching tabs
  • Minimize and maximize windows

Check out our Selenium toolkit for tutorials on these commands and more.

What is Page Object Model and Page Factory?

To get more readable maintainable Selenium tests, you need to have a structure that likely involves design patterns including Page Object Model and Page Factory Object.

You should be able to speak to what they are, the differences between them, why they’re important, and how you use them.

What’s your process for reporting?

Because Selenium has limited reporting capabilities, testers’ method of reporting will differ, and your future employer will likely be interested in getting a better understanding of how you report your Selenium tests.

How would you automate this?

In order to prove your technical capabilities, you may be asked to write a test case. This not only helps the interviewer understand whether you’re able to accurately use Selenium, it also helps them understand your process.

Straight From the Manager’s Mouth

These previous questions are common in the test automation space, but to give you a more well-rounded look at what you could be asked, we also asked a few others might ask for Selenium interview questions:

“There are a few questions I ask our candidates. In most cases, they are a bit general, however it will give us an overall picture of what to expect from your future employee. I value the logical and intuitive thinking of a candidate, as I understand that in tandem with their knowledge, this will make them perfect team players and valuable members of our company.

These questions help me assume if a candidate follows the best practices in work with Selenium. It also helps me to understand how much expertise the candidate has and if he or she is ready for further learning and improving as a QA professional.

  1. In your opinion, why we should we choose this tool, and what are its advantages among others similar tools? What is the fundamental benefit of Selenium?
  2. What are the limitations of Selenium? For what purpose it is not the best choice?
  3. What type of tests have you automated?
  4. What are some differences between CSS selector and XPath based locators? Explain how one is better than another and why. What are the testing process particularities while working with CSS selectors and Selenium?
  5. What is a hybrid framework in Selenium?
  6. What is Page Object Model and Page Factory in Selenium? Provide an example in which case we can use them.”

–QArea Software Development Company

 

“1. How many tools are there in the Selenium tool suite? Tell me more about each tool.
2. We use WebDriver’s APIs to write our tests. What do you think about calling WebDriver’s APIs directly inside the test case?

These first two questions are easy questions to start the interview. There is no need to continue if they fail these.

3. What are the solutions to eliminate bad test design problems when using WebDriver? Compare and contrast your solutions.

I expect the interviewee to show deep knowledge on industry best practices on test designs such as Page Object or BDD.

4. What is your preferred control locating technique? And Why? Compare.

Interviewee should have good knowledge about capturing UI controls, and ability to determine what is the best locator for different situation, knowledge about different locator techniques.

5. Do you have any experience integrating your Selenium tests with any CI/CD tools? If Yes, Explain your experience.

This question helps to identify the superstar candidates. One element is that in order to successfully answer this, they will have to have had to have some real experience working with CI/CD tools to explain the answer in detail.”

-Thanh Viet Le, Automation Project Lead & Framework Architect, LogiGear

 

“1. What are some common exceptions you have encountered in Selenium and what are the steps you took to handle those exceptions?

This question is asked to understand the candidate’s knowledge and experience of Selenium exceptions and handling. This enables maintainable automation and we want to ensure these basics are understood by the software engineers.

2. What would the optimal Selenium automation framework look like?

This question is asked to understand the candidate’s in-depth knowledge of automation testing and the automation process, as well as gauge the candidate’s technical knowledge of how Selenium frameworks are architected.

3. Can you provide a few examples of major challenges you have experienced when automating test cases with Selenium and how you went about resolving them?

This question is asked to get an understanding of the types of complex test cases the candidate has automated with Selenium and the strategies the candidate used to resolve the testing challenges.

4. What are some major innovations or enhancements you have created in your Selenium automation framework?

This question is asked to identify the candidates out of the box thinking and assess the candidate’s self-driven qualities. It allows us to gain insight into how a Selenium automation engineer thinks in regards to ensuring the frameworks are maintainable, robust, and portable.

5. Can you describe some changes that have been made in Selenium 3.0 from Selenium 2 and the reason for those changes?

This question is asked to learn about candidate’s complete knowledge of the Selenium testing framework and the candidate’s enthusiasm/curiosity about product updates/new versions. It also gives us a clear idea if they stay up-to-date on enhancements and best practices.”

-Vivek Kumar, Head of Automation, QASource Automation Testing Services

What are we missing? Leave the Selenium interview questions you’ve asked or answered in the comments.

What Robotic Restaurants Can Teach Us About Test Automation

By Leave a Comment

robotic restaurants test automation

To most people, going out to eat doesn’t evoke the image of robots. But to the four founders of this restaurant, robot chefs are the only viable solution.

In this automated kitchen, robots begin cooking your meal in temperature controlled pans as soon as you submit your order. The bots a programmed to spin the ingredients and spices around until everything’s perfectly cooked. Then, it dumps your meal into a compostable bowl.

While the restaurant may seem overrun by restaurants when you look around, there are more humans involved than you think.

Once in the bowl, a person takes it and adds yogurts, sauces, and dry toppings, then hands you your finished product. Behind the scenes, the Culinary Director and Executive Chef are the masterminds behind the delicious recipes, making sure the finished meals have the right balance of flavor. Additionally, there are people prepping the ingredients. Of course, there are also the four engineers behind the robots making sure that everything runs smoothly.

So what does this have to do with software testing?

In the shift from manual to automated testing, we often get ahead of ourselves in trying to automate as much as possible, or even attempting to automate everything.

But just like a fast-casual restaurant cannot operate on robotics alone, neither can we rely on automation to fulfill 100 percent of our testing. There are certain qualities that testers possess that we can’t automate.

So as we continue to explore the ways that automation can aid in our testing efforts, here are a few takeaways we can gain from robotic restaurants:

  • A Good Recipe is Key – Just as we need a chef to create the recipes for the robotic sous chef to follow, there needs to be the first test for automation to follow. Whether this is done through record & replay or by writing out a script, the quality of the first test is crucial to designing a suite that will provide insightful results. Just like a robot can’t follow a recipe it isn’t yet programmed to make and can’t create one on its own, a test automation tool can only test what it is told. Additionally, every bowl is only as good as the recipe, just as every regression test is only as good as the first test or script.
  • Robots Don’t Have an Eye for Detail – Just like humans are the final step in garnishing your meal at a robotic restaurant, human testers have an eye for detail that is critical in testing. There are many soft skills that software testers have — empathy, creativity, curiosity, and knowledge of the software, for example — that are impossible to teach to an automation system. This is why it’s important to have real people that can perform exploratory tests on new features, layouts, or other noticeable application updates, so they can bring unique perception and understanding to provide valuable insight. Not to mention the customer service aspect — communication is important that there’s someone on the team advocating for testing as well as the end users and who’s able to talk to other stakeholders.
  • There is Risk Associated with Automation – Automation usually accounts for “normal” situations but doesn’t consider edge cases or stress. What happens if a customer has an issue with their bowl? What happens if a concert gets out, and the influx of customers is 10 times what it normally is? What happens if you run out of utensils? These are the instances where automation can no longer solve the problem, and a human needs to step in. Automation can’t go outside of what it’s told, which means there will always be risk involved when expected situations arise.
  • Automation is Best for Repeated Actions – A McKinsey report said the food industry is most susceptible to automation because there are so many predictable physical activities involved. Similarly, during code changes and new integrations, it’s important to repeat some of the same tests run previously to make sure they work. This is where automation becomes a good candidate for testing — regressions that have programmable repetition that helps to reduce human error. Rather than have someone perform the same actions over and over, having a machine take over the tester has more time to add new tests is what will result faster productivity in less time.
  • Cut Costs Without Cutting Jobs – Because of reduced labor costs, the robotic kitchen’s bowls start at $7.50 — around $3 less than comparable options. If you’re getting to the point where your solo tester doesn’t have time to run all the regression tests they need across browsers in addition to new exploratory tests, introducing an automation solution can alleviate some of those time-consuming tasks and give you a higher ROI. However, if you implement automation, there needs to be a human behind that as well. It’s also important to have someone who understands automation and the tools in order to properly execute and maintain tests.

The Biggest Takeaway

You can’t automate everything. Though you can add significant automation to a project, there will always be a place for human testers. Additionally, automation should be used as a tool to support manual work rather than replace it.

Interested in hearing more about how test automation is like a robot restaurant. Listen to Episode 6, Manual vs Automated…Restaurants? of SmartBear’s podcast series “The Good, the Bad, and the Buggy” with an interview with Paul Grizzaffi.

 

How to Choose a Test Automation Tool

By Leave a Comment

how to choose a test automation tool
Once you or your team decide you’re ready to get started with automated testing, then comes time to choose your tool.

You’ll quickly find that there are a lot of different options to make your decision difficult, but there are going to be many differences between them as well. Looking a little bit closer will help you find the right fit, but first, you have to ask a few questions during the evaluation.

How long will it take to get started? – Depending on the tool, you may have to do some downloading, installation, and setup. You may have the time and patience for this, but if you want a tool that’ll take minimal effort to set up, then it’s something to take into consideration when looking. You also want to think about what the onboarding process is like. Some platforms will be fairly straightforward, while others take a while to get the hang of. This is something you can usually evaluate during the free trial. Determine how quickly you can get into the tool and start building out tests and if the interface is simple and clear, or if it will take some time to get familiar with the tool. Many tools will be intuitive and easy to dive into, but not all are. If it’s something that will take weeks of setup and training during onboarding, it may be worthwhile to consider another option.

How is customer support? – The trial is also a good time to evaluate some of the intangibles, such as customer support. Try contacting the support team to see how responsive they are if you run into issues during trial, whether that be setting up and getting started or running an automated test. Consider how helpful they when getting to know the tool, since you will likely rely on them as a regular resource. Check out the documentation and resources they provide in addition — tutorials, webinars, and other learning materials may be your best friend as you get started with automation. Open source options will be lacking in providing actual human support engineers, but it also helps to have a robust user community, or rely on the support team from a third-party integration tool.

Who will be using it? – How many people will be using the tool? Will it mainly be used by automation engineers and developers who are familiar with programming, or will you need something a bit more flexible for different technical levels? Some automation tools will require programming knowledge and other technical expertise, while others may be better suited for beginners or those with little automation experience. Has your team done automation before? If you’re moving from manual to automated, it might be worth it to look at which tools will help with the on-ramp. Look at your options with a Record & Replay tool versus a scripting tool, and decide which option makes more sense. Keep in mind that some tools offer both, which may be helpful if you have a wide range of technical skill sets on your team.

What systems does your organization already have in place? – Your choice may be confined to a few pre-determined details in your organization and the way it already runs, so it’s important to look at the integrations. If your team works in specific programming languages, you want something that can support those and has documentation for them. You also want to think about unit and end to end frameworks, CI/CD platforms, open source tools, and even your methods of communication like Slack and Jira. If your new tool doesn’t work with your former tools, you’re going to have to find different workarounds. Instead, make life easier on yourself by scoping it out first.

What will you be testing? – Are you testing desktop, web, mobile applications, or all of the above? Some tools are better suited for testing different environments or for certain kinds of applications, but others will provide an all-in-one solution. How many browsers and devices will you want to test on, and which ones do you need? It may be a good idea to look at the most popular browsers and devices your customers using, so you can evaluate whether the tool you’re looking at that offers them for automation. Additionally, consider how many test cases you have that you want to automate and how expansive your application and test coverage is. Thinking about the need for the tool and knowing what you’re specifically looking to test is paramount to finding a solution that will help you achieve your business goals.

How is it to maintain tests? Like your application, your tests will change over time. In addition to building tests, automation also demands that you spend time on maintenance for reusability and editing. Does the tool keep a comprehensive history or archive of your test cases and suites? Is it simple reuse or rerun previous test cases for regression testing? How easy is it to go in and edit a test that may need to be changed? With automation, you have to think beyond the first execution of the test to determine longevity, which means you want the tool to support this as well.

What does reporting look like? – Perhaps the only thing as important as the execution of your tests is the reporting of your tests, which means that paying special attention to reporting features should be one of your considerations when choosing a test automation tool. How is reporting handled and displayed in the tool you’re looking at? Is it easy to share results with different team members, both within and outside the tool? Screenshots, notes, and tracking features can make all the difference when it comes to making results are properly communicated.

What are the requirements and nice-to-haves? – You should be going into your tool research with a checklist of requirements. Look at some of the previous touchpoints to decide what you need in a tool, and make sure that your choice checks all the boxes. However, some tools might overlap and you might find you have more than one option. In that case, it’s also a good idea to consider what value-add features would be nice to have. Keep in mind the complexity of the application, though — just because it’s large and popular doesn’t mean it’s the right choice. Make sure that features that are included are ones you use and that you’re not spending extra money or time on things your team doesn’t need.

Finding the Right Test Automation Tool

There is no one answer when it comes to finding the best test automation tool, and everyone will have a different opinion. At the end of the day, your choice should be what’s right for your team.  Spending the time coming up with the most important criteria will ensure the tool you choose now and into the future as you continue to expand automation across your organization.

How to Prove the ROI of Your Test Automation Efforts

By Leave a Comment

test automation roi

The reason for test automation usually boils down to the fact we all want to get more testing done, faster. But when it comes to proving the return on investment, or the ROI, of your automation efforts, things can become a bit blurry.

Whether you’re trying to convince management, your teammates, or whoever’s in charge of your budget, prescribing an exact value can be difficult. But this doesn’t mean you should give up on the dream of implementing automation across your organization.

The Trifecta

When we talk about software testing and development, we want to address three major elements: speed, quality, and cost. As we evolve our processes to encompass Agile, CI/CD, and DevOps in our workflows, this becomes especially true.

We want to create and test more features in less time, rather than releasing on a monthly, quarterly, or yearly basis. We want those integrations to be innovative and functional at the same time, rather than releasing updates that cause the software to break. We also want to do all of this in a way that’s cost-efficient.

Often we get stuck and are only able to focus on one or two out of these three priorities. But the true return on investment of adopting automation and doing it successfully is addressing all at once.

So how does automation help answer speed, cost, and quality all at the same time?

Speed

We know that automation is supposed to save us time, but how does it do that? Especially when it takes additional time to write the code or create the test, how can we be sure the investment of time is worth the time we save?

Where automation really come into play is with repetitive tasks and repeatable scripts. Say you have a test case that takes about an hour total. Realistically, this isn’t that much time — if you’re only doing it once.

But if this test is at the top of your risk analysis, it means you should be performing it during regressions. In addition, it means that you want to perform it in multiple browsers and devices.

If your team is integrating new code twice a week and you identify 5 configurations that are necessary to run as part of your regression suite, this means this test is taking 10 hours of your work week.

And that’s only one regression test. If you have 5 of these, that’s 50 hours just to check that everything is still working as intentioned; it doesn’t include testing any new features or areas that should be explored.

Of course, you could skip regression testing completely. But there’s always the possibility that when the new integration is pushed live, it will break something that will cost the entire team even more time to fix.

Automated testing takes care of time-consuming tasks so you can focus on testing new areas and building new tests, rather than manually executing the same tests over and over.

In this example, if you were able to automate those 5 tests, you would be saving the time it takes to repeat them in every browser and during every regression. While the initial time investment is there, the return on that investment is greater because you can reuse the scripts instead of manually repeating them.

Not only does this save time, gives you more time to perform new tests and create more automated tests. It also decreases the time it takes to get the test results back to the development team and thus helps to speed up the entire SDLC.

The best way to prove the ROI of automated testing for speed if it’s part of your current strategy is to add up the time it takes to do manual regression testing, then compare it to the time it takes to automate those regression suites.

Cost

It can be hard to determine the true cost-benefit of automation, but there are a few ways you can understand ROI when it comes to budget.

A good place to start by thinking about cost in relation to time — as they say, time is money. Quite simply, if you’re paying one or more software testers, you’re getting more for your money with an automation strategy in place to alleviate time-consuming work by replacing it with new tests of value, as we previously discussed.

However, another way to consider cost is to look at how much money you’re saving every time you find a bug before it reaches your customers.

The earlier on you find each bug in the software development life cycle (SDLC), the better according to IBM research, which found that cost to fix a bug increases as it moves through the SDLC.

In fact, the cost to fix an error found after release was about 15 times higher than if it’s uncovered during the design phase. It’s up to 100 times more expensive if it’s identified in the maintenance phase.

For example, if a bug costs $100 to fix in Gathering Requirements phase, it would be $1,500 in the testing phase and $10,000 once in Production.

The idea here is that it’s harder to fix an issue and more expensive as it approaches the end of the development cycle. Once it reaches the customer, it’s the most costly — not to mention intangible costs to your reputation and trust as a company when customers find bugs that you missed.

By getting more testing done sooner and verifying that old and new code is functioning, you avoid the cost that comes with finding a bug as you move down the SDLC.

Quality

Improved application quality comes with finding more bugs before new integrations reach customers as we just discussed, but it also accounts for the fact that time allows for more exploratory testing ideally being conducted during each development cycle.

The job of the tester isn’t simply to find bugs. An often overlooked element of the tester’s role is to provide feedback and insight when going through the application to make suggestions and affect changes for future changes and additions to the application. It’s impossible to do this if the tester is spending all their time checking whether the code is stable and functional.

By making more time exploratory testing and leaving automated scripts to check regressions, testers can focus on suggesting improvements, collaborating with programmers, and helping the software evolve, rather than spending the bulk of their time manually going through the same actions on different browsers every time they need to check core functionalities.

Additionally, automation takes out the factor of human error. While manual testing is essential to make observations about the application, tests that are merely for conducting the same steps over and over is left up to risk when done by a person. Automation is the answer to ensuring that your results are consistent and will provide a “Pass” or “Fail” when run.

Automation should never replace manual testing but support it in regression checking to allow for more time for manual tests that require human observation. Not only will this go towards confirming the quality of the current application, it will also aid in making changes to support improved quality in the future.

Additionally, by allowing more time for these manual tests and regression checks, automation helps address broader code coverage, meaning that the application that reaches the customer is more likely to work the way it should.

How to Prove ROI

You can see where speed, cost, and quality come together to help create a more productive development workflow and a better application.

But, if you’re still confused on how to prove ROI, here’s a few actionable items you can take:

  • Relate it Back to Your Business – It’s impossible to give one blanket statement for ROI that works for every company in terms of effectiveness. If you’re finding a challenge in convincing members of your team that automation is the right option, do your best to add up the manual hours that could be solved with automation. Identify which tests could make a good case for automation and where automation could help you could save time in your day-to-day.
  • Offer a Success Story – There’s a reason why everyone’s talking about test automation. Plenty of individuals, teams, and organizations have found success in adopting automation practices. Take America’s Test Kitchen. Their team found that they weren’t able to cover all the tests they wanted on all the configurations they needed to and keep up with the continuous iterations of the development team. By adopting a Selenium testing strategy and bringing on an automation engineer, they were able to find a solution to some of these pain points.
  • Leverage Parallel Testing – Parallel testing allows you to run the same test in multiple browsers at once, which means it’s extremely easy to prove the ROI. With every test you run in parallel, you’re increasing your test coverage that many times. For example, if you run two tests in parallel, you’re testing twice as much in the same amount of time. If you up it to 10 tests in parallel, you’re testing 10 times faster than running them on after another. Those are numbers that anyone can agree to.
  • Do Your Homework – Take the time to learn tools and automation practices on your own time. Understand what will make automation successful and what skills you can learn to make the strategy effective, so once you’re given the green light you can start to see the benefits. Learning any test automation tool does take onboarding and time to learn, but staying ahead of the curve means you’ll be more ready to jump in and start making tests. If you’re already automating to an extent, share the results and insights with your team. Record the amount of time it takes to create tests and how much time it saves, so you can have tangible proof of the ROI.
  • Look at the Browser Gap – A major benefit of automation is that it takes out the manual work of repeating a test in different browsers. If you’re not currently testing more than a few browsers, it’s worth it to investigate and see where you could be compromising quality. Check out which are the most popular browsers and devices your customers are using — you may have a gap that automation could help fill.
  • Do the Math – If you’re looking for hard numbers, this guide on measuring the ROI of automated testing may be a good place to start. By calculating costs that relate to automation of new tests, automation of prior tests, coverage across environments, reduction of defects, test redundancy and reusability, you can back up your argument for test automation with real data.

The ROI of Test Automation

Yes, automation does take some investments to get started. It’s easy to see why teams can be hesitant to hop on board with automated testing when looking at these factors.

The value comes with the ROI. Why it may not be immediate, you will see the ways that automation aids with time, costs, and quality over time.

As the team becomes more well-acquainted with the ways automation can help move the business forward, it will continue to be more of an asset. In fact, you might begin to question how you ever managed without the help of test automation.

Top Takeaways from Agile Testing Days USA

By Leave a Comment

Agile testing days usa

Agile Testing Days came to the US for the first time this year, and if we do say so ourselves, they chose one of the best city for the venue — Boston!

SmartBear was one of the event sponsors, so luckily we were able to sit in on some of the amazing speakers, and we learned a few things along the way about Agile testing. Here are just a few of the top takeaways from Agile Testing Days USA.

How Agile Are You?

If you’re at Agile TD, your team is probably working in an Agile fashion or looking to become more Agile. But Selena Delesie noticed something in her time working with different companies — many were saying they were Agile without actually working in an Agile manner.

Many times she found that people wanted to control everything and strictly plan out their sprints, which didn’t lead to the flexibility that Agile demands. Additionally, she found that many people were no longer engaged in their work and were not present with themselves and their coworkers.

“Stop doing Agile; stop treating it as a process,” said Selena. “Start being Agile. Doing and being are very different things”

Selena challenged the audience to reconsider what Agile means to them and think about whether their work processes were actually Agile.

Going back to the Agile Manifesto — specifically thinking about individuals and interactions, customer collaboration, working software, and responding to change — and even the definition of “agile” are good places to start. Agile is supposed to mean that you’re able to think on your feet, be active, and lively but many companies have lost that, and agile has become a heavier process than Waterfall.

Selena says that having more process and control does not help them do their best work, it makes them mediocre. She especially emphasized connecting with people and building relationships. Making the effort to have conversations with people, listen to them, and show compassion can make all the difference.

It’s especially important for management to instill these values and be transparent in order for an Agile organization to work effectively. Selena suggested to stop having meetings behind closed doors and instead, show trust in your team. Let go of feeling the need to control things and make more of an effort to discover and learn.

 

Look Beyond the Tool

Test automation is has proven essential for Agile teams, which means you need certain test automation tools. But when implementing automation into your workflow, it’s important to consider a few factors besides just which tools you’re using. In “Automation for the People”, Christin Wiedemann suggested that you additionally think about the “what, why, and who” of automation.

Before even starting with automation, you should first determine why you’re automating and how it’s benefitting the rest of your team.

Christin notes that automation is much more than running script using a tool — you should expand the scope of what you define as automation to understand what you want to automate. This is the best way to avoid waste, whether it’s waste of time, waste of effort, waste of money, or waste of trust in automation.

Oftentimes, automation is used to take over repetitive tasks to eliminate human error that comes with doing things over and over, but a lot of people start automating without specific intention.

Saving time isn’t always the answer because you might end up replacing the time it takes to execute a test with coding and maintenance. The most important thing is to have a reason for automation and set goals.

You also want to think about who will be automating. This isn’t limited to thinking about who will be writing the scripts and running them. Testing is a team effort, and everyone has different skills and ways to add value, which means those people should be contributing to automation efforts. “At the end of the day, we all want to contribute to the same objective,” says Christin.

Additionally, Christin emphasizes the importance of seeing automated and manual testing as supporting practices rather than contrasting them against each other. Meeting your objectives will require a mix of both manual and automated testing, and she says that it’s important to remember that the results are more telling than the method of achieving them.

By broadening our definition of automation to think beyond tools, we can get a better understand of test automation to further the success of testing in our teams.

Musicians and Testers Aren’t so Different

Have you ever thought about how much test automation has in common with music history? If not, Angie Jones’ talk “Owning Our Narrative” gives a lot to think about.

Angie took us through the evolution of music, starting with Thomas Edison’s phonograph in 1877 and explaining how that idea developed to lead way to new recording instruments like the player piano and jukebox, then more recently to the cassettes, CDs, and MP3s that we know today.

As these changes took place, people began to develop new preferences when it came to music. Rather than being the social aspect it once was, music was now being sorted in genres. Record labels were recording electronically to get better sound, and automated machines were able to play one song instead of making people buy a whole album.

With all this innovation, musicians were forced to make adjustments and change the way they made music to meet the requirements of the different ways it was recorded and played. However, while the role of the musician changed in many ways, the skills remained the same.

If we think about the timeline of test automation, it’s very similar. We’ve found new ways to write and record tests, which has required testers to learn new skills as we embrace agility and shift left, but at the core, the role of the tester also remains the same.

Angie advocates that testers need to find a way to stay true to this role no matter the changes that are going on around us. Musicians were upset about automated machines taking jobs while everyone else was excited, and it was only by adapting to the new environment of music they were able to be successful.

She encourages us not to make the same mistakes as musicians. Embracing industry changes and making it work to your advantage will be far more productive than resisting it.

Angie notes that sometimes it can be difficult to convince others the value of our role in the changing landscape, but the more conscious testers are to those changes, the more prepared they are to own their narrative.

BONUS: Strategize Test Automation

We love Angie Jones so much, we wanted to make sure we invited her to SmartBear while she was in Boston. Luckily Angie accepted our invitation and taught our attendees about “Which Tests Should I Automate” at our Ministry of Testing meetup.

In this interactive session, Angie gave us a look at how she would perform a risk analysis of different test cases at Twitter to decide which tests should be automated, which tests you might want to automate, and which you would skip.

Many times we make the decision of automation off a gut reaction without considering the different aspects of each test — which are the most user critical? Which break the most often? Which would take the longest to fix?

By evaluating these factors against each other, we can make better decisions when it comes to test automation.

See the video below to watch the lesson for yourself.

The Top 8 Things We Learned From the QA Team at America’s Test Kitchen

By Leave a Comment

Americas test kitchen quality assurance

With three core websites, America’s Test Kitchen has a lot of content to cover. Yet when Megan Collins came on as QA manager, she was the only tester. Since then, America’s Test Kitchen has exponentially scaled their testing, adopted an automation strategy, and expand the QA team to three members, including Automation Engineer Michael Ng.

We recently had Megan and Michael join us for a webinar to talk a little bit about the web testing strategy behind America’s Test Kitchen and how they handled their shift to automation with the help of CrossBrowserTesting.

Here are the top 8 things we learned in the webinar:

  1. Fast deployment schedule demand testing — Most organizations dream of achieving the agility that America’s Test Kitchen has reached, deploying code to production every two days. But when you’re integrating as often as they are, you can’t just rely on developers to test each others code or you’ll never achieve consistent test coverage. By bringing in Megan Collins as QA Manager, she was able to spend her time on something that had been severely lacking — testing. This way, proper exploratory tests could be performed when there are any changes to code, and the development team can continue working doing their jobs instead of worrying about the intricacies of whether a new feature works or not.
  2. Real devices matter – When discussing browser simulators, mobile emulators and real devices, there’s a time and place for each one. In fact, Megan mentions that the developers on her team use emulators. However, the only way to see exactly what your users are seeing is to use real devices, which is why they are so important to the QA team. In fact, they even ran into a bug where membership details weren’t displaying for a user that wasn’t found on the emulator they were using — they noticed it once they performed the test on a real tablet device. While emulators are highly accurate, they do not replace real devices, and this is a large reason why they need a tool like CrossBrowserTesting that includes all three in order to prioritize both speed and accuracy during mobile testing.
  3. It’s important to understand different user journeys – The success of the America’s Test Kitchen QA team has a lot to do with their ability to identify the different users they have and how it affects their testing. Because a lot of the web content they have is for members only, there are paywalls that occur when someone accesses from a non-paid account, for example. By separating user states into two focused areas for their automated tests, static and dynamic, Michael is able to create more precise tests for different parts of the website. Understanding that they need to test different based on whether users are new, registered, on unsubscribed helps them build out different test cases and scenarios.
  4. Capture tools can help prove a point – Sometimes seeing is believing, and it can be hard to convince other team members of a bug they haven’t seen for themselves. Tools like CrossBrowerTesting’s screenshots and video capture help the America’s Test Kitchen QA team build out their test reports to include visual evidence of different occurrences during tests. This way, Megan doesn’t have to repeat the test case or find it on the right browser when she shows it to development, she can just show the screenshot or video instead so bugs get fixed faster. Additionally, by using the screenshot comparison engine, Michael has been able to create an archive of each homepage over time so it’s easy to show what changes in each regression.
  5. Use automation to make your life easier – America’s Test Kitchen isn’t automating every single test on their website. In fact, manual testing is still a large part of their strategy. Automation comes in mostly with repetitive tasks that take up a lot of time. For example, they use automation to sign in as a member in order to see whether content can be viewed instead of going through the actions of logging in in every time. This saves Michael valuable time in performing different test cases under different user scenarios — instead of replicating the same actions every time, he can focus on how each test differs.
  6. Collaboration is key – We say it over and over again, but communication with your team is important to productive testing. By leveraging tools for organizing your tests, taking notes, and tagging, it’s easier to remember which bugs you’ve encountered before and which are new. Additionally, keeping open communication with the development and design teams has been essential to America’s Test Kitchen for understanding priorities, risks, and hot spots for more insightful testing. Megan and Michael have found collaboration with the development team to be especially crucial in the shift to automation — learning new skills, choosing what to automate, and being notified when new code is added.
  7. Test while you sleep – Many of the automated tests that the America’s Test Kitchen QA crew create run at night, meaning that you literally can test in your sleep. Overnight testing is at the crux of their Selenium strategy, meaning they can check tests in the morning to see if they pass or fail. Because they have so many tests that run for an extensive amount of time, they rotate their browsers each night to get coverage of all the different configuration their users are on to see which tests are failing on which browsers. By running these regression tests, they’re able to test a subset of test cases across different pieces of functionality that they need to make sure are working day to day.
  8. Tests are only as good as how much time you put into them – Michael runs a lot of automated tests, but he also spends a lot of time maintaining them. The America’s Test Kitchen QA team emphasizes that maintenance is crucial to the success of the project. This is one of the main reasons they have a dedicated QA automation engineer. In order to ensure that Selenium tests aren’t just been created left and right, Michael spends a lot of time on each one to make sure it is accomplishing his goals (and may even be a few steps ahead of Elon Musk in this way).

Megan and Michael gave us a great look at how they use automated and manual testing to further the goals of America’s Test Kitchen. By taking the time to learn from other successful QA teams such as this one, you can apply some of their practices to your own testing strategy.

If you want to view the full webinar, you can watch below.

Join Over 200,000 Testers & Developers And Start Testing Today

Learn more Free Trial