CrossBrowserTesting.com

A Design, Development, Testing Blog

Hands-On UI Testing With Python FAQ

By Leave a Comment

On August 14, 2019, I teamed up with SmartBear to deliver a one-hour webinar about Web UI testing with Python! It was an honor to work with Nicholas Brown, Digital Marketing Manager for CrossBrowserTesting at SmartBear Software, to make this webinar happen.

Check out the On-Demand version of the webinar!

In the webinar, I showed how to build a basic Web UI test automation solution using Python, pytest, and Selenium WebDriver. The tutorial covered automating one test, a simple DuckDuckGo search, from inception to automation. It also showed how to use CrossBrowserTesting to scale the solution so that it can run tests on any browser, any platform, and any version in the cloud as a service!

The example test project for the webinar is hosted in Github here: https://github.com/AndyLPK247/smartbear-hands-on-ui-testing-python

I encourage you to clone the Github repository and try to run the example test on your own! Make sure to get a CrossBrowserTesting trial license so you can try different browsers. You can also try to write new tests of your own. All instructions are in the README. Have fun with it!

The Q&A

After the tutorial, we took questions from the audience. Here are answers to the top questions:

How can we automate UI interactions for CAPTCHA?

CAPTCHA is a feature many websites use to determine whether or not a user is human. Most CAPTCHAs require the user to read obscured text from an image, but there are other variations. By their very nature, CAPTCHAs are designed to thwart UI automation.

When someone asked this question during the webinar, I didn’t have an answer, so I did some research afterwards. Unfortunately, it looks like there’s no easy solution. The best workarounds involve driving apps through their APIs to avoid CAPTCHAs. I also saw some services that offer to solve CAPTCHAs.

Are there any standard Page Object Pattern implementations in Python?

Not really. Mozilla maintains the PyPOM project, but I personally haven’t used it. I like to keep my page objects pretty simple, as shown in the tutorial. I also recommend the Screenplay Pattern, which handles concerns better as test automation solutions grow larger. I’m actually working on a Pythonic implementation of the Screenplay Pattern that I hope to release soon!

How can I run Python tests that use Selenium WebDriver and pytest from Jenkins?

Any major Continuous Integration tool like Jenkins can easily run Web UI tests in any major language. First, make sure the nodes are properly configured to run the tests – they’ll need Python with the appropriate packages. If you plan to use local browsers, make sure the nodes have the browsers and WebDriver executables properly installed. If you plan to use remote browsers (like with CrossBrowserTesting), make sure your CI environment can call out to the remote service. Test jobs can simply call pytest from the command line to launch the tests. I also recommend the “JUnit” pytest option to generate a JUnit-style XML test report because most CI tools require that format for displaying and tracking test results.

How can I combine API and database testing with Web UI testing?

One way to handle API and database testing is to write integration tests separate from Web UI tests. You can still use pytest, but you’d use a library like requests for APIs and SQLAlchemy for databases.

Another approach is to write “hybrid” tests that use APIs and database calls to help Web UI testing. Browsers are notoriously slow compared to direct back-end calls. For example, database calls could pre-populate data so that, upon login, the website already displays stuff to test. Hybrid tests can make tests much faster and much safer.

How can we test mobile apps and browsers using Python?

Even though our tutorial covered desktop-based browser UI interactions, the strategy for testing mobile apps and browsers is the same. Mobile tests need Appium, which is like a special version of WebDriver for mobile features. The Page Object Pattern (or Screenplay Pattern) still applies. CrossBrowserTesting provides mobile platforms, too!

Check out my blog for other tips and tricks when working with Python! AutomationPanda.com

Selenium Academy 101 FAQ

By Leave a Comment

With the conclusion of our very first Selenium Academy class we wanted to ensure we armed you with the answers to some of the most common questions that came up during our live QA session!

In our Selenium 101 class we started at the very beginning by explaining what Selenium is, why you will want to use Selenium for your UI testing and especially why you will want to use Selenium with CrossBrowserTesting. But let’s take a look at some of the most common questions that came up during last week’s webinar.

What is the difference between a Selenium grid and CrossBrowserTesting?

While you can make your own Selenium grid with a costly device lab or VMs, many have found moving their Selenium grid to a third-party cloud like CrossBrowserTesting is faster and more productive, allowing more tests to run on more machines in less time. With over 2050 browsers to choose from, the possibilities of parallel test execution with CrossBrowserTesting are almost limitless. Our browser, device, and operating system selection includes all the latest configurations as well as legacy browsers so you can test what your users are on without worrying about any of the associated costs or upkeep. There are certain real costs of building an in-house environment as well as countless other secondary costs of not going with a cloud provider, like CrossBrowserTesting.

Selenium is open source right, why would we need to buy CrossBrowserTesting?

We can give you 8 good reasons!

  1. We make it easier to get started.
  2. Just install Selenium bindings for your programming language of choice and be done!
  3. Eliminate the work of managing and maintaining Selenium Servers, WebDrivers, and Browsers
  4. Screenshots, videos, and even network data can be captured during a Selenium test and shared with anyone
  5. Higher concurrency means faster execution
  6. Run dozens of tests at once without having to run dozens of RAM-hungry browsers
  7. Spend less time waiting and more time doing what matters
  8. Instant access to thousands of browsers and devices, including mobiles

What languages are supported with Selenium?

C#, Java, JavaScript, PHP, Python and Ruby

Which language is the easiest to get started with?

When talking with users about what language is easiest to get started with, we always recommend Python. The reason we recommend it for beginners is because it reads like English and tends to be easier for users to understand when getting started. Take a look at our blog on the Best Programming Language to Learn First.

What’s the best way to set up tests to run within a build pipeline using Jenkins?

There are only two things you need to be able to run your selenium tests from a Jenkins pipeline. You need a shell script or batch file that Jenkins can call to start your test, and Jenkins needs access to any libraries or modules that you use during the test – including the Selenium language bindings. If you are using Python, you can use VirtualEnv to create a virtual environment for Python with Selenium already installed. If you are using Java, make sure to include Selenium in your pom.xml file and run mvn build.

Does Selenium have the ability to open two webpages at the same time?

Yes, Selenium can be used to easily manage multiple pages at the same time. You can use the built-in command “switch_to_window()” documented here: Learn More Here

Is it recommended to use find element by xpath always?

Quite the opposite. Using Xpath to find elements, especially absolute Xpaths, tends to lead to very brittle tests. Any small change in the hierarchy around the element you are locating could cause the wrong element to be selected. Those bugs in your Script can sometimes be hard to debug. You should save yourself the trouble and use element IDs as much as possible

So, find_element_by_id is the best function?

Yes, it’s the most resilient to change and the easiest to use. If you can, add human-understandable IDs to any elements of your page that you want to automate.

For AngularJS based applications, do you have any specific built in functionality?

CrossBrowserTesting does not provide specific functionality to test websites built with AngularJS, but CrossBrowserTesting does work with Protractor – a Selenium-based framework created by Google for testing AngularJS. We have a short guide on getting started here: Help Doc for Protractor

How well does protractor work with just regular Angular?

It works great! That’s what it’s made for. If you run into any problems, feel free to reach out to our support team and they’ll help get you up and running.

Want to register for our 201 class on August 27th? Register now!

Using DevOps Practices to Improve Productivity and Happiness

By Leave a Comment

retain software testers

DevOps has become such a common topic of discussion that it almost sounds like a buzzword at times.  But there are some really great ideas that live under the DevOps umbrella, you just need to peel the onion back a little.  Even if you are tired of hearing people say, “We use JIRA, we’re Agile now!”, I believe it’s worth checking out.

Look, I get it, this is probably the third or fourth silver bullet you’ve been promised. This isn’t a silver bullet, but more of a roadmap to help you make the right choices for yourself and your team.

The core of DevOps can be broken down into three categories:

  • Reducing the risk of a failure – If we can reduce the chance of something failing, that’s less time we spend fixing it, less cost associated with the fix, and less frustration because of the downtime.
  • Reducing the impact of a failure – If a failure can wipe out the entire system, we have to be exceptionally careful to make sure that nothing ever fails. But we all know that no matter how careful we are, we’re going to see a failure at some point. If we can reduce the impact of a given failure, we can see the same benefits as reducing the risk.
  • Reducing toil – Toil is mindless, repetitive work. No one likes doing mindless work. By reducing toil, people are able to spend more time on things that they’re uniquely qualified for, allowing for more impactful improvements for the product and business.

Getting Started

In general, just because you’re following “best practices” doesn’t mean you’re actually gaining much value from them.  When you’re evaluating a process change, consider whether it will provide value to you and your team.  By going through this process, you can evaluate the pros and cons to ensure it’s the best process moving forward.  At the end of the day, you want to make your team better. Wasting time building out a process that doesn’t help is inefficient.

Automated Tests

Automated tests can be a simple place to get started; they can help your developers become more confident in what they’re writing, without having to invest significant time in verifying every change they’re making.  This can increase the efficiency of those developers, allowing them to do fewer repetitive tasks (reducing toil) and reducing the risk of a failure. A common side effect of this is that code will be more thoroughly tested, since it’s low effort now to do the testing, so people will be more willing.

Automated Deployments

Everyone has that one bit of software that is awful to deploy to production – maybe it takes a long time, maybe it’s error prone, maybe it’s just a lot of steps to try to get through. By automating a deployment, rather than requiring it be done by hand every time, you can reduce the risk of a failure (a well-crafted automated process will be consistent and not accidentally fat-finger a key) while reducing the time investment to deploy.

Cattle Over Pets

This is one of the biggest things to work towards, in my opinion: the idea of treating a system as replaceable, rather than as something that we have to take special care of means that we don’t have to worry about what will happen if a specific failure occurs. We have the ability to rebuild on the spot instead, toss it out, and away we go on our merry way again. So if something does fail, it’s much less likely to be a truly catastrophic failure, and the process of fixing it is the same as setting up the box to begin with – reducing impact and reducing toil.

Continuous Integration and Continuous Delivery

Continuous Integration and Continuous Delivery serve slightly different purposes but are very much related to each other at the same time. Both stem from the same principle: by doing work in small, concrete chunks, we have an easier time making sure those changes are valid, and we have an easier time deploying small, concrete changes over monolithic, far-reaching changes. I’ve heard both of these referred to using the idea of a blast radius – even a lot of smaller changes won’t be enough to completely destroy something, but one big change can be enough to bring down pretty much anything. Judicious use of both CI and CD can help reduce risk, reduce the impact of a change, and reduce toil.

 Where to go from here

A common theme for a lot of these changes is that they are designed around making it easier for developers to do the right thing. Build it and they will come. Build the tools that make peoples’ lives easier, and they will use them.

The benefits of testing earlier (“shifting left“) and more often are immense – if a bug caught early in the development cycle costs $50, for example, one that makes it into production can easily cost 10 times as much, and one that lives in production for a while just gets more and more expensive as time goes on.

Take inspiration from what other people are doing but realize that you don’t have to be held to that. Just because something works for someone else doesn’t mean it will work for you, and it probably won’t if you don’t understand the problem they were able to solve with a tool. As long as you’re making things better, that’s the important part.  It doesn’t have to be scary.

Why You Shouldn’t Skimp on Both Functional and Visual UI Testing

By Leave a Comment

visual UI testing functional testing differences importancevisual UI testing functional testing differences importance

There are many components to a successful web testing strategy, but one of the most often overlooked is the importance of visual UI testing in addition to functional testing.

Most teams will focus on one over the other, but to truly catch as many bugs as possible, you’ll need to incorporate both. First, you need to understand what the difference is and why they’re both needed.

Functional Testing

Let’s start by going over what functional testing actually is. Functional testing examines how the software actually works in relation to the given requirements. For example, a homepage on an e-commerce web application might have a menu button; if clicked, it’s expected to drop down with different options.

Functional testing is executed on tests of all sizes from unit tests to end-to-end cases, and it’s critical for making sure that user flows are working as intended and consistent with the product manager’s vision.

So for the same example with the menu button, you may also check to make sure that clicking one of the options on the drop-down will take you the correct page. You can continue to perform a functional test case that includes adding an item to your cart and checking out to ensure each of those actions work one after another and allow the user to successfully make a purchase.

To dive a little deeper, functional tests are basically actions and validations — the tester or tool performs an action with an expected validation, which either works or doesn’t work.

Most of the time, when we talk about functional testing, we’re referring to automated testing where we’re getting a pass or fail test result. When thinking about cross-browser and device testing, functional testing should be included after deciding which browsers and devices to test in order to make sure those tests pass across different configurations.

If you’re in development, design, or QA, this information probably isn’t news to you. However, differentiating between functional and visual testing, and understanding when each is needed, will inform a more intelligent testing strategy.

Visual Testing

While it’s clearly important to test the functional elements of your application, some teams will make the mistake of doing the bulk of their testing to check functional efficiency with little or no regard for visual validation.

Visual testing requires the tester to consider how the application looks in addition to how it works. Often times, the UI of an application can pass a functional test, while design elements that are visually defective can go under the radar.

This becomes extremely important as we look at responsive design and consider the myriad of different devices, browsers, and operating systems. If an application is not built to be responsive, design elements may suffer on different screen sizes, compromising the user experience. This can go unnoticed if just functional testing is performed.

Visual bugs can be annoying or unpleasant to users, but they can go beyond inconvenience to be more severe. In fact, visual inconsistencies can affect the user journey in a way that makes it difficult or even impossible to complete their intended actions even though the test is functionally proficient.

You can see the spectrum of severity when it comes to visual bugs in the following images from the Baymard Institute. In the screenshot of Amazon’s website, the text overlay has not rendered properly and is hard to read. In the example with the form field, visual issues make it difficult to fill out.

Photo via Baymard Institute

 

Photo via Baymard Institute

The issue may be that teams don’t have the time or capacity to do visual testing, or that they don’t prioritize it, but there are many organizations that are simply unaware that it should be an integral part of their strategy.

Automating Functional and Visual UI Testing

As mentioned, functional UI testing is most often done with a test automation tool, such as Selenium or Record & Replay, that will run your test in multiple browsers and give you a pass or fail result to tell you whether the application is working as intended.

However, this doesn’t mean you have to do visual testing all by hand and manually compare your website in different browsers. So how do you speed up visual testing?

A tool like CrossBrowserTesting can be used for visual UI testing to take automated screenshots across configurations. With the screenshot engine, you can compare a page on different browsers and devices side-by-side with your baseline configuration and evaluate highlighted layout differences that let you know where there may be bugs.

Since your team will probably be adding new features and changing the UI from time to time, you can also look at historical versions of your application for regression testing with an integration like Applitools Eyes.

Additionally, to ensure that your web application is put through regular functional and visual testing, you can also schedule tests in a multitude of ways — through Jenkins with Selenium, or with scheduling for Record & Replay or automated screenshots.

By implementing automation practices and incorporating visual testing into your existing strategy, your team can achieve more testing coverage without wasting any time. And your release cycles won’t be a guessing game.

In Summary…

  • Make sure your application works correctly and looks great
  • Visual bugs can be a mild inconvenience, or they could prevent your users from completing a crucial task
  • Ensure a positive overall user experience by testing across different browsers and devices
  • Leverage tools to automate both functional and visual testing for faster feedback

Introducing Scheduling for Record & Replay and Automated Screenshots

By Leave a Comment

automated test scheduling record and replay screenshots

This year, we introduced Record & Replay for teams that want to automate tests without scripting. This method of codeless automation has allowed teams to maximize the value of every manual test by reducing execution time while increasing speed and coverage. And just like with Selenium testing, once your tests are recorded in Record and Replay, you can run them in parallel to get even more out of each suite.

Additionally, when paired with automated screenshots, teams of any skill level have the ability to visually and functionally verify their web application on thousands of browsers at once. By leveraging our Comparison Engine, you can take visual testing to the next level by evaluating differences that are highlighted at the pixel level.

Now with Scheduling for Record & Replay and automated screenshots, you can set tests to run on a daily, weekly, or monthly basis and have the results sent to you via email or Slack. Scheduling is the next step in helping our customers achieve continuous testing without needing to learn highly technical CI tools like Jenkins, TeamCity, and VS Team Services.

If you’re thinking this sounds too good to be true, find out how easy it is to get started with Scheduling in CrossBrowserTesting and see for yourself.

How to Get Started With Automated Test Scheduling

For Record and Replay:

  1. Go to Replay a Suite under Record & Replay in the CrossBrowserTesting app
  2. Select one of your suites. Note: You need a previously saved suite in order to schedule a Record & Replay test. Follow these directions to find out how to create a Record & Replay suite.
  3. Scroll down to “Suite Schedules” and click “Add Schedule”
  4. Decide if you want to run the suite on a daily, weekly, or monthly schedule and set a time for it to run. Choose if you want notifications via email or Slack and select “Add Schedule”

For Automated Screenshots:

  1. Go to View Test Results under Screenshots in the CrossBrowserTesting
  2. Select which screenshot test you’d to schedule and click “Test Results”. Note: You need to have previously run a screenshot test in order to schedule one. Follow these directions for how to take an automated screenshot of your website.
  3. Click “Add Schedule” and set to run daily, weekly, or monthly at a time of your choosing. Select whether you want notifications through email or Slack and click “Add Schedule”

That’s it — just a few steps to functionally and visually testing your UI on a continuous basis with no code.

If you want to learn more about Scheduling for Record & Replay and automated screenshots, join us in our upcoming webinar to learn just how easy it is to start scheduling your tests today. We’ll run through the new feature and give example use cases of Scheduling that you can immediately start implementing in your workflow.

We’ll also go through CI with Jenkins to further our pursuit of continuous testing in CrossBrowserTetsing for teams that may be interested in Selenium automation.

Register for the webinar now.

6 Questions to Ask Yourself Before You Decide What Test Cases to Automate

By 1 Comment

decide what test cases to automate

Even just the idea of agile test automation is enough to get the attention of most software teams. But while the idea of faster testing may seem like the silver bullet, there’s a time and place for automation.

Before you jump in head first, you should evaluate how automation is actually fitting into your organization. By asking these questions, you can get a better idea as to whether automation is the right move, or if you should take another route.

  1. What’s the risk involved? If you’ve performed a risk analysis of your application, you probably have a good idea of which test cases are the most important to check in each code change and which are a lower priority. Consider how human error could affect test data if it’s not automated. Think about how different test cases that may critically impact the customer and/or impact internal team downtime if broken. Additionally, you might want to automate high traffic page, features that depend on security, and problematic areas and other functions that rank high on your risk analysis.
  2. How often am I going to run this test again in the future? Running the same test more than once may seem repetitive and tedious, but if you’re not actually going to be running it frequently, it might not be worth it to automate. For example, if you know you’re going to have to test a feature two or three times but won’t be keeping it as a permanent part of the application or won’t be testing it after that, you might want to rethink automation. The tests you decide to automate should be a foundation of your regression testing strategy, so if you don’t feel like they’re a critical part of the application to be checked in every code change, it may be best to keep them separate.
  3. How much time am I willing to invest? When automation requires so much time and maintenance, you have to determine the ROI of your efforts. Some regression tests may seem like they’re a candidate for automation but upon closer inspection, might not be worth the energy for the value they’re providing. Automation is not a quick, one-and-done solution to your problems. Take a lesson from Elon Musk — you don’t want to drop automation into the mix without a second thought. Spend the time making sure your first test is checking all the boxes and meeting requirements. Then determine how often it will need to be updated or reevaluated in order to establish whether or not automation is worth your time investment.
  4. Are there parts of this test case that still need to be explored? You might think that you’re ready to automate, but keep in mind that automation comes after a thorough exploratory testing session. Just because you can automate doesn’t mean you should, and many tests are better off being performed manually. Are there still instances that could benefit from human observation? While you can’t expect to manually cover every part of the application, getting a foundation during exploratory testing is what leads to more successful, stable, and insightful automated tests.
  5. Does automating support the needs of my team? If you’re automating a test case, you want to make sure the tests you write are going to actually answer the questions your team is asking. First, take a hint from Bas Dijkstra, and discuss the “why” before firing up a test automation tool. While Selenium is a great tool for automation, understanding the nuances of the many commands and abilities of it can take some getting used to, so if you’re not able to write a script that encompasses all of the requirements your team is requesting, another method of testing may be better. If you still feel that automation is the best process for the job but don’t feel confident in your Selenium skills, a tool like Record & Replay may help you achieve your testing goals. There are many choices when it comes to automated testing tools, so you need to make sure the one your organization uses will support the testing you’re trying to accomplish.
  6. Why are you automating? Sometimes the most simple questions can give you the most telling answers. If you’re automating because you think it’s what you should be doing or because your boss is telling you to, you might not be automating your tests for the right reasons. Your objective with test automation should be to provide insight to you your team and give faster feedback. If you have ulterior motives, your efforts won’t take you very far.

Conclusion

Even if you could automate every test, you would quickly find it’s not helping you or the rest of your team very much. Next time you’re deciding on whether or not to automate, ask the right questions first.

7 Reasons Your Testing Team Will Love Record & Replay

By Leave a Comment

Imagine there was a way to have your manual tests recorded so you could easily replay them on hundreds of browsers without needing to perform the same exact actions over and over again. Record & Replay does just that, allowing you to test once and play it back on the largest selection of browsers in the cloud.

This may seem too good to be true, but many testers have already experienced the power of a codeless automated testing tool. If you’re still a little lost, we’ll make it easy for you. Here’s exactly why you should be excited about Record & Replay:

  1. No coding knowledge needed – Have you been wanting to experience the speed of test automation but having trouble getting started with Selenium? If you don’t know how to code, you have a long way to go from learning a programming language to writing your first working script. Luckily, with Record & Replay, no coding knowledge is needed. Rather than writing script, all you have to do is perform them and CrossBrowserTesting will record the actions. If you’re struggling with complicated frameworks and commands to create a successful Selenium test, Record & Replay is a hassle-free way to automate with the skills you already possess.
  2. Ability to edit tests – What happens when you perform the wrong action in Record & Replay or want to change part of a test? No need to stress, all you have to do is go back and edit recorded scripts. Delete mistakes with one click, or go in and edit your script by clicking on the script name and “Edit Steps”. There, you’ll see all the recorded values for each step variable, and once you’ve made your adjustments, all you have to do is save it to put your changes into action. Whether it be a typo or missed step, there’s no reason to spend valuable time redoing an entire test when all you need to do is fix one part of it.
  3. Unbeatable browser selection – There’s no question — CrossBrowserTesting offers Record & Replay on the widest selection of browsers. In fact, most of our competitors aren’t doing it at all. But we aren’t here to brag, we just think that a test is only as valuable as the number of browser configurations it can be run on. That’s why it makes sense that Record & Replay users have access to our numerous configurations, which means whether you want to test IE 8 or the newest Chrome release, you have the resources to check cross-browser functionality where and when you need to.
  4. Parallel testing capabilities – What’s better than automating one test at a time? How about automating ten browsers in parallel? With Record & Replay, it takes seconds to select your test suites and play them back at the same time against different browsers. Parallel testing gives you time back to focus on exploratory test and untouched code in your application. Stop running test after test in different browsers, and start running them all at once to make time for the tests you haven’t run yet.
  5. Access to recording tools – Since tests are recorded as a script, sometimes you may want to insert certain commands that you can’t do manually to get the results you want. For example, if you want to verify element properties to keep visual consistencies across different browsers, you might need extra capabilities to allow you to do that. The tools included in Record & Replay allow you to add quick commands such as expects, sleeps, and hovers to make sure you’re making the most of your tests.
  6. Start shifting left – Software development teams aren’t shifting left because it’s the cool thing to do, it’s actually just a really good way to make sure quality is accounted for earlier. Test automation has been a big part of the shift left movement, allowing teams to test faster and thus earlier and more often. For teams that haven’t yet made the shift or who are just getting starting with automated testing, Record & Replay provides an on-ramp to automation, and sharing capabilities through Slack and Jira means encourages open communication with your entire team so you can get testing involved sooner in the software development lifecycle.
  7. Testing for all – As the roles in the software development lifecycle begin to overlap and blur, testing isn’t just for testers anymore. While we do believe that organizations should depend on a dedicated QA team, there is more room for other software stakeholders to take part in some small testing tasks as well. Previously, the ability to automate was only available to automation engineers or testers with programming knowledge. Now anyone can test from developers and product managers to marketers and designers can take part, so your entire team can run like a well-oiled machine.

Conclusion

If your QA team is still playing a game of catch up to a development team that’s continuously integrating new code, you have a clear need for an automated testing tool for continuous testing.

Trying to understand automation can be daunting, but it doesn’t have to be. Record & Replay can help you take the first steps to faster testing and increased code coverage to support your team from planning to deployment.

How to Fix a Flaky Selenium Suite

By Leave a Comment

flaky selenium test
Selenium has changed the game for many software teams, but it doesn’t come without its many challenges. One of the most common headaches for testers is recurring flaky tests, or tests that fail randomly when there hasn’t been any change to the code.

It’s no surprise that this can get annoying since flaky tests affect progress. After a while, it can significantly slow down teams’ productivity when they are no longer able to differentiate between a test that has a bug and one that they’re wasting time on because of an invalid fail.

Eventually, too flaky tests may mean testers might ignore the results, which of course isn’t a good option either. In order to put an end to the flakiness, we have a few tips to keep your builds stable.

  1. Find the source – When your tests start going flaky, you first have to investigate why they’re going flaky.In order to identify the problem, it’s important to be familiar with common reasons for flaky tests so you can identify them. Oftentimes, this can be something really simple such as poor use of bad locators. Craig Schwarzwald stresses the importance of locators that are unique, descriptive, and unlikely to change. If they don’t meet that criteria, it’s a bad locator, which means your tests are likely to fail.  However, there are many other factors that can lead to flakiness. Richard Bradshaw says that lazy manual testing practices, poor knowledge of tools incomplete data, and bad assertions have all been culprits of flakiness, which goes to show that it simply comes back to how well your tests are written.
  2. Isolate and fix – Once you understand the root problem of the issue, it’s much easier to isolate the test and debug. You want to do this as soon as possible because the longer a flaky test sits in your suite, the more likely you are to attract more flaky tests. As Angie Jones points out, all it takes it one flaky test to ruin the rest of a build. She advises separating tests into different paths — one for stable tests that only fail when something is wrong, and one for unstable tests with flaky fails. This way you can divide attention between the red builds that need to be fixed or rewritten and the green builds that developers should focus on.
  3. Document flakiness – Good testing includes three things — documentation, documentation, and more documentation. Just because you’ve debugged a flaky test doesn’t mean you should be done with it. Documenting will make sure that the team is aware of the problems that caused the flakiness and has the resources to go back and fix it should the problem arise again. Additionally, if you have many flaky tests or a flaky test that is hard to determine the cause of failure, documentation helps recognize patterns that will prove critical for debugging. Nebojša Stričević says that you should document every flaky test in your ticketing system and add information as you discover it so your team can collectively figure out how to fix the suite. It’ll also do you well to keep organized reports before flakiness is found so that you rely on them when you run into an issue rather than trying to gather all your information once a bug is found.
  4. Use page object pattern – We’ve waxed poetic about page objects before, but they are especially useful when it comes to maintaining your Selenium suites. That’s because the page object model will help you make robust testing frameworks that are resistant to small tweaks in the UI. The clean separation between test code, page-specific code, and layout plus the single repository for the services or operations offered by the page allows any modifications required due to UI changes to all be made in one place. Nikolay Advolodkin goes into great detail about how page object patterns can often be the key to stabilizing test automation. If you truly want to eliminate flakiness, page objects will get you on the right track.
  5. Stabilize your environment – One common reason that your tests are unstable is that your environment is unstable. If you have a browser crash, server bug, or bad network latency, it’s going to affect your tests. This may be unsurprising, but it’s a commonly overlooked aspect of keeping your Selenium tests stable. You want to make sure you look at carefully the testing environment and separate automation from the rest of QA for maintainability purposes. Emanuil Slavov advocates that stabilizing his test environment helped go from a pass rate of 50 percent to 100 percent through this method of separation.
  6. Wait for it – Incorrectly using waits and sleeps will be a sure-fire way to have your tests switch from green to red. Not using them will make your life even more difficult. This is because page loads are unpredictable — some aspects will load quicker than others. If you’re trying to automate an action on a feature that hasn’t been loaded yet, you’re going to have inconsistent results. In order to avoid flakiness, differentiate between implicit waits, explicit waits, and sleeps, and add them to your script appropriately. Alan Richardson covers this aspect of synchronization as one of the top causes for failing automation — if your tests pass without waits, it’s probably because you got lucky. He says that state-based synchronization is the answer to making it work.
  7. Frameworks are your friend – Frameworks aren’t just helpful in writing tests, they’re also essential for maintaining a stable suite. Joe Colantonio says that pretty much any time you’re working with automation, you should have a framework in place for reuse. By following the guidelines that frameworks provide, your tests will be more repeatable and maintainable for your entire team.

Don’t Be Flaky

Flaky tests can result from a variety of reasons, but at the end of the day, it’s up to you as the tester to find the root and dissect the issue. Fortunately, everyone deals with flakiness from time to time, so there are many resources dedicated to helping you identify what went wrong.

Thinking about how to prevent flakiness before you start seeing red will be the difference between automation that actually helps you meet your goals and flakiness that just slows you down.

The Best Uses for Live, Visual, and Automated Testing

By Leave a Comment

uses for automated testing

From our humble beginnings as a manual testing tool with just around 50 browsers, CrossBrowserTesting has come a long way to be the reliable end-to-end testing tool you know today. With a range of offerings as old and problematic as IE6 and as new as Chrome 63 amongst our 1,500 configurations, we’re unquestionably the most complete cross-browser testing platform.

Another part of growing up and expanding was adding visual testing and automated testing capabilities in addition to live testing. Today, individuals and teams across the country depend on these features, but we want everyone to get the most bang for their buck. So, in order to truly understand our core functionalities and the best times to implement them, we’re explaining the best uses for live testing, visual testing, and automated testing for websites.

Live Testing

We started with live testing to give people a way to manually test their website in a variety of browsers. Today, manual testing is as important as ever. As teams adopt Agile and Continuous Testing, it’s important that manual testing isn’t overlooked and continues to be a part of the SDLC into the future. Having dedicated testers who understand software and are focused on finding defects and reporting bugs is quintessential to proper live testing.

Best Use Case – Exploratory testing.

Exploratory testing is not a strict set of rules checking whether or not the application works, like automated testing. While there should be a plan going into exploratory testing, it’s also a somewhat ad-hoc process based on testing the limits of a new integration within the software, or even a completely new application, before it’s deployed. It takes a keen sense of observation and an understanding of the application design and purpose to be executed with value.

Using the live testing feature in CrossBrowserTesting, exploratory testing becomes more manageable with access to browsers, operating systems, and devices that teams may not have had previous access to, but their customers are using. By remotely accessing a variety of configurations rather than just the browser your developers are, owning this detailed observation for a broad user base is much more manageable.

Tips for Live Testing – Check out our tips for exploratory testing an eCommerce site and user personas playbook for ideas on what to look for during an exploratory test. Additionally, our native debugging tools allow you or any of your teammates to fix a bug, on the spot, in the very browser it was found on.

Visual Testing

Applitools Eyes Visual testing CrossBrowserTestingVisual testing is used to assess a web application’s responsiveness across browsers. By performing visual testing, you’re looking at the UI/UX components on the front end to decide whether the application under test is acceptable on a variation of browsers, devices, and screen resolutions since they all provide a different experience.

Best Use Case – Screenshot comparisons for visual validation.

Our screenshot comparison engine lets you take screenshots across multiple browsers in a matter of seconds, allowing you to easily compare full-page layouts. By highlighting the differences from your baselines browser, it’s easier than ever before to decide whether your web page is consistent across browsers without having to manually compare them. With the option to test on hundreds of devices, you can be confident that you’re accounting for every customer.

By performing visual regression testing, you can also compare new changes to historical versions to make sure any added integrations or application updates are supporting improvement rather than adding any glitches to the user experience.

Tips for Visual Testing – Run a risk analysis and pick five to ten of your highest priority pages. Find out what the most popular browsers are that your users are on, and run each of those targeted pages on those configurations to give you a basis as to whether or not the application is visually acceptable for the majority of your users. If you happen to find that a page is faulty, take to live testing to debug.

Automated Testing

Automated testing in CrossBrowserTesting relies on Selenium and Appium open source software to allow you to create test scripts in any major language that can be run across multiple browsers. While effective automated testing with these frameworks does require programming knowledge, teams prefer it to be able to run tests in a fraction of the time without needing to perform every step manually. Alternatively, Record & Replay provides a lightweight, codeless test automation solution for teams that have yet to fully learn how to write Selenium scripts.

Best Use Case – Functional and regression testing.

You can take all the screenshots in the world, but just because something looks great, doesn’t mean it also works like it’s supposed to. Automated testing digs deeper into the backend of a web application to analyze unit, integration, and end-to-end test cases and make sure that code renders properly on different browser configurations. By looking beyond the surface of the application, you can better insight on whether or not it will actually work correctly for users at different stages in their journey through your application.

Test automation is also a preferred method for regression testing since repeated actions of long test suites don’t have to be done manually. Instead, running an automated regression test can check that previously stable code still is functioning as it’s supposed to after a new integration, feature, or bug fix is added.

Automated Testing Best Practices – Running an automated test across browsers in parallel can make testing up to twenty times faster, depending on how many configurations you include. While automation is already a great way to speed up repetitive or boring tests, parallel testing allows you to run that same script on other configurations at the same time rather than waiting until each is done to start the next.

Conclusion

While live testing, visual testing, and automated testing each have it’s own purpose, designing a strategy that involves all three is the key to truly comprehensive testing. As your team and product grow, using these capabilities to help your organization balance speed and quality will be what sets you apart from the competition and ensures your brand’s trustworthiness.

6 Ways to Speed Up Your Testing Cycles

By Leave a Comment

faster automated testing

In the battle between speed vs quality, teams that practice test automation know that there’s a practical way to aim for both. However, while testing inherently places value on precise observation and careful consideration over making mistakes for the benefit of speed, how can you make testing rapid enough the meet the needs of Agile development demands without skipping over essential steps?

Faster testing is easily attainable by refining your current automation strategy and adjusting the processes that are already in place. Not only will the following tips make your life easier, they’ll also accelerate your testing to help make the most out of your time.

  1. Get organized – Keeping your tests organized may not seem like a priority day-to-day, but it will make things much more streamlined for you in the future. You should be keeping your test cases lean and reusable, automating features at the unit level. By practicing good test management methods, maintaining accurate data, and keeping clean automation reports, it’ll be easier for you and your team to run tests whenever they’re needed.
  2. Talk to your teammates – Speaking of team, your teammates are some of your most important tools to driving testing that’s effective and efficient. Whether you’re the solo tester on a software team mostly made up of developers or you’re one of fifty other automation engineers, there are always other stakeholders in the software. Communicating with these people will help everyone stay on the same page as far as documentation and reporting, as well of the status of the software included what’s new, what needs to be tested, what’s been tested, and what needs to be fixed.
  3. Know when to automate – Of course, automating your tests will make things exponentially easier and faster, but automating the wrong test cases is just going to slow your progress down. While testing manually might seem to take longer, new test cases require exploratory testing, and taking the time to do it at the beginning will save you from having to go back and do it later after you’ve already tried to automate it. Ultimately, understanding the best cases to automate before you actually start running them will prevent you from wasting time — a software tester’s worst nightmare.
  4. Try parallel testing – While automation will unquestionably help you test more parts of your application faster, it falls short in the way that it is only able to accomplish this in a sequential fashion — no more than one automated test can generally be run at one time. However, with parallel testing, automation gets a much-needed boost by allowing different tests to run simultaneously. If you’re smart enough to understand the importance of the same tests across multiple browsers, operating systems, and devices, you can also see the value in doing this all at once with parallel testing instead of running the script over and over on different configurations.
  5. Use tools to your advantage – Being a great tester will only take you so far if you don’t know how to leverage the right tools to help you. Between adopting open source testing frameworks and investing in automation tools, there’s no shortage of ways to make your life a little easier. With the previously mentioned parallel testing, for example, you need to rely on a tool like CrossBrowserTesting for a cloud-based Selenium grid or Record & Replay.
  6. Cut out flakiness – While tools like Selenium are an asset in automation, it can also lead to flakiness. False negatives will just slow you down — no one likes seeing red all the time and dealing with inconsistent results. In order to avoid this, it’s best to attack the problem head-on by isolating flaky Selenium tests from the build to fix them before rerunning the test again.

Conclusion

Working towards faster test automation doesn’t have to mean starting over. Balancing speed and quality is something you just have to learn along the way, but getting to know test automation hacks will get you there even sooner.

Join Over 600,000 Testers & Developers And Start Testing Today

Learn more Free Trial