CrossBrowserTesting.com

A Design, Development, Testing Blog

Top Automation Testing Frameworks Professionals Love

By 1 Comment

test automation Frameworks

Automation testing frameworks are basically the groundwork for automating test scripts. Different automated test frameworks offer different rules, guidelines, and procedures to help efficiently develop and execute tests.

The best choice for each tester is, of course, dependent on their preferences and needs. In this post, we’ll differentiate the multiple automation testing frameworks and open source automation tools that sit on top of Selenium and help you make the choice for better testing & development.

No Reason to Make Your Own

While it is possible to create your own framework, there are plenty of pre-defined automation testing frameworks to simplify test execution and help you meet your browser automation goals.

“I’ve seen some teams take it to the extreme by building their own, elaborate test framework from scratch,” said TestTalks Founder Joe Colantonio. “This drives me crazy, because they could have easily made use of existing open-source tools and libraries that would have met their needs without writing any code — in most cases with better results.”

Rather than wasting time trying to build an adequate test framework, check out some of developers’ trusted choices. These resources make it easier for your team to collaborate for better quality, security, speed and readability.

Tool Frameworks

The automated test frameworks that we discuss further along in this post require a prior knowledge of foundational browser automation tools. Selenium, Appium, Capybara, and Watir are the original open source tools that provide a basic infrastructure for other frameworks to add to for increased features, capabilities, and efficient test execution.

Selenium WebDriver is a tool and API framework for web browser automation. The big selling point of WebDriver for developers is that it allows them to test in any preferred programming language, which also makes it one of the most popular open-source automation frameworks.

Appium, on the other hand, is an open source automation framework for mobile apps. The framework allows native, hybrid, and mobile testing and drives iOS, Android, and Windows apps using WebDriver. Just like Selenium, it’s a cross-platform tool that can run scripts in any language.

Capybara is a library written in Ruby and is another framework that makes it simple to automate web application testing using an intuitive API that simulates an actual user.  It offers a user-friendly DSL (Domain Specific Language) and locates elements in the DOM (Document Object Model).

Watir (pronounced like water) is another family of Ruby libraries for automating web browsers in an app no matter what technology it’s been developed in. Watir only supports Internet Explorer on Windows, but Watir-WebDriver (which is powered by Selenium) supports Chrome, Firefox, Internet Explorer, Opera and can also run in HTMLUnit headless mode

What all these tool frameworks have in common is that they communicate directly with the browser to perform tasks in the same way that a human would, so executing automated tasks is as accurate as manually testing them.

Automation Testing Frameworks

WebDriver.io – WebdriverIO makes it easier to setup, manage, and write Selenium tests in JavaScript by implementing the WebDriver protocol API. The custom libraries include a test runner for integration testing and provides a plethora of shortcuts to make code cleaner and easier to read. Learn more about testing with WebDriverIO here.

Istanbul – Istanbul is another JavaScript test coverage tool with line counters that make it easy to track how well your unit-tests exercise your codebase and generate coverage reports. Because of this, Istanbul is particularly helpful when it comes to unit testing and code coverage. Not to mention, it works well with most other JavaScript automation testing frameworks like Mocha and Ava.

Karma – Karma is a test runner that is particularly helpful for integration testing and providing an environment with instant feedback. It allows you to test code on real browsers as well as headless browsers like PhantomJS where you can generate, display, and watch test results.

Robot Framework – Robot Framework is implemented using Python as its core but can also run on Java and other languages through a remote interface. It utilizes keyword-driven testing, which makes it simple for testers to perform either normal test automation or ATDD. The modular architecture can be bundled with self-made test libraries in addition to the many included tools and libraries.

Nightwatch.JS – For end-to-end testing, this easy to use Node.js based framework uses Selenium WebDriver’s API to perform commands and assertions on DOM elements. Nightwatch.js also has a built-in command-line test runner that can execute tests sequentially or in parallel. Learn more about testing with Nightwatch here.

Jest – Jest is the test framework Facebook first used to test Javascript and is now used by other tech giants such as Instagram and Twitter. It’s known as a “zero configuration testing platform” that has an easy setup, instant feedback, and snapshot testing. Jest does everything from providing a test environment, structure, and assertion functions to generating code coverage reports, making it increasingly popular among testers outside Zuckerberg’s team especially since it’s so easy to get started with.

Jasmine – As a simple JavaScript testing framework for browser and node.js, Jasmine has an easy setup and broad Angular support with assertations and test double utilities, while it’s BDD style API makes it similar to Mocha

Cucumber – Cucumber is a Behavior Driven Development (BDD) framework that lets you execute plain-text functional descriptions as automated text. It’s written in Ruby but also tests in other languages including Java and Python. The point of Cucumber is that even people without highly technical knowledge are able to read the code, making it more accessible for collaboration across different business departments.

Lettuce – Another BDD test framework but for Python, Lettuce allows you to execute automated tests from plain-text descriptions that almost anyone can understand. It’s actually based off Cucumber, which has a few more features, but Lettuce aims to cover the most common tasks on BDD and is a fairly straightforward option. Learn more about testing with Lettuce here.

Protractor – Protractor is an end-to-end test framework for AngularJS applications that can simulate a user’s interaction with an Angular application running in a browser or mobile device. It has added hooks for running in Selenium that enable you to get more data and benefit from other Angular features. Additionally, a really cool feature is that it lets you separate your tests into various test suites, which allows you to enter a single command and run a group of test cases for different scenarios. Learn more about testing with Protractor here.

Mocha – Mocha is a feature-rich JavaScript test framework running on Node.js and in the browser. It’s great for unit and integration testing, and self-described as making asynchronous testing “simple and fun.” It also includes test coverage reports and use of most JavaScript assertion libraries — most popularly Chai. Learn more about testing with Mocha here.

Tape – Tape relies on a very few number of APIs to provide a framework that’s good for producing test harness for node and browsers. Because it maintains a minimal core and is fairly low-level, additional features are usually added through another module and can be used for both unit and integration testing — some even feel it’s superior to Mocha.

7 Things to Consider Before Hiring a Software Tester

By Leave a Comment

Hiring a test team

How do you hire your testers?

So, you’ve finally decided you need to start the process of hiring a software tester. We’re very excited for you. There are, however, a few steps you should follow before you settle on a final decision and extend an offer, and we have a few tips on how you can make the best hire.

    1. First Thing’s First. The first thing that you should look out for is whether or not the person you’re considering has any errors in their resume. While this should be a given for any job or position, it’s especially crucial in industries where an attention to detail is essential to your job (a.k.a. testing). If the person in question used the wrong “your vs you’re” or spells the company name wrong, they’re probably not going to notice a small, discreet bug.
    2. Know what you want. Are you expecting this person to do a lot of manual testing, or are you looking more for an automation engineer? Are you transitioning to an agile methodology? What’s the company’s preferred coding language? While many qualified testers will come and go, you want someone who is going to be right for the specific jobs and projects you need completed. It’s not just about hiring a software tester, it’s about finding the right one.
    3. Do they know the basics? Can they explain functional testing? What’s the difference between unit and integration testing? Have they ever coded before? While you shouldn’t expect your interviewee to be giving exact dictionary definitions, they should be able to convince you that they know what they’re talking about and are familiar with the software development and testing industry.
    4. Ask about their process. Giving your potential new tester scenarios and asking them how they’d go about finding bugs or how they’d get rid of them is a more in-depth way to assess their testing knowledge. It’ll also give you a more expansive insight into their work style and help you recognize if they’ll be the right fit for the jobs that you need them to do.
    5. Test them. You can challenge someone to define testing terms all day long, but the evidence lies in how well someone can actually execute that test in real life. Whether you have them complete an assignment before you meet, in-person at the interview, or request that they complete one at home afterwards, having them perform a coding or testing job will give a more accurate understanding of their skills, method, and technique. This is also a good way to evaluate how much effort they’re willing to put in. A senior tester that makes sloppy mistakes should be re-considered if you have a junior tester that took the request seriously and clearly put more time and energy into the task.
    6. Have they worked as part of a team? You might find the perfect candidate with years of experience who checks all the boxes. However, being a tester isn’t always a one-person job. Keep in mind that the person you hire will have to regularly collaborate with various people on your team, from designers and programmers to management, engineers, and other testers. Even the best tester will be potentially damaging to your organization’s goals if they are unwilling to take feedback and work with others. You want to make sure they fit the company and not just the job description.
    7. Are they interested in the job? You need to know what you want from your candidates, but they need to be invested in the opportunity, as well. Do they seem interested in the projects they’ll be assigned and eager to bring new testing strategies to your QA game? Your new tester should be willing to learn and be invested in the development process. Make sure that either way the person is in for the long haul and is going to be an asset to your objectives.

Hiring a software tester is a great opportunity to grow your company, learn new testing methods, improve current procedures and advance the efficiency of your software development strategy. In this way, it’s essential to the success of your organization, your product and your team that you put careful time and thought into the job offer.

Image by สุวรรณา วิเศษแก้ว via Wikimedia Commons

The Difference Between Manual vs Automated Testing

By Leave a Comment

manual vs automated testing

When you begin web testing, one of the primary decisions you’ll have to make is when you’re going to test manually or use automated testing.

While neither of these options are technically better or worse than the other, the size, budget and time allowance of a project will certainly be factors that affect which method will work best in your testing procedure.

Below, we’ll outline the best times to use each, and the disadvantages and advantages of automated testing and manual testing.

Manual Testing

Manual testing is essentially self-explanatory — testing of a web application is done manually, by human action. This means that someone actually goes on a device to evaluate numerous components including design, functionality, and performance by clicking through multiple elements or units of a web application without support from a tool or script. Sometimes, people will interchange the terms exploratory testing or live testing with manual testing, as well.

Advantages

The method of manual testing is inherently valuable to consumer-facing websites because the tester is able to have the full user experience, making it easier to assess visual components such as text size and layout as well as UI.

Additionally, most QA processes will begin with some degree of manual testing because it’s an ideal way to perform one-off testing and exploratory test cases as well as usability and ad-hoc testing.

Disadvantages

The limitations of manual testing are that it’s not the best method for test cases that have frequent repetition. For code that’s updated by the day or hour, manual testing would take an exponential amount of time and human resources, and therefore, would not be an efficient way to approach Agile development.

Furthermore, because manual testing is performed by a human, there’s more room for human error as opposed to the accuracy of automated testing. It can also be a boring, time consuming and tedious task for developers to continually perform manual tests every time they update a code, rather than spending their time actually improving the code itself.

Manual Testing v Automated Testing

Manual Testing v Automated Testing for CI

Automated Testing

Automated testing uses the assistance of tools, scripts, and software to perform test cases by repeating pre-defined actions. You can perform test automation by writing scripts with Selenium or through a codeless tool automation tool. All automated testing, however, requires some manual testing at first to create initial testing scripts.

Advantages

The biggest draw of automated testing is that it’s noticeably faster and more accurate than manual testing. This is because automation allows multiple tests to be run in parallel, since the test cases can be distributed across different machines.

In comparison, manual testing has to be done sequentially, and so only one test can be run at a time. In this way, the accuracy of automated testing makes it ideal for finding bugs when running tests repeatedly over a long time period or running multiple tests on different browsers.

Additionally, automated testing is preferred for large projects that require regression, load, and performance testing or any repeated execution. Specifically, developers that use an Agile methodology depend on automated testing to be able to retest scripts as they continually change code because manually testing every time a small fix was made would be exceptionally time-consuming.

Disadvantages

The downfall of automated testing is that, unlike manual testing, there is little human observation involved. While automated testing is more efficient in many ways, developers will sometimes find that they need to execute manual testing in order to add a human element and meet specific consumer needs.

Often people will attribute automated testing as being more expensive because it requires a greater investment in tools. However, if your project calls for automated testing, this investment will most likely pay for itself and be more cost-effective over time.

automated testing vs manual testing

ROI of Test Automation

Complementing Testing with Tools

Whether you’re deciding to pursue a manual or automated testing strategy, there are a number of ways to make the process faster, easier, and more accurate for your QA team.

Tools such as CrossBrowserTesting offer solutions and features that complement manual and automated testing alike. These value-add features including live testing, automated testing benefits for websites Selenium, record & replay, and visual testing all contribute to a well-rounded project execution.

However, the success of your testing process will first depend on your ability to measure which testing method is best for your project. By evaluating the pros and cons of manual testing and automated testing as well as their best uses and what you would like to accomplish through testing, you can launch a solid strategy that will continue to improve your web application and users’ experiences.

To learn more about beginning test automation, check out our eBook.

Read Now

What Is Parallel Testing?

By Leave a Comment

 

Parallel Testing

 

Parallel testing is your ticket to faster testing and a quicker turn around in deployments. When testing websites or applications, it is important to remember that time is a factor- you always have a finite amount of time to test before deployments or to increase your coverage. Testing 100% of an application is a noble cause, but no developer wants to spend more time testing than developing their product. Parallel testing lets you get more testing done in a tighter window.

With Continuous Integration, testers and developers are asked to be constantly writing new test scripts for different features and test cases. These scripts take time to run, and the more test cases running against an increasing number of environments can spell doom for a deployment. Think that testing is only going to take 2 days? Wrong, your tests have been firing off for a week, and they still have a day left. So how do we speed up testing and do more QA’ing with less time between deployments? The answer is Parallel Testing.

Sequential Test Execution:

Sequential Test Execution

 

Parallel Test Execution:

parallel-test-execution

Instead of running tests sequentially, or one after the other, parallel testing allows us to execute multiple tests at the same point in time across different environments or part of the code base. You can do this by setting up multiple VMs and other device infrastructure or by using a cloud test service like CrossBrowserTesting.

Parallel Environments

The growing number of devices and browsers your customers are using can be a challenge when trying to test quickly and efficiently. Let’s imagine a real-world example: in release 3 of your product, you have 8 hours of sequential regression testing to perform before the team feels confident to deploy.

By release 5, this may be twice the amount of hours you’ll need to run your tests and as a bonus, your product is getting popular and is being used by more users on an increasing number of different devices. Before you were testing for only Chrome and FireFox, but now you see that you need Android and iOS devices, Safari and multiple versions of Internet Explorer. So you have 16 hours of tests and 10 different devices or browser to cover. This would take us 160 hours for complete test coverage before our deployment. With parallel testing environments, we can run our 16 hours of tests on 10 different devices at the same time, saving us 146 hours of testing time.

Parallel execution also has the distinct advantage of isolating test cases and runs to one specific OS or browser, allowing for the testers and developers to dedicate their meaningful resources of serious problems with cross-platform compatibility.

Parallel Testing In CBT

We have two methods for parallel testing — Record & Replay and Selenium Testing. Choosing one depends on the amount of programming knowledge you have. If you are familiar with how to write a Selenium script, it’s easy to plug that script in with CrossBrowserTesting. If you are more of a beginner, Record & Replay is a great place to start for codeless automation. Whichever you choose, you have the ability to run suites in parallel to drastically cut down testing time and extend coverage.

The maximum number of automated Selenium and Javascript unit tests (API initiated tests) that can be run concurrently is limited according to your subscription plan.

We understand the importance of running as many tests concurrently as possible to increase the velocity of your continuous integration flow. To support this, we do not place arbitrarily low constraints on the number of parallel tests.

A major advantage of our service is that we provide a combination of real operating systems and an extensive array of physical phones and tablets. While we do maintain a large number of mobile devices, they are not unlimited. Ensuring the load is spread across the configurations is important in providing availability for all our customers.

If you want a toolkit to help you, we like TestNG for Java, there is a GitHub repo below that you can look at to get started

Python Examples
NodeJs Examples
Java Examples
Ruby Examples

Selenium Best Practices for Browser Testing

By Leave a Comment

The Selenium automation framework is a great tool-set for automating interactions with the browser. Because of its powerful open source features and standard adoption across many browsers, Selenium has become the most popular web testing framework in the world. You can run your automated testing scripts in a variety of languages against local browsers, a lab of browsers and devices using Selenium Grid, or a device cloud like CrossBrowserTesting. To get the most out of automation, we’ve put together five Selenium best practices for browser testing.

Use The Right Locators

At the heart of the Selenium framework is interaction with the browser, letting you navigate, click, type and check different objects within the DOM using a few set actions. It does this using several different types of locators:

  • Class
  • ID
  • Link Text
  • PartPartial Link Textial
  • Tag Name
  • X Path

Selecting the right locators can be the difference between a test script that is flexible and successful, and a brittle test that breaks on the slightest UI change. If they are present, use unique Classes or IDs as Selenium locators, as they are less likely to change without someone on your QA or Dev team knowing. But locators like link text can change all the time – think about a dynamic button based on logged in state, or your marketing team running A/B tests. You can also use Xpath to navigate the webdriver, you can read more about Xpath to CSS conversion here.

Communication Between QA, Developers, Marketers, and Designers is Key

There are two aspects of collaborating with different teams that can make your Selenium testing a success: constant communication and instilling the importance of testing to every department. Instilling the importance of the QA process on your development team can save time and efficiency when building a testing framework. Make sure they know how important unique CSS selectors are to your testing efforts, and ask them to write semantic and human-friendly code. Having constant communication across teams can help QA plan for changes to the UI of a website or application mandated by Marketing or the Design team. Having small conversations about the front-end aspect of your app decreases the chance of surprise UI changes that will go untested.

Page Object Model. Use it.

A popular test automation design pattern, the page object model will help you make robust testing frameworks that are resistant to small tweaks in the UI. The page object model has two core advantages:

  1. There is clean separation between test code and page specific code such as locators (or their use if you’re using a UI map) and layout.
  2.  There is single repository for the services or operations offered by the page rather than having these services scattered through out the tests.

In both cases this allows any modifications required due to UI changes to all be made in one place. See SeleniumEasy’s explanation and example:

Make Tests Independent Of Each Other

Testing one single action or process, independent of any other tests is known as atomic testing. This testing strategy will keep you from making chained, brittle tests. Chaining together tests, while saving time upfront, can be a huge blocker for a truly agile or CI workflow. Keep tests as small as possible.

Use Your Resources

Selenium best practices can be tough to learn because it requires a certain amount of tangential knowledge. You have to have some understanding of programming and to truly be a skilled Selenium tester, an understanding of test automation best practices. Luckily, there are some fantastic resources available on the internet to learn from. While we run monthly webinars to get you up and running with Selenium best practices, here are a few of our favorites:

Test Automation and the Benefits of Selenium

By Leave a Comment

It’s Open Source…And Free.

Selenium is an Open Source automation framework built for web-browser testing by some of the brightest developers tired of scripting with expensive legacy automation tools not truly built for the web. Because of this, Selenium’s community is vibrant, growing and is seen as one of the real strength’s of the tool. It has become the most reliable web automation tool because of the ease of generating test scripts to validate functionality. Selenium also works with testing frameworks like WebDriver.IO, Nightwatch.js, and countless others. Oh, and it’s free, with no upfront costs.

One Script for True Cross Browser Testing

With only one script Selenium can test Chrome, Firefox, Safari, Edge, Opera, and Internet Explorer on almost any platform available: Windows, Mac, Linux, Android, and iOS. Customers are using an increasing amount of devices, as they become cheaper and more powerful. Keep up and improve your test coverage with just one script.

Works with The Languages You Use

You can use your favorite language with Selenium, so testing starts right away. With support for Java, JavaScript, C#, PHP, Ruby, Python and Perl you’ll avoid the hassle of learning another programming language just to start browser testing.

Easy to Learn and Scale

You’re not alone! With an incredibly vibrant, growing community, Selenium tutorials, testing, and development help is usually just a short search away. The documentation at SeleniumHQ is helpful, and there countless tutorials and how-to sites for you to quickly learn “Selenese”. You can even use Selenium IDE, a Firefox Record and Replay extension, to generate scripts! Selenium was built to scale with their local testing Grid, so you can be assured that as your application grows, so too will your test coverage. Here are a few of our favorite tutorial sites for Selenium:

Integrations for Every Stage of Testing

Selenium has a solid foundation of standards that make it able to be used easily with popular software development and testing tools. Integrate easily with SC compilers like Maven and ANT, CI systems like Jenkins or TeamCity, cloud-grids like CrossBrowserTesting, and almost any test management tool.

Using Selenium Builder and CrossBrowserTesting

By Leave a Comment

We have released a CrossBrowserTesting plugin for Selenium Builder that allows you to record interactions with your site and play them back against any of the selenium enabled browsers at CrossBrowserTesting. This makes it easy to get started with Selenium scripting – you just point and click to record your script, saving individual scripts or test suites. You can then choose to execute these scripts against our remote browsers. To get started, check out our tutorial.

In addition, you can use our Selenium Builder plugin to save a selenium script to our screenshot system. This allows you to script the screenshot engine to a certain point, then take a windowed, full-page, and full-page chromeless screenshot of this page. You can then use the automated screenshot comparison engine to look for difference between pages.

Using the Selenium Builder extension with Automated Screenshots

Using the Selenium Builder extension with Selenium

Joining The SmartBear Family

By Leave a Comment

We are happy to announce we are now part of the SmartBear Software family.

Who is SmartBear Software?

Glad you asked. SmartBear builds software quality tools used by more than four million software professionals and over 25,000 organizations in 194 countries. They are the folks behind the popular testing tools TestComplete and SoapUI, as well as the killer open source API definition platform Swagger. Impressive, right?

What’s going to happen to CrossBrowserTesting?

The logo is going to change. What? You were expecting everything to change? One of the critical reasons we decided to join the folks at SmartBear is because they are equally pumped about turning CrossBrowserTesting into the #1 cloud testing platform in the world. The resources and investment from SmartBear will enable us to build a bigger and better product much faster than doing it on our own.

To give us full autonomy, the team at SmartBear has decided to maintain CrossBrowserTesting as a standalone service with Memphis continuing to serve as our headquarters. In fact, we just signed a 5-year lease for a larger office right here in Memphis. So don’t worry, we aren’t going anywhere.

SmartBear has an exceptional track record in acquiring companies and allowing them to focus on making their products better. Take the recent acquisition of Swagger. Since acquisition, Swagger has continued to innovate and accelerate its product development process with new offerings such as SwaggerHub.

The SmartBear team brings their deep expertise and understanding of the testing market and we look forward to working with them.

What happens to my customer account?

Nothing. Nada. Zilch. Your account does not change in the slightest. If you have any questions, don’t hesitate to contact us, we will be here to answer just as we always have.

A very genuine thank you to all our customers!

Without the support and love of our customers, we wouldn’t have made it this far. So we want to thank all of you. Your comments, suggestions, and feedback have been critical in developing a tool that people love. We look forward to continue serving you and showing you the new stuff we are working on.

Ken, Mike & Tony

Join Over 600,000 Testers & Developers And Start Testing Today

Learn more Free Trial