regression testing Archives | CrossBrowserTesting.com

Making Your Site Pixel Perfect with Visual Regression Testing

August 2, 2018 By Alex McPeak Leave a Comment

When we think of regression testing, we tend to think about functionality and how new code affects the way previously working elements behave. Will a new integration cause something to break? Will everything still work the same after updating backend code?

But just because you’re looking at functionality doesn’t mean you should let visual design testing fall through the cracks during regressions.

In fact, visual regression testing is necessary to make sure that style issues don’t pop up and your web application continues to look just as good as it works.

What is Visual Regression Testing?

Imagine trying to use a web application on your phone and not being able to find certain elements because of an overlapping image or text. Visual testing isn’t just important for making sure that your application looks good but also ensuring that users are able to navigate it without running into style issues that functional testing might pass as technically correct.

But while we might know the need for visual testing, what’s visual regression testing?

While visual testing compares screenshots of a web page in its current state side-by-side with different browser versions of the same page, visual regression testing is used to look at historical differences.

As we know regression testing is used to check that new code has disrupted the functionality of a previous version, visual regression testing allows to do the same with a visual testing tool to make sure that a web page still renders as expected across different browsers after changes to the code.

Many organizations, such as America’s Test Kitchen, use visual regression testing to archive versions of a certain page to record how it changes from iteration to iteration.

With software teams that are shifting left and moving towards more continuous integration and continuous testing, visual regression testing is important to make sure that new changes don’t cause chaos to your layout. This is important to consider as the application evolves from version to version as well as across browsers.

Especially with mobile testing where smaller screen sizes can throw off responsive design and browser rendering, visual regression testing helps spot major differences before your users do.

How Can You Automate Visual Testing?

Tools like CrossBrowserTesting use a screenshot comparison engine to take automated screenshots of the same page across multiple browsers and devices. From here, you can choose a baseline browser, then compare highlighted layout differences side-by-side.

However, this process is not completely automated. Since pages are not passing or failing like in a Selenium test, it’s up to you to decide what differences are acceptable and which aren’t. For example, an image may be off by a few pixels in one browser but not the other. This is probably fine unless it’s throwing off other elements.

The same goes for visual regression testing. CrossBrowserTesting can automatically spin up the recent and previous versions of a web page, but you have to analyze where there are bugs or inconsistencies.

Tools and Tips for Automated Visual Regression Testing

There are a few ways to do visual regression testing that are worth considering.

PhantomCSS – PhantomCSS is basically a combination of PhantomJS or SlimerJS, CasperJS, and ResembleJS. This tutorial on CSS Tricks gives a good look at how to use PhantomCSS and Grunt for visual regression testing. However, it’s worth noting this requires installation and JavaScript knowledge.
Wraith – You can also use Wraith, which works by crawling two websites, taking screenshots, and comparing them. Like PhantomCSS, it also does take installation and scripting to use, but Wraith is more useful for testing sites with dynamic content in comparison. You can see this tutorial using Wraith for visual regression testing.
Selenium – Selenium isn’t traditionally known as a visual testing tool, but that doesn’t mean you there’s no way to take screenshots. Our Selenium 101 tutorial shows you how to take screenshots during automated tests. This way you can have, automated, functional, visual, and browser testing all-in-one.
CrossBrowserTesting and Applitools – For visual regression testing that requires no installation, setup, or coding knowledge, our integration with Applitools is your best bet. You can run visual comparisons of the same web page on the same browser or mobile device from a previous test session and get alerted of regression errors upon deployment or new releases.

Making Your Website Visually and Functionally Flawless

You don’t have to be shallow to care about how your application looks.

A working web application is important, but to ensure it’s also visually appealing takes its own type of testing.

Visual regression testing allows you to pinpoint every pixel at the UI level so that HTML, CSS, and JavaScript differences don’t affect your user experience.

7 Times You’ll Wish You Had a Visual Testing Tool

October 9, 2017 By Alex McPeak 1 Comment

Recently, visual testing tools have been a popular way for designers and developers to evaluate a website’s responsiveness across browsers. However, to some software teams, a visual testing tool might seem like a luxury instead of a necessity.

While the idea of a good full-page screenshot is satisfying to any UX enthusiast, when does visual testing become really valuable? There are a few key moments where software teams tend to end up wishing they had a visual testing tool.

1. When you change code – In a perfect world, we could simply integrate changes to code every day without worrying about it breaking another part of our application. Sadly, that’s not an accurate reality. When we change code, we have to check to make sure everything still works. Then, we have to do this again on a few different browsers. However, doing this manually every time on your work computer is not only annoying, it’s also inefficient. Instead, running visual tests lets you compare your new changes to historical versions across browsers for easier and more accurate regression testing.

2. When you don’t have the same machine your customers use – Again, life would be a lot easier if you could get away with just testing on the same computer you use for developing every time. Of course, your customers are actually on hundreds of different browser and device combinations. Check out Google Analytics to see which configurations you should be testing on, and make sure they’re being visually verified. You can do this laboriously with a device lab or smoothly with a visual testing tool for screenshot comparisons — the choice is yours.

3. When a new browser, OS, or device is released – Even after looking at Google Analytics, you can’t depend on these few machines to cover your testing needs for very long. New browsers, operating systems, and devices are coming out all the time. While a good amount of users might be on the iPhone 7 now, the iPhone X will surely throw a wrench in your testing. Unless you want to go out and buy all these devices yourself and physically compare them side-by-side, a visual testing tool provides a way to access browsers that are continuously added, updated, and maintained via the cloud.

4. When you want to increase communication – Test reporting and documentation is always a struggle, not to mention getting results to the people that need them. Fortunately, visual testing tools usually have a few ways to make test reporting more accessible for everyone. With integrations like Slack, Hipchat, and Jira, screenshots can be easily shared on the messaging platforms that your entire team uses. Additionally, features like visual reporting make it simple to analyze usage and stay on the same page every sprint.

5. When you need extra help finding layout differences – There’s a lot of reasons we still need dedicated testers. But if you’re working on building your QA team or operating as a one-man-band (a.k.a. freelance developer/designer) then having a tool that literally highlights browser differences can be a lifesaver. This way, you can stop searching each browser for elements that ruin your design and instead, get to fixing them faster.

6. When you want to speed things up – We’re living in the golden age of test automation. It’s not good enough just have to have access to unlimited browsers at the drop of the hat, getting the results should be automated. While live testing is a great tool in itself, it can only take you so far when it comes to gaining fast feedback and meeting deadlines. Automating your visual testing is multi-beneficial — you can more easily pull up screenshots, evaluate them more quickly, and debug them in a more timely manner.

7. When you want to test your design before the public sees it – The whole point of testing is to make sure your application looks great before it gets to the user, so what’s the point of running regression tests after you put a redesign into production? Using a tool that includes local testing allows you to address issues before your website goes live so a bug doesn’t ruin your latest unveil.

Conclusion

Visual testing is more than just a pretty interface, it’s a tool that’s inherently helpful during development. Additionally, as teams continue to shift left and users continue to access the web from increasingly fragmented devices, an efficient website visual testing tool will be an asset for creating your goal app.

5 Steps for Taking Full-Page Screenshots of Your Website

August 28, 2017 By Alex McPeak Leave a Comment

Testers know the value of using browser screenshots for manual debugging and regression testing, but we take it a step further with capabilities for full-page automated screenshots. CrossBrowserTesting allows you to automate screenshots on real browsers and devices and compare them against a baseline to evaluate layout differences and find issues that need debugging more easily.

But what do we mean by a full-page screenshot? A full-page screenshot is more than what you see when you look at a web page on your device (a.k.a. a windowed screenshot) — it’s a capture of the entire web page above and below the fold.

This means full-page screenshots account for any area you have to scroll up, down, or sideways to view. As opposed to manually taking multiple windowed screenshots of one page and trying to stitch them together and compare them to multiple screenshots of another page; our algorithm scrolls the page, captures an image at each viewpoint, and merges them into one view in order to provide a fast and accurate screenshot of the entire page.

By evaluating different browser versions visually, side-by-side with our comparison engine, it’s easier to not only see layout differences but detect smaller problems that might go unnoticed.

Full-page screenshots are a great tool for

Exploratory testing – Exploratory testing is performed the first time a tester looks at new software or new features. Being able to multiply these efforts by visually exploring multiple browsers saves time while increasing coverage.
Regression testing – Any change to working code means the possibility that it’s been compromised. Regression testing is used to ensure that a recent change hasn’t impact any previously functioning features, and comparing them with pass/fail results makes the process easy, effective, and efficient.
Usability testing – Testing the UI/UX of a web page ensures that current and potential customers are happy with your application and will continue to come back. Usability testing across browser and devices configurations further considers diverse user experiences.
Responsive design – Responsive design by definition means that everything looks visually acceptable from screen to screen, device to device, and browser to browser, which means visually validating this on real devices is paramount.

Here are five easy steps to taking full-page screenshots of your website with CrossBrowserTesting:

Log into the CrossBrowserTesting app, click screenshots, and start a screenshot test by entering the URL of choice. Keep in mind, you can also take screenshots locally or behind a firewall so you can test and debug before your app reaches the public eye.
Choose from over 1,500 real browsers, devices, and operating systems to replicate real user scenarios. Hint: use Google Analytics to decide which browsers you should prioritize, or choose the “Popular Browsers” option.
Use the full-page setting and view high-resolution results within minutes. Compare to your baseline, filter by device, and easily examine highlighted layout differents that anyone could understand to find bugs, spot UI issues, and identify elements that are hidden or misplaced.
Download the results, save your reports, and share with your team through Slack, Hipchat, email, or Jira. These communication channel integrations make it easier to provide feedback, start discussions, and affect change by encouraging collaboration with different roles on the development team.
Take to live testing for remote debugging.

Full-page screenshots prove to be a great resource in many stages of testing, and automating them with a tool like CrossBrowserTesting makes the entire process much more streamlined. Or, you can just photoshop your own full-page screenshots together if you’d like, but we prefer our way.

By increasing test coverage and speeding up debugging time, you can more easily identify gaps in quality and quickly deliver better software to your customers. Additionally, the CrossBrowserTesting Comparison Tool provides the best of both worlds, striking a balance between manual and automated testing by a visual representation of your web page while eliminating shortcomings of human error.