Case Studies
Products
Resources

  • Support

  • Company

Sign Up

Case Study:
How CrossBrowserTesting Helped America’s Test Kitchen Move From Manual to Automated Testing

America’s Test Kitchen is the home of two magazines, two renowned cooking shows on public television, dozens of best-selling cookbooks, and a cooking school with hundreds of courses available. Much of this content can be found on one of their three core websites.

America’s Test Kitchen has a single mission: to help you succeed every time you cook. What some of their audience doesn’t realize is how much web testing goes into that mission.

However, as the demand for culinary content grows, so does the QA team’s need to scale testing efforts. In their transition for manual to automated testing, CrossBrowserTesting played a critical role in supporting those efforts in order to find the right balance of manual and automated testing and create a long-term strategy.

 

Starting From Scratch

The America’s Test Kitchen web experience includes multiple subscription-based websites that each have separate focuses and require different testing.

The QA team’s current methodology considers different user states and leverages automation, but testing hasn’t always been as seamless it is today.

When QA Manager Megan Collins joined as the first QA team member, there were twelve developers testing the code with deployments almost every day. The lack of dedicated testers meant that measures of quality would slip through the cracks, and it was difficult for her to keep up with such frequent iterations.

While they did initially introduce a solution that utilized remote manual testers, test results were subjective and time consuming, and assessing them led to too much overhead. This did not scale well as they continued to add coverage across the different sites, and QA continued to be a bottleneck for releases.

“We needed a better solution. We needed to spend less time analyzing results and more time building testing,” said Megan.

Megan Collins
Megan Collins
Quality Assurance Manager

America's Test Kitchen is the home of Cook's Illustrated and Cook's Country magazines, America's Test Kitchen and Cook's Country from America's Test Kitchen public TV shows, dozens of best-selling cookbooks, member websites, and the America's Test Kitchen Cooking School. We have a single mission: to help you succeed every time you cook.

Location:
Boston

Sector:
Publishing

Products Used:

Why Us?

  • - Wide Selection of Desktop & Mobile Browsers
  • - Real Device Lab
  • - Scale Selenium Usage
  • - Ability for Entire Team to Live Test

Cooking Up a QA Strategy

Rather than outsourcing to remote manual testers, Megan began focusing her efforts on growing the QA team internally and quickly realized automation would have to be part of their evolving testing strategy.

Another goal for America’s Test Kitchen included the ability to run automated nightly regression tests to ensure stability. This way, automation would give objective results that wouldn’t require the lengthy analysis they were currently facing.

And of course, browser coverage would be important since their audience is spread across a range of desktop and mobile devices. Real devices were also a must to mimic real user experiences to best reflect their audience.

“We ran into a bug at one point where we were using an emulator for Chrome and missed it on a real tablet device,” said Megan. “Membership details were not displaying for the user. We caught it, but it reinforced the fact that using emulators is fine for development to check the work but we needed real devices in QA.”

Once they had a checklist of basic requirements, they decided Selenium would help get there -- it was scalable, objective, and had a fairly quick ramp up with the help of the development team.

Finding a tool that integrated with Selenium and helped achieve these goals would provide the extra support they needed in the transition and be integral to their growth.

Moving to Automation

Using CrossBrowserTesting and Selenium was a recipe for testing success, hitting all the major needs America’s Test Kitchen identified as important.

With over 1500 real desktop and mobile browsers, there was more than enough support for automation efforts across the same browsers and devices that reflected America’s Test Kitchen users.

America’s Test Kitchen was immediately impressed by CrossBrowserTesting’s integration with Selenium. Automated Testing allowed them to create new tests easily, integrate them quickly, interactively view automated test runs.

They were particularly excited about using Selenium and CrossBrowserTesting for handling different user states. With different account access based on membership, it was important to the QA team that they were able to automate actions that could be reused in different scenarios.

Additionally, tests could now be run overnight, saving valuable time during regressions and allowing the team to come back to straightforward pass or fail results in the morning.

Not to mention, the ability to prioritize these results was an important value-add feature. CrossBrowserTesting helps to organize and process them easily and effectively with tags, descriptions, and labels that let others know why tests might be failing.

When a test fails, screenshots, video capture, and links make it easy for different team members to evaluate scenarios and pinpoint what went wrong.

In this way, CrossBrowserTesting has proven to be more than just a testing tool. It’s also a communication tool that helps QA share what they find with the entire team including development, design, and UX.

“CrossBrowserTesting with Selenium had what we needed to make testing a success.”

Megan Collins

QA Manager, America's Test Kitchen

Going Beyond Selenium

While Selenium is a fundamental part of the America’s Test Kitchen QA strategy, CrossBrowserTesting helps in other ways as well. With other functions such as Live and Visual Testing, the team is able to find the perfect balance of manual and automated testing.

Not all tests can be automated, which is why America’s Test Kitchen relies on CrossBrowserTesting’s Live Testing feature for the real device coverage needed when performing tests manually.

Even though stakeholders may not be in the same office, testing in the cloud makes it’s easy for different contributors to discuss a bug or new feature remotely, and anyone on the team can access the specific browser or device without having it on hand.

Video capture has also been an asset for recording videos and sharing links to clarify a confusing test. This has been crucial for helping the QA team identify and share bugs that may be hard to reproduce, so they can identify what needs to be fixed and determine how to fix it.

Additionally, the QA team has been able to set up a database of homepage images over time for each site with CrossBrowserTesting screenshot automation, allowing them to visually compare how each site changes whether they want to evaluate day by day or year over year.

Complementary Ingredients

Implementing test automation takes time, dedication, support, and a healthy mix of manual testing to be successful.

America’s Test Kitchen uses CrossBrowserTesting in several ways to meet these needs from building and running tests to organizing results and collaborating with stakeholders.

The team has experienced tremendous growth on their road to test automation, expanding to three QA members, refining their Selenium skills, strengthening their manual testing strategy, and increasing test coverage thanks to a powerhouse QA team and a little help from CrossBrowserTesting.

As the America’s Test Kitchen website content and user base continues to grow, they can be confident that their testing strategy is built to scale

“CrossBrowserTesting offers a great amount of browsers whether it's IE, edge, or older versions of Chrome on desktop or mobile devices.”

Michael Ng

Automation Engineer, America's Test Kitchen