CrossBrowserTesting.com

A Design, Development, Testing Blog

  • Product Overview
  • Pricing
  • Back Home
  • Free Trial
  • Design
  • Development
  • Manual Testing
  • Test Automation
  • Visual Testing

Why Your Tests Are Suddenly Running Faster

February 5, 2020 By Harold Schreckengost Leave a Comment

Over the past few weeks, you may have noticed that your Selenium tests on CrossBrowserTesting are running faster and with more consistent timing.  This isn’t a fluke or a random variance. We made significant changes to the way our Selenium Hub works in our infrastructure that greatly reduced the time it takes to execute each selenium command.  This change along with reworking some back-end load balancing means shorter overall test run times as well as less variation between test runs.

By moving our Selenium hub infrastructure to Kubernetes, we have not only improved our performance (quite drastically, as you’ll see), we have made it easier to update, troubleshoot, and scale.

The Numbers

Before the change, our internal load test timings on our Selenium hub were good, but not where we wanted them to be.  With our new Selenium hub, we not only hit our performance mark under load, but are also now delivering a much better experience to the regular users of our service.

You can probably tell just from glancing at the above graph exactly when we put the new hub in place – the sharp drop in test time on December 6th is due to the new Hub being placed into production.  In the above 1-week graph, you can see that our average command execution time dropped from approximately 25 seconds to right around 12.5 seconds – half the time of the previous week.  The daily average metric (not pictured) bears this out as well, going from 22 seconds to 11.

As mentioned, our timings are also more consistent – while I don’t have metrics through the end of today (as of time of writing), this graph of our standard deviation (a measure of variance) shows quite clearly that our variance in timing has diminished, from 11 a week ago to 7.  This means that not only do tests run faster overall, but they will run in a narrower band of timings.

How We Made This Happen

Traditionally, we have had a Selenium Hub service running on several physical servers inside our data center.  While this was easier to understand and maintain when it was originally implemented, the needs of our Selenium Hub, and our users, was well on its way to expanding outside what we could support with the existing infrastructure.  We could expand our Selenium Hub to more boxes, but this becomes a maintenance bottleneck, and, as I mentioned previously, toil is expensive and can reduce productivity well beyond the time taken to handle it.

So we built out a Kubernetes cluster that runs on some of the hardware we have recently repurposed, and after extensive testing, we felt comfortable rolling it out to users.

This was a fairly straightforward process, which was great, because this is the first time we’ve used Kubernetes for any of our workloads at CrossBrowserTesting.  We also had another advantage – because of the way that our Selenium Hub was written, putting it into a Docker container required exactly zero code changes to make it work properly within a container, only requiring setting a few environment variables, exposing a port, and picking a starting point to build from.

We knew going in that Kubernetes offered certain things, but overall, we found these to be the biggest points in favor of using it for workloads:

  • Completely decoupling software from the environment it’s running in is mindbogglingly helpful – if we want to update software to work on a new version of Python, that change is as easy as rebuilding on a Python image, and there’s no need to worry about the impact it might have on other software using Python.
  • Deployments are now safer, faster, and easier. Writing push-button deployments for Kubernetes is easy, especially when you start using Helm on top of it.  The deployment process is considerably faster, with most of the heavy lifting being handled by built-in components.
  • Changing resource configurations becomes incredibly simple – if we want to change the count, scaling bounds, scaling parameters, or even change to a completely different image, that’s a single clean roll out away.
  • Adding resources to a cluster is as easy as setting up a server and joining it using a token.  From there, that node can immediately begin handling some of the workload, which becomes especially useful when we start adding in Kubernetes built-in auto-scaling functionality.
  • Having a unified way to handle logging and metrics makes working on these systems much simpler and more efficient.

Of course, there is no silver bullet to any problem.  While Kubernetes has incredible power and flexibility, there is always a cost:

  • Kubernetes may not work for every workload, and this may include yours.  Stateful applications will have trouble running well in Kubernetes in a meaningful way, and the very small overhead imposed by Kubernetes and Docker may cause an issue if your workload requires the absolute most performance your hardware is capable of.
  • Kubernetes and Docker are ecosystems unto themselves, and if you don’t have someone who can read the docs and glue things together, it becomes much more difficult to actually make it all functional.

Overall, I think this was an important change at CrossBrowserTesting – not only have we made a vital user-facing service considerably higher performance and more resilient, we have set out a path for ourselves to begin improving even more of our platform.  We already have some really exciting plans in the works for the future to make the platform better, faster, more resilient, and more useful no matter who you are!

Filed Under: Development Tagged With: automation, Kubernetes, Selenium

August Product Update

August 13, 2019 By Joan Liu Leave a Comment

August Product Updates

Hot out of the hottest month on record, here are product updates from the last month. It’s a short one but it highlights some valuable updates from the team. Be on the lookout for some bigger updates in the coming months!

Free automation minutes

Live plans now have access to a limited number of automation minutes per month. This means you can now launch screenshot tests via our API, use our Replay suites, and schedule recurring screenshots and replays. This is a great option to set up some very basic automated tests of your site or application. The minutes are reset every month on your billing date and there is no rollover.

Here’s a refresher video on our scheduling feature.

More Filters

We received some customer input on filters they would like to see and we’ve added two. You can now filter screenshot results by those that have annotations.

We also added a filter so you can easily see your running or queued tests.

New Browsers and Operating Systems

We released Firefox 68, Chrome 76, and Opera 62.

Bug of the Month

We noticed that some of our Edge browsers were displaying pages a little strangely. Digging into this issue, we discovered that Edge thought it was supposed to be displaying the mobile view, even though it was a desktop configuration. After some investigation, we determined the issue was due to a race condition between our code to resize the browser to full screen and code to launch the browser. We have fixed this Edge case.

Try out all the new improvements for yourself. Log in now to get started. 

Filed Under: Product Update Tagged With: automation, browser update, product update, screenshot comparison, screenshots

Selenium Academy 101 FAQ

August 7, 2019 By Nick Brown Leave a Comment

Selenium Academy 101 FAQ

With the conclusion of our very first Selenium Academy class we wanted to ensure we armed you with the answers to some of the most common questions that came up during our live QA session!

In our Selenium 101 class we started at the very beginning by explaining what Selenium is, why you will want to use Selenium for your UI testing and especially why you will want to use Selenium with CrossBrowserTesting. But let’s take a look at some of the most common questions that came up during last week’s webinar.

What is the difference between a Selenium grid and CrossBrowserTesting?

While you can make your own Selenium grid with a costly device lab or VMs, many have found moving their Selenium grid to a third-party cloud like CrossBrowserTesting is faster and more productive, allowing more tests to run on more machines in less time. With over 2050 browsers to choose from, the possibilities of parallel test execution with CrossBrowserTesting are almost limitless. Our browser, device, and operating system selection includes all the latest configurations as well as legacy browsers so you can test what your users are on without worrying about any of the associated costs or upkeep. There are certain real costs of building an in-house environment as well as countless other secondary costs of not going with a cloud provider, like CrossBrowserTesting.

Selenium is open source right, why would we need to buy CrossBrowserTesting?

We can give you 8 good reasons!

  1. We make it easier to get started.
  2. Just install Selenium bindings for your programming language of choice and be done!
  3. Eliminate the work of managing and maintaining Selenium Servers, WebDrivers, and Browsers
  4. Screenshots, videos, and even network data can be captured during a Selenium test and shared with anyone
  5. Higher concurrency means faster execution
  6. Run dozens of tests at once without having to run dozens of RAM-hungry browsers
  7. Spend less time waiting and more time doing what matters
  8. Instant access to thousands of browsers and devices, including mobiles

What languages are supported with Selenium?

C#, Java, JavaScript, PHP, Python and Ruby

Which language is the easiest to get started with?

When talking with users about what language is easiest to get started with, we always recommend Python. The reason we recommend it for beginners is because it reads like English and tends to be easier for users to understand when getting started. Take a look at our blog on the Best Programming Language to Learn First.

What’s the best way to set up tests to run within a build pipeline using Jenkins?

There are only two things you need to be able to run your selenium tests from a Jenkins pipeline. You need a shell script or batch file that Jenkins can call to start your test, and Jenkins needs access to any libraries or modules that you use during the test – including the Selenium language bindings. If you are using Python, you can use VirtualEnv to create a virtual environment for Python with Selenium already installed. If you are using Java, make sure to include Selenium in your pom.xml file and run mvn build.

Does Selenium have the ability to open two webpages at the same time?

Yes, Selenium can be used to easily manage multiple pages at the same time. You can use the built-in command “switch_to_window()” documented here: Learn More Here

Is it recommended to use find element by xpath always?

Quite the opposite. Using Xpath to find elements, especially absolute Xpaths, tends to lead to very brittle tests. Any small change in the hierarchy around the element you are locating could cause the wrong element to be selected. Those bugs in your Script can sometimes be hard to debug. You should save yourself the trouble and use element IDs as much as possible

So, find_element_by_id is the best function?

Yes, it’s the most resilient to change and the easiest to use. If you can, add human-understandable IDs to any elements of your page that you want to automate.

For AngularJS based applications, do you have any specific built in functionality?

CrossBrowserTesting does not provide specific functionality to test websites built with AngularJS, but CrossBrowserTesting does work with Protractor – a Selenium-based framework created by Google for testing AngularJS. We have a short guide on getting started here: Help Doc for Protractor

How well does protractor work with just regular Angular?

It works great! That’s what it’s made for. If you run into any problems, feel free to reach out to our support team and they’ll help get you up and running.

Want to register for our 201 class on August 27th? Register now!

Filed Under: Test Automation Tagged With: automated testing, automation, FAQ, Selenium, selenium 101

Webinar: How to Increase Test Coverage Over Time with Automation

July 30, 2018 By Alex McPeak Leave a Comment

abstracta test coverage webinar automation federico toledo crossbrowsertesting

abstracta webinar test coverage automation federico toledo crossbrowsertesting

When your application has endless possible test cases, do you know how much coverage is enough?

Test coverage gives us ideas of what to test and helps us know the extent of our tests. We often use automation a tool to increase test coverage by checking more parts of the application under test, faster.

But complete coverage is nearly impossible to achieve even with automation. And sometimes, more coverage isn’t always better — it’s a great measuring stick, but iti doesn’t guarentee that our application is bug-free.

The ideal amount of test coverage should be based on your priorities and analysis of risk. The good news is once we know which test cases to prioritize, we can apply long-term coverage criteria to optimize test coverage over time.

If you want to know more about how to get the most our of your test coverage, join us on August 7th at 2 PM ET.

CrossBrowserTesting joins Abstracta Co-founder and Director Federico Toledo in a webinar to discuss strategies for measuring test coverage and maximizing it for the long-term.

In this webinar, you will learn:

  • What test coverage is and how to measure it
  • Laying out a plan to optimize test coverage in the long-term
  • Demonstration with CrossBrowserTesting
  • Q&A

Save your seat

Filed Under: Test Automation Tagged With: abstracta, automation, test coverage, webinar

Why You Need to Stop Relying on Developers to Perform All Testing

February 7, 2018 By Alex McPeak Leave a Comment

testing by developers why you should hire software testers why developers shouldn't automate

testing by developers why you should hire software testers why developers shouldn't automate

As companies decipher why testing matters based on the cost of a bug or a customer pointing out their mistakes, teams that produce and deploy software applications ultimately come to terms with the fact the testing should never be skipped.

However, some people think they can get the job done without paying the price — the price of a software tester, that is. While testing might seem simple at first glance, there are countless factors that go into effectively evaluating an application.

Organizations that rely on developers to do the job will quickly run into roadblocks that prevent on-time delivery and impact customer satisfaction. Here are a few reasons why your software team needs testers:

  1. Testing is a specialized skill set – Developing and testing are two different jobs with two different skill sets. Just like a doctor shouldn’t try to do a nurses job and vice versa, building an application and having an understanding of the techniques to properly test and provide feedback for that application are two different things. While it a developer could maybe perform a high-level test of the application, a tester is truly needed to take a deeper dive and handle more complicated procedures that go beyond a basic click-through.
  2. Testing takes time – Testing is not a quick one-and-done. In fact, most testers find that they don’t have enough hours in the week to get to everything they wanted to test. A big reason behind this is that it’s virtually impossible to test everything — there are limitless possibilities and there will always be some part of the application that’s going to go under the radar, no matter how much time and effort is spent on it. This means that if your developers are doing the testing, they will either be skipping a lot of tests that they need to run due to time split between their responsibilities, or they’ll be doing an acceptable amount of testing that will keep them from doing their main job. Let your developers develop and your testers test so you can stay on track for frequent integrations, fast delivery, and quality software.
  3. Knowing how to code ≠ knowing how to automate – Learning a programming language is the first step to getting started with automated testing, so it might seem natural to have developers learn to automate and perform checks because they already know how to code. However, you’ll find instead that the execution doesn’t go as smoothly as expected. There’s a lot to learn when it comes to automation, and coding is just a very small part of that. You need to know when to automate, how to write comprehensive scripts, how to approach challenges with automation tools, how to manage test cases, how to maintain tests, and much more. This comes back to the idea that testing takes time and a special skill set — it’s more likely you’ll find your developers are frustrated and falling behind than making any strides in automation.
  4. Reporting and documentation are important – A test that’s run but isn’t reported or documented is only so helpful in the long run. Even if your developer is willing to run through their own part of the application, it doesn’t mean they’re making scripts that are maintainable and reusable. And it definitely doesn’t mean they’re going through different parts of the application from other developers to conjure effective integration and end-to-end testing strategy. If all your developers are testing their own code, all you’re going to end up with is fragmented information that probably won’t do much good catching bugs anyway.
  5. You need to be in the right mindset – If you’re testing your own code, you’re not really accounting for new scenarios. That’s to say that if someone writes a feature, they’re only going to test for the use cases they already considered in development. Having a tester come in brings a fresh set of eyes to look at the feature in new ways that the developer may not have thought of. It’s like having someone edit a paper for you — oftentimes they’ll find mistakes or make suggestions that you missed no matter how many times you read it yourself. A tester brings this same value to the table by not only finding errors that were missed but also providing insight.

Having your developers test their own code is a poor use of time, money, and skills. While you may be tempted to resort to using your developers to quickly test a new integration before release, it’s important to understand that there’s a good chance you won’t get the results you want.

Hiring for testings and bringing in a team that is specifically trained to inspect software quality and focused on finding and reporting bugs will not only improve time to market but also increase the capacity of your developers and improve the software you release. Don’t skimp on testers the same way you know not to skip testing.

Filed Under: Development Tagged With: automation, developers, testers

The Most Common Selenium Challenges

August 3, 2017 By Alex McPeak Leave a Comment

Selenium WebDriver Challenges

Selenium Challenges

Selenium WebDriver (Selenium) automates the browser to mimic real user actions on the web. As the web gets more complex, using the Selenium testing approach has become increasingly popular. Although it has made web testing far easier for many teams and organizations across the world, it still has its fair share of challenges due it’s open source nature.

Most of the issues that testers experience have fairly straightforward solutions, which is why we outlined the most common Selenium challenges and how to fix them.

Pop-up Windows

When a simple, prompt, or confirmation alert pops-up, it can be difficult to automate it to either accept or close. Additionally, Windows-based alerts are beyond Selenium’s capabilities since they’re part of the operating system instead of the browser. However, since Selenium WebDriver can operate multiple windows, web-based alerts can usually be handled with the switchTo method to control the pop-up while keeping the browser in the background.

Learn more about handling pop-up windows with Selenium.

Dynamic Content

A web page that has dynamically loaded content has elements that may not be at first visible when you visit the site. This can mean that the page is user specific and will display different content based on different users, new content appears on the page after a certain amount of time, or it appears after clicking something on the page. For when a locator cannot identify a new element present at that time, Selenium WebDriver comes with an integrated Explicit Wait where you can specify an amount of time to wait before automating a command. This should help give the page enough time to load and identify the element to interact with.

Learn more about testing dynamic content with Selenium.

Flakiness

Sometimes Selenium will give you flaky tests, which means they’ll come back as positive or negative when they actually are the opposite. Eventually, too many flaky tests may mean testers might ignore the results, though this isn’t a great option either. Oftentimes, unstable builds can be a product of one of many factors: poor test isolation, flaky external services, and timeouts. By examining the elements of your Selenium tests, it’ll be easier to find out why your builds are unstable so you can approach the problem head-on.

Learn more about managing flaky Selenium tests.

Mobile Testing

While Selenium WebDriver can test on any operating system and browser on desktops, it’s limited when it comes to mobile testing in that it cannot run on native operating systems like iOS and Android. So, for example, the version of Facebook you pull up on Safari on your Mac can be tested with Selenium, but not the one on mobile Safari – on your iPhone 7. The Selenium family has a solution. It’s cousin, Appium, is an open source test automation framework that drives iOS and Android native, mobile, and hybrid apps using the WebDriver protocol and can be used for this purpose for testing mobile apps instead of web applications. As users increasingly move to mobile devices, it’s no surprise that developers and testers are taking advantage of the functionalities of Appium for this purpose.

Learn more about automating with Appium.

Limited Reporting

While Selenium will exponentially increase your automated testing capabilities, because it’s an open source tool it is limited and features and does not support much reporting on its own. For this purpose, Selenium-based testing is best complimented with a third-party tool like CrossBrowserTesting that can capture browser screenshots and share reports through integrations like Slack and HipChat. However, you can also set up a framework to generate an output folder after a test with report information like errors, execution time, pass/fail count, etc. like TestNG.

Learn more about generating reports.

Multi-tab Testing

Selenium WebDriver has the ability to test in multiple tabs, but it might present as an obstacle if you don’t know the correct commands. For example, if you want to open a new tab to perform a specific action without leaving the original tab, you’ll still want to be able to control both tabs and go back to the parent tab if users are likely to perform these kinds of actions. The best method for this is to store the previous window handle in a variable and then continue in the new window and store that in a second variable. Then, you can use the switchTo method to go between both windows as needed.

Learn more about how to test multiple tabs.

Manual Testing

Many testers will get excited about Selenium’s capabilities and expect that they can automate everything when the reality is that this just isn’t possible. In fact, a lot of automation relies on basic manual testing efforts. Rather, learning to prioritize which tests to automate will be beneficial to any tester. Additionally, as we mentioned, Selenium WebDriver lacks capabilities for visual testing, which is important for testing beyond functionality to assess overall user experience. The most efficient way to do visual testing is by outsourcing to a third-party screenshot tool that integrates with Selenium.

Learn more about the need for manual testing, prioritizing automated tests, and how to take screenshots during a Selenium test.

Scalability

Again, while Selenium WebDriver allows you to test on pretty much any browser or operating system, it’s still limited in how many tests it can run at once and how fast it can run them based on how many hub/node configurations the tester has. And again, as your software scales, so must your testing. Without a Selenium grid, you can only test sequentially. However, with a Selenium grid and a device lab or a third party cloud tool like CrossBrowserTesting, you can test in parallel. This means multiples tests on different configurations can run at the same time, reducing the time it takes to run automated tests and increasing the configurations you can test on.

Learn more about parallel testing.

Conclusion

One of the best things about being a software tester is that there’s always something new to learn. As the industry shifted from manual to automation, testers were tasked with learning new skills such as programming.

Now, as they become more familiar with automation techniques and tools such as Selenium WebDriver, it’s admissible that they’ll encounter more challenges as they explore and familiarize themselves with new testing trends. Fortunately, as software continues to evolve, experts can depend on insights and support from the rest of the testing community to make sure they’re keeping up. Pick your preferred programming language to get started with Selenium testing here.

Filed Under: Selenium Tagged With: automated testing, automation, challenges

9 Signs You Should be QA Testing in the Cloud

June 7, 2017 By Alex McPeak Leave a Comment

cross-browser testing QA testing in the cloud

cross-browser testing cloud testing

We’re all familiar with the cloud. As a topic that’s clearly trending and growing in software, our appreciation of the cloud testing stems from its innovation as well as its effectiveness.

The cloud is only projected to become more popular and more common in years to come. In fact, 95 percent of organizations are using a cloud in some way, and the market is supposed to increase 22 percent annually according to SD Times.

In software development, cloud testing provides an easy-access, remote testing environment to produce real user scenarios. As organizations consider the benefits of the cloud testing, here are some signs that tell you it’s time to use it for browser testing.

  • You’re starting a business – One of the best times to check your web design across browsers are in the early stages a.k.a the exploratory tests. New companies should ensure they establish the UI of their website to look and function responsively across multiple devices when quality can make or break a business reputation.
  • You’re moving from manual to automation – Cloud testing offers an easy onboarding for organizations whose testers are a little newer to programming and automation. As you continue to recognize the benefits of automating your tests, the cloud is a practical way to make the process pain-free. With on-demand access, it’s easy for your team to test whenever they need and learn as they go. Additionally, browser screenshots, extensions, frameworks, debugging tools, and integrations are included to optimize usability and make testing more enjoyable. Plus, customizable preferences and customer-centric user interface assure an optimal user experience.
  • You’re realizing there aren’t enough hours a day to test everything – If you find yourself with test cases that are taking more time than there are hours in the day, it might be time to think about adding some more machines to your parallel testing. Parallel testing is a feature exclusive to cloud testing, which will allow you to significantly cut down your testing time by testing different cases on multiple machines at the same time, instead of one after the other. Additionally, the cloud is available all day every day, so you can test whenever you want without having to be on-site. In fact, you can even run automated tests in your sleep. This will make it easier to test all the configurations you need to, while still meeting deadlines.
  • You have a hard time keeping up with all the tech – Software testing is not what it used to be. With so many new browsers, OS versions, and devices coming out all the time, it can be difficult to keep track of all the configurations you need to test on. With cloud testing, new releases are added for you so you don’t even have to think about it.
  • You think new smartphones are way overpriced – If your device lab budget isn’t quite allowing all the configurations you need, cloud testing might be just the right answer. When an average Android is about $670 and an iPhone rounds out at around $700, the numbers can add up fast when you try to test on every device your user has — not to mention maintenance.
  • You recognize the benefits of simulators, emulators, and real devices – While emulators and simulators have their place in testing, real devices are crucial in order to test on the machines that your users are on. To get the most comprehensive breadth of simulators and emulators plus real devices for testing, cloud testing is the best way to access all three for different purposes, and the combination ensures that you always have access to testing.
  • You’re scaling your team – You had QA all figured out when you had just one (albeit really talented) tester. But, as your company scales, so must your testing team. When everyone’s trying to complete different tasks that all seem to be of top priority at the same time, the cloud gives you more room to accommodate growing pains. Since everyone has access to the same devices at once, you can ensure you’re supplying your growing team with the resources they require.
  • You have testers in different offices – Similarly, it’s a little bit difficult to share the capabilities of your device lab among a QA team that’s spread out among different cities, countries, and continents. Cloud testing provides one platform that further encourages communication between different departments and remote teams with integrations like Slack sharing.
  • Your customers are getting demanding – Has your customer support team been extra stressed out lately? Are you getting user reports saying that your website is slow, or there’s a bug, or it’s not working on their browser? Your customers have high expectations of you. They might be using a wider variety of devices than you think, and it’s compromising their interaction with your web application.

What pushed you to make the decision to adopt a cloud testing service? Tell us in the comments!

Filed Under: Test Automation Tagged With: automation, cloud, parallel testing

What is Cross Browser Testing?

March 8, 2017 By Alex McPeak 1 Comment

what is cross browser testing

Browser testing is a method of quality assurance for web applications across multiple browsers. That is to say, it ensures the quality of your website on different screens. It’s implemented to ensure a website’s functionality and design and includes testing a range of devices and operating systems being used in the market and customer base. On top of that, screen size, screen resolution, OS version, and browser versions all change and contribute to how someone is viewing content, making the practice of cross-browser testing increasingly necessary for understanding diversified user experiences.

While fundamentally self-explanatory, browser testing has an expansive amount of components. Understanding the many factors can exponentially impact your web application and increase customer satisfaction while ignoring its importance can have a negative effect on reputation and bottom line goals.

Why it’s Needed

There used to be only a handful of browsers existing across the internet, and they were mainly situated on desktops. Additionally, with a waterfall model design process where developers were primarily making quarterly or monthly changes to code, manually testing each one was relatively simple.

Times have changed, however, and today’s consumers are on a multitude of devices ranging from desktop computers and laptops to mobile phones and tablets, while there are too many browsers and browser versions to count at this point. Furthermore, to parallel the rise of mobile and desktop internet consumption, design and development have moved to an agile methodology. Developers are no longer making yearly updates but instead instituting continuous integration with updates by the hour or day.

Meanwhile, each browser responds to these code changes differently, and so with every change of code comes a risk that any of those browser/device/OS combinations may not respond according to expectations. Layout, accessibility, connectivity, quality, and performance can vary slightly or exponentially from browser to browser, as each renders CSS, HTML, and JavaScript differently.

Developers inherently know and learn the lowest risk browsers and devices, and in turn, use them every day to view the product of their coding. However, a misconception is that another user will be visiting the same website on the same browsers and devices. “It works on my machine” is not a valid argument.

While about half of users will be on the popular, low-risk browsers that developers use such as Chrome and Firefox, the bugs will mainly be saturated in high-risk browsers like Internet Explorer. You could direct a user who’s experiencing issues on a bugged browser to either update their browser version or download different browser altogether, but it’s more likely that user will be unwilling or unable to switch browsers just to properly view your website. Instead, they’ll just leave your page.

Though it may be unrealistic to test 100 percent of all browser platforms, it’s important to create an acceptable user experience amongst the majority of your user base. The more browsers that are tested and optimized, the more inclusive your site will be for a variety of visitors. This is not to say that each should look exactly the same, but instead that there is an element of responsive design that has been implemented and tested.

Bottom Line Goals

As a developer, your job is to make your website accessible and appealing to the broadest range of users.

When someone on Internet Explorer is able to access and enjoy a website the same way somebody on Google Chrome is, it relays a dedication to the user experience and a thorough design process. This means that consumers are less likely to encounter bugs and off-putting layout elements, and most likely to foster a sense of brand loyalty.

Not only does browser testing encourage a positive reputation, it also impacts bottom line goals when users can easily view content and experience optimized functionality no matter what device, operating system, or browser they access it from.

How it’s Done

Pick Your Devices

The first thing that needs to be done is to decide how many browsers you want to test on and which ones. A popular method to determine where the majority of your audience is saturated is to measure traffic with Google Analytics, although keep in mind this only provides a sample of your visitors as well as potential visitors.

Decide on Execution

After you choose which devices and browsers you’re going to test, you have two main options for execution — building a device lab or testing in the cloud with a tool like CrossBrowserTesting. With both options, you have to decide whether to test on a real physical devices or a desktop or mobile emulator.

First and foremost, we recommend real devices because they replicate the actions of the user, while the results received on emulators tend to vary from a true user’s experience. However, it’s also important to include emulators at different stages of testing, as well.

The cost of a device lab, of course, will vary with how many machines you want to buy and test. We estimate that the minimum cost of purchasing to include all the essential devices and operating systems is around $10,000, but this will only reach about half your user market. This number also doesn’t consider the cost of replacing the devices every few years.

In this case, VMs may seem like an attractive solution to save money at a base of about $2,500 a year if you’re willing to give up some accuracy. However, the hidden costs of licenses, maintenance, and updates can result in unforeseen expenses that will continue to add up over time and put the two at comparable costs.

The other option is to go with a testing platform where the added value is in cost, speed, features, and spectrum of devices. Since cross-browser testing platforms charge a monthly fee to test on hundreds or thousands of devices, you’re likely to have a greater range of configurations to test, while saving quite a chunk of change.

Additionally, parallel testing is exclusive to automated cross-browser testing tools, allowing you to simultaneously run a multitude of tests in a fraction of the time, often cutting hours and even days off of a boring and laborious manual testing regime.

Included features such as browser screenshot comparisons, live testing, and local testing further simplifies the process for testers and developers so they have the time to focus on designing a better product instead of individually testing numerous browsers and devices every time a code is updated.

Begin Testing – Manual and Automation

Where you’re at in your development and design process always has a say in the way you’re going to execute your browser testing.

For early-stage development, one-off or exploratory testing is usually appropriate and can be done through manual testing. Once you’re further along in the process, regression testing and functional test automation is going to provide the breadth and expansion required to understand numerous coding elements and detect bugs.

No matter how you’re testing, you should be considering how your site both looks and performs across numerous browsers. Whether you manually test or automate the process, CrossBrowserTesting has a multitude of options to help save time, cut costs and improve the overall quality of your web application.

Filed Under: Browsers Tagged With: automation, browser testing, real devices

An Introduction to Selenium Open Source Automation Tool

March 2, 2017 By Alex McPeak 2 Comments

selenium testing webdriver

Selenium

Selenium is an open source testing tool that allows users to test web applications across different browsers and platforms. Selenium includes a suite of software that developers can use to automate web applications including IDE, RC, WebDriver and Selenium grid, which all serve different purposes.

In this post, we’ll explore the multiple components of the Selenium testing framework and its evolution as well as how it’s used and why developers and testers depend on it for browser testing.

What’s the Use?

The uses of Selenium increase as you advance in the suite, and each component has advantages and disadvantages. The basic fundamentals of Selenium though revolve around browser testing, as developers and testers alike love the open source framework’s ability to mimic a real user’s actions in the browser. Developers create test scripts that run locally or in the cloud to assess basic functional correctness and verify visual layouts. These test scripts can range from simple JavaScript unit tests around an action like a button click, or fully automated end-to-end functional tests that resemble the full actions of an everyday user, like the checkout or login process.

What makes Selenium so easy to learn is its ability to bind to several popular languages, letting you write in your preferred programming language including Java, Javascript, PHP, Perl, C#, Ruby, .Net, and Python.

A Brief History of Selenium

Selenium has a dynamic history since it came to being in 2004 when Jason Huggins, who was a ThoughtWorks programmer at the time, decided to make a tool for testing web applications. By the end of the year, it had become a wildly popular open source tool.

The first version of Selenium had many limiting factors like its Javascript based automation engine, causing Huggins to seek other contributors. Simon Stewart, another ThoughtWorks employee, stepped up to the plate and they produced Selenium Webdriver, or Selenium 2.0

The combination of Selenium and WebDriver in 2006 became a master web application tool know as Selenium 2.0 and is one of the leading free testing tools for programmers around the world.

While Selenium 3.0 has been released, it’s basically a new and improved version of Selenium 2.0 with the same fundamental purpose and structure.

Breaking Down the Selenium Suite

The Beginning – Selenium IDE

Selenium IDE (Integrated Development Environment) is the Firefox plug-in tool that allows you to record, playback, edit and debug test scripts to export them in a preferred programming language. By providing a test domain specific language, Selenese, IDE lets you test without learning a new scripting language and makes it easier to send commands to the browser and automate testing. However, IDE is limiting in the way that you can only use it to test web applications in Firefox, and therefore it cannot be used to test other browsers.

Selenium IDE is mostly used now for bug reproduction scripts, exploratory testing, learning about Selenese commands and exporting simple test cases to WebDriver. Selenium Builder is also a Firefox extension that can import HTML files recorded in Selenium IDE to run tests locally or on a Selenium server and further assist automation.

The Evolution – Selenium 1 and 2

Selenium RC (Remote Control), or Selenium 1, was a method used to run the test script in different browsers by launching them as a proxy server, but has since been abandoned since the creation of WebDriver, or Selenium 2.0. Selenium 2 is what you would use in testing today, as RC is no longer supported.

This is because WebDriver doesn’t rely on JavaScript for automation like RC does, and instead, directly communicates with the browser from the operating system making it faster and more accurate. WebDriver also supports more browsers and operating environments than RC including the HtmlUnit browser, and has a more concise API.

Selenium 2 is popular among developers and QA teams for regressions tests and cross browser testing, since it allows you to test on a spectrum of browsers and operating systems using the native operating system functionality. This means that commands perform as a human would if they were using the same browser.

Optimization – Selenium Grid

Selenium Grid allows you to multiple run tests at the same time on different machines and browsers in order to speed up testing. For CI motions,  testing demands to be done at a rapid pace, and running tests sequentially can be painfully slow. Running in parallel on a grid can improve the speed of your testing greatly. Selenium Grid is comprised of one machine called a hub and multiple nodes for execution. The more nodes you have, the more tests you can run simultaneously.

While you can make your own Selenium grid with a costly device lab or VMs, many have found moving Selenium grid to a third-party cloud is faster and more productive, allowing more tests to run on more OS systems and browsers simultaneously.

Filed Under: Selenium Tagged With: automation, browser testing, Selenium grid

Try CrossBrowserTesting

Everything you need for testing on the web. Browsers & Devices included.


  • Grid In The Cloud
  • Simple CI Integrations
  • Native Debugging Tools
  • Real iOS and Android
  • 2050+ Browser/OS Combinations
  • Intuitive REST API

Start Testing Today

Want Testing Tips?

Want the latest tips & strategies from industry experts right to your inbox? Sign up below.
 

Join Over 600,000 Testers & Developers And Start Testing Today

Learn more Free Trial

Features

  • Live Testing
  • Selenium Automation
  • Automated Screenshots
  • Screenshot Comparison
  • Local Testing
  • Real Devices

Solutions

  • Automated Testing
  • Visual Testing
  • Manual Testing
  • Enterprise
  • Internet Explorer

Resources

  • Browsers & Devices
  • Blog
  • Webinars
  • Integrations
  • ROI Calculator

Company

  • About Us
  • Careers
  • Plans
  • Terms of use
  • Security

Support

  • Help Center
  • API Docs
  • Schedule A Demo
  • Contact Us
  • Write For Us