Why do screenshot tests fail or return unexpected results?
Since our screenshot testing service is based on launching a real browser and interacting with a live web page to get screenshots, there can be certain obstructions to getting intended results. Listed below are some limitations that may cause a screenshot test on a particular browser to fail or return unexpected results for a web page.
To see an overview of the screenshot test process, see the following FAQ:
How do screenshot tests work?
First we will define a some terminology so that the descriptions below will make a little more sense.
- Windowed Screenshot
- Image of a web page within a browser.
- Fullpage Screenshot
- Image of a fully scrolled web page from top to bottom within a browser. This is created by scrolling a web page, calling screen captures, and appending, or pasting, the images together.
- The viewable area of the web page that can be seen within the browser. This area changes as a web page is scrolled vertically.
- Screen Capture
- A snapshot image of the browser during the screenshot creating process.
- The software that "drives" the browser during the screenshot process, backed by the Selenium browser automation framework.
Page Load Timeouts
A web page has about 30 seconds to load such that the browser returns a signal that the page is ready. In some cases, after the 30 second timeout, additional steps are made to verify or not if the page is in fact ready but just has not communicated this back. Within 30 - 60 seconds of requesting the page, if the ScreenshotDriver is unable to begin interacting with the page, the system will give up and mark the test as failed. This issue can be related to operating system, browser, or network issues, as well as slow response time by the website requested or lots of elements on the page itself. Ensure your page is able to load in a timely manner to help avoid timeouts.
Long Web Pages
An image is captured for each scrolled viewport for later building the fullpage version of the screenshot results. There is a limitation on the number of viewports that the system will scroll. This is due to memory restrictions for storing every image necessary during the process, as well as the fact that some web pages will continue adding content to the bottom of the page and theoretically never reach the end. The current limitation is 25 scrolled viewports for desktops and 40 viewports for mobile devices. If the web page surpasses this limitation, the resulting fullpage screenshot will be cut short, or may even fail to complete succesfully.
Growing or Shrinking Web Pages
Rotating Content / Images
Fixed Position Layers
Some web pages are programmed to use "fixed" position layers such that an HTML element remains in it's same position within the viewport as a page is scrolled. A common example is a custom toolbar for interacting with the website at the bottom of the page. Since fullpage screenshots are created by scrolling the page, getting screen captures, and pasting the final images together, the result will show this fixed position layer several times down the image.
Web Page Rendering
Framesets & Page Scrolling
Most iframe elements on a web page are not a problem for the ScreenshotDriver, however, they have been known to cause issues with the browser signaling that the page is loaded, since all frames on a page must return this status before this signal occurs. The ScreenshotDriver has a few catches for this case and attempts to move forward after a page load timeout but iframes can cause the screenshot test to fail in some instances, especially with Internet Explorer tests.
Since the ScreenshotDriver runs tests against actual browsers, some results will show browser messages and confirmations such as saving passwords, language translations, permission to continue, https errors, and more. These can be "pop up dialogs" (mostly older browsers) that prevent interaction with the browser or "information bars" (newer browsers) that are just visual obstructions. Even though the ScreenshotDriver makes attempts to handle these cases across the wide variety of browsers offered for testing, some screenshot results will show these dialogs and messages. This is both good and bad. One one hand, it's not exactly helpful in verifying the layout of your web page, but on the other hand, it might be good to know that the particular browser will display this type of message to users on this page.
To get screenshots of a web page that requires form-based authentication, a user creates a login profile when creating a new screenshot test. There are three HTML selectors, or "locators", for finding the login fields on the page; the username field, password field, and submit button. The ScreenshotDriver uses these references to locate the input fields on the page, plugin the relevant values, and then click submit. Here are couple of concepts to be aware of for this process to work as expected:
- The HTML for the input fields on the login form should be the same across all browsers and platforms being tested.
- There is more than one way to specify the element on the form: id, name, xpath. If a browser has trouble logging in, try another locator type.
- The login button should submit the form and load a new page.
- The ScreenshotDriver expects the URL requested to contain the login form. A separate login page cannot be specified.
- If the login process takes the browser to a different URL, the originally requested URL will be reloaded. Example: the screenshot test URL is "http://mypage.com" and after login the page is at "http://mypage.com/login_successful", the ScreenshotDriver will reload "http://mypage.com" before a screenshot is performed.
Basic or HTTP Authentication
Depending on the setup of directories on the website and the browser's behavior, screenshots may fail or show the username/password prompts for sites with folder protection. This usually occurs when there are .htaccess files in subdirectories with different rules, calls for redirects, or resources requested outside of the authorized folder, in which case the browser will prompt for the username and passowrd multiple times, causing the ScreenshotDriver to be blocked and the test to fail.
Invalid SSL Certificate
When a user clicks "Take Screenshots" for a new screenshot test on our website, the URL is checked to ensure a proper status is returned. In the case of SSL Certificate issues, a warning is displaying containing the nature of the error with option for the test to continue, disregarding the error. Depending on the configuration of the browser, the screenshots will fail or succeed. We've attempted to configure each browser to be lenient with these certificate errors, but some cases cannot be bypassed. If you feel you are getting certificate errors in the browser when you should not, verify your SSL and contact us with details.
Please contact us if you did not find help with an issue you have listed here. We are always looking for feedback to improve the screenshot service and answer questions you may have.