CrossBrowserTesting.com

A Design, Development, Testing Blog

3 Methods for Hands-free Continuous Testing

By Leave a Comment

continuous testing with jenkinscontinuous testing with jenkins

In order to strike the perfect balance between speed and quality, teams that hope to achieve Continuous Integration and Continuous Delivery must also implement Continuous Testing.

This is often easier said than done. It’s virtually impossible to have your team running tests at every moment — they do have lives outside of work after all, and it can be difficult to put dinner on the table while running a Selenium script at the same time. So how do you run your scripts, even if you aren’t at your desk?

Fortunately, there are various hands-free options for continuous testing that you can implement, which will keep tests running for you while you eat, sleep, or maybe even get some exploratory testing done.

Why Continuous Testing

What’s the point of continuous testing? There are a few major advantages that will enable you to achieve more in your day-to-day.

  • Test while you sleep – With continuous testing, you can literally test in your sleep. By setting tests to run overnight, you can experience the luxury of coming in the next morning to have your test results waiting for you and knowing with a glance whether they passed. Just ask America’s Test Kitchen about how convenient it is.
  • Keep up with Agile, DevOps and CI/CD – When software development teams are constantly integrating, you need a testing strategy to ensure those new features, fixes, and changes are put through the quality process, no matter how frequently they’re implemented. Continuous Testing is the only way QA teams can keep up with constantly changing features, frequent integrations, and fast workflows.
  • Catch bugs in each regression – When there are constantly small changes to an application’s code, it can break something that was previously functioning. And you don’t want your customers to be the one to catch it. Continuous Testing will help you be first to spot new bugs, so you can quickly debug and release to users with confidence.
  • Level up your automation game – Automate your automation with continuous testing, and set your tests to run as often as you need. Then come back to test results or screenshots that tell you exactly what you need to know about the health of your applications. Plus, testing in parallel allows you to further accelerate Continous Testing to increase browser coverage without compromising runtime.

Continuous Testing With a CI Server

Selenium WebDriver has long provided teams with the ability to automate tests across browsers. But in order to run those tests continuously, a Continuous Integration tool such as Jenkins, VS Team Services, Buildbot, Bamboo, CircleCI, TravisCI, Codeship, or TeamCity is needed.

By far, the most popular option is Jenkins, an open source automation server written in Java. In fact, according to the SmartBear 2018 State of Testing Survey, 66 percent of participants choose Jenkins to be their CI server. For development teams adopting Agile and DevOps workflows, Jenkins is a reliable tool for continuous integration and delivery.

But what about testing teams that want to leverage Jenkins for Continuous Testing with Selenium? The CrossBrowserTesting integration with Jenkins lets you run automated functional tests with Selenium, Appium, and visual screenshot tests, which means that every time you run your build process, you can simultaneously run your testing suite across thousands of browsers.

This is ideal for teams that already work with Selenium and a CI tool for deployment to start testing on a continuous basis. While Jenkins tends to be the most popular choice for CI and automated test scheduling, CrossBrowserTesting also integrates with TeamCity and VS Team Services, so no matter what tools you’re already using, it’s easy to integrate CrossBrowserTesting into your current workflow.

Continuous Testing With a Record and Playback Tool

For teams that may be less familiar with Selenium but still wish to automate, a record and playback tool might be a more practical option.

Record & Replay enables teams to achieve automation across browsers, just like they would with Selenium, without requiring knowledge of scripting and frameworks. This has provided teams with a more accessible option for less technical team members, or those who are just getting started with automation, to create automated tests. As tools like Selenium IDE and its alternatives are brought back into the spotlight, these options are becoming more and more attractive to modern testing teams.

But where record and playback tools will normally only run the test once to check if the test passes, how can testers implement Continuous Testing?

With new Scheduling in CrossBrowserTesting, you can schedule a Record & Replay suite to run as often as you need them. For organizations that have less technical testers, this allows you to achieve continuous testing without requiring the skills to run Selenium and Jenkins or another automation server.

Continuous Screenshot Testing

Teams that solely focus on functional testing without incorporating visual testing will be disappointed when all their tests pass and users are still pointing out visual bugs.

When visual elements can change from browser to browser or device to device and make the user’s experience difficult or impossible while still functioning correctly, it’s important to run compare screenshots in regression. Comparing these pages side by side and with their historical versions means you can be more aware of changes that break the application in order to debug before they reach the end user.

While you could always run automated screenshots in CrossBrowserTesting, it required a manual motion to rerun those tests on a certain cadence. You can now set those screenshot tests to run with the new Scheduling feature, just like you can with Record & Replay.

Continuous Testing for Every Team

No matter what tools you’re already using, what technical level you’re at, or how far your team is in their test automation journey, there are options to help every team achieve continuous testing in CrossBrowserTesting.

With options to schedule tests with Selenium, Record & Replay, and automated screenshots, you can take your automation to the next level and receive test results through email or Slack on a daily, weekly, or monthly basis that you decide.

The State of the New Selenium IDE

By Leave a Comment

Selenium ide alternatives

Last week, Applitools teamed up with Simon Stewart for “Introducing Selenium IDE: The Next Generation”.

Selenium IDE is part of the Selenium suite and has been a depreciated record and playback tool, largely because it could only be used in Firefox and had stopped being supported by contributors of the Selenium project.

But don’t call it a comeback — Selenium IDE 2.0 (still to be officially named) is more so a new and improved version of its former self.

What’s New?

Some of Selenium IDE you might recognize from the first version, such as their UI. However, while it may look similar at first glance, you’ll soon notice many improvements including better locators and support for common tasks.

Potentially the biggest differentiator with the new Selenium IDE is that it’s built as a web extension, which is standardized by the WC3, so it no longer only works in Firefox.

This means that the new Selenium IDE can run in any browser that supports web extensions such as Chrome, making record and playback cross-browser testing much more achievable than in the previous version.

How to Install the New Selenium IDE, Record, and Play

In the webinar, Simon showed us how we can install Selenium IDE and run our first test.

In Firefox:

  1. Go to Tools
  2. Click Add-ons and Get add-ons
  3. Search Selenium IDE and click Add to Firefox
  4. Go to the Menu bar and select Selenium IDE

In Chrome:

  1. Go to Chrome Web Store
  2. Search Selenium IDE
  3. Add the extension (make sure to select the one from SeleniumHQ.org)

In the example, Simon does a Google search for “cheese,” which is as easy as firing up the new IDE, choosing to record a test, entering the URL you want to start with, and then going through the actions that you want recorded manually.

The new IDE also has options during testing such as assert and verify. If you want to edit the test steps after, you can do that, too.

As mentioned, there are quite a few value-add features and enhancements in Selenium IDE gathered from users over the years.

If you want to run tests with a command line, you can put tests into a CI system, or allow them to run locally with the Selenium IDE runner. The runner supports additional features such as parallel execution through a Selenium grid or a server like CrossBrowserTesting.

There are other enhancements with the new IDE, as well. Control flow statements enable more maintainable tests, whereas before, it was only available through plugins.

Additionally, fallback locators will provide better test stability and make tests more resilient despite changes. Previously, Selenium has been known as brittle and unstable because locators have been hard to write and maintain.

There currently is no option to export recorded tests like there was with the original IDE, but this is immediate on the product roadmap.

Selenium IDE Alternatives

As we discuss Selenium IDE, the conversation around record and playback tools is brought back into light.

More and more individuals and organizations are realizing the value of record and playback tools like Selenium IDE and Selenium IDE alternatives, such as Record and Replay in CrossBrowserTesting.

Both of these tools are ideal for the use cases that Simon reviewed in the webinar. Record and playback is helpful for:

  • Team members that don’t aren’t familiar with programming languages
  • Someone who wants to put together a bug report
  • Someone just getting started with automating a test suite
  • Automating boring and repetitive tasks

Of course, the benefits of Selenium IDE and its alternatives don’t stop there — the more teams adopt record and playback tools the more uses they’ll find.

So, whether you decide to stick with Selenium or try a tool like Record & Replay, it’s worth exploring the possibilities of a record and playback tool.

Next Steps for Teams Who Want to Move to Agile Project Management

By Leave a Comment

agile project management development

It seems that the companies we all aspire to be more like are moving quicker and releasing faster than ever before. So, how can the rest of us strive to follow in the footsteps of software giants that release innovative software on the hour? While there may be no single, clearly-defined answer, we do know that many are seeing success with agile project management and development methods.

By focusing on feedback from stakeholders throughout the development process, not only is agile found to be a faster development method, it’s also more effective in the way that it allows and encourages changing requirements.

However, as waterfall is the traditional methodology for software development, teams that are stuck in their old ways might find the transition to agile hard to wrap their head around, so here are 7 tips for teams who want to take the next steps and move to agile project management.

  1. Identify what or who is missing to make agile work. In order to move to agile, you first have to recognize that it will be process your entire team will have to get on board with and fully embrace in order to make it work. Identify the roles and people you need on your team to make a successful transition. It might be helpful to onboard a project manager, scrum master, or agile coach to help adjust the current workflow. Look at your current team, as well, to understand where there are silos and barriers to communication. Teams that don’t communicate will find it hard to be successful. Agile project management works best when this is cross-departmental collaboration, so it’s important to work from the inside out.
  2. Prepare to shift left. Agile will require more testing earlier in the software development lifecycle. Because each phase of the SDLC is done in short iterations, rather than long incremental ones like with waterfall, testing shouldn’t wait until the end when new features are fully developed. With a focus on continuous integration and continuous delivery that comes with agile development, continuous testing will be the key to fast feedback and iterative changes that are needed to release high-quality software. Evaluate whether you will need to introduce test automation, additional QA team members, new tools, or something else to help you reach maximum test coverage.
  3. Poke holes in what you’re doing now. Look at your current planning process, and outline how your strategy will have to change in agile development to allow for more feedback and flexibility. Where are the bottlenecks in your current process, and what prevents your team from delivering high-quality software or meeting short deadlines? This will be crucial to shifting to a new way of thinking and transforming your team’s workflow. Familiarize yourself with the different agile methodologies such as Scrum, Kanban, and Extreme Programming. While these all follow agile principles, they achieve it in different ways. Based on the makeup of your team and the objectives you discussed, it will affect the way you carry out agile project management.
  4. Get on board with sprints and standup. No matter which methodology you choose, a few things will probably be the same, and they’ll help you properly implement agile project management in your organization if you take the time to do them correctly. Sprints will be the duration of you planning — usually they’re two weeks long but can be anywhere from one to four — and standups are where you talk about what you’ll be doing in each sprint. Take advantage of these opportunities for strategic planning and collaboration with the entire team.
  5. Backlog issues and tasks. Identify the projects that currently challenge your team and what you’d like prioritize when you transition to agile. Create a backlog of tasks, gaps, objectives that need to be addressed so you can start ranking them on most to least important and having conversations about future goals. Discuss what is needed from each team member, how it will be accomplished, and what information is currently needed to make it as “done”. Once you begin to organize your team and the work that needs to be done, you can start building out detailed tasks and sprints.
  6. Don’t skip your retrospective. As you adjust to agile and the many changes that come along, you’ll be constantly learning about what’s working well and what isn’t, as well as what individuals are successful with and what they’re struggling with. Ensure that everyone has an equal say in each sprint, so it’s not just leaders and managers giving their two centrs. Focus on how you can improve in each sprint. However, don’t just wait until retrospectives to collect feedback from your team — encourage communication and checkins throguhout the sprint.
  7. Be open to change. Requirements change, tasks get de-prioritized, clients and customers change their mind, and feedback will alter the sprint process. The nice thing about agile project management is that it encourages these adjustments. Embrace agile and understand that these frequent changes are improvements that will help your team achieve continuous integration, testing, and delivery. If you feel like you’re getting lost or confused, always resort back to the agile manifesto to realign your team with the principles behind agile.

Finding your groove with agile won’t happen overnight, but there are certain steps your team can take to make the transition more seamless. For more, take a hint from what agile teams can learn from Netflix.

 

The Spectrum of Testing Practice: Why Are Some Companies Stuck in the 90’s?

By Leave a Comment

modern testing culture lisa crispin

Those of us who are passionate about delivering valuable, high-quality software to our customers frequently and at a sustainable pace are living in exciting times.

Many are embracing “modern testing” principles. We’re acquiring new skills such as how to help non-testing teammates learn to test, how to analyze production use data, and how to use that data for testing. Testing is at the heart of DevOps culture, providing new opportunities for testers. We have amazing tools to help us with activities such as regression test automation and learning from production monitoring, logging, and observing.

At the same time, I still encounter many companies who are doing testing the way most people approached it 20 or more years ago. They have siloed testing/QA teams who don’t collaborate with development teams, operation teams, or even product and design teams. They have no automated regression tests or are struggling mightily to get traction on that automation. They’re doing only manual regression testing, working from written scripts, and no exploratory testing.

Why??!!!

I’ve seen leading agile practitioners scratch their heads and wonder – after two decades of sharing agile values, principles, and practices with the world, why are so many people not using them?

Teams that do use a whole team approach to testing and quality are successful at improving their processes and their product. The State of DevOps report shows correlations between the use of modern testing approaches and high team performance. So why isn’t everyone trying to use what we’ve seen work well for 20 years now?

I have no actual evidence as to why this is, but I have some unscientific theories which I’d like to share. I’d love to hear your theories too.

Lots of Newbies

The number of new software professionals is growing fast. The Bureau of Labor Statistics predicted a 24% increase in software developers alone from 2016 to 2026. Despite having heard “testing is dead” for the past 15 or so years, I see more and more testing conferences with more and more people attending them, so I surmise our profession is also growing fast.

Universities are still generally poor at teaching modern development and testing approaches, so people come out of school without expertise in agile or DevOps values, principles, and practices. They certainly don’t learn much about testing in university. So, they have to somehow discover all this once they’re on the job. If they join a company whose software process is still stuck in the 90’s, doing poor waterfall at best, they’re unlikely to be exposed to modern testing.

Culture is Hard to Change

In my experience, it’s extremely difficult to change an established company culture, especially in a large organization. Even big enterprise companies who “go agile” often transition from role-based silos to having dozens of siloed Scrum teams that don’t talk to each other. All too often, an IT organization transitions to “agile” and either leaves the testers on their own test/QA team or sticks them in a cross-functional delivery team with no training or support to figure out how they should now work with people in other roles.

Large companies often have a complicated power structure. Upper managers may be more interested in protecting their domain than in delivering better software to their customers more frequently – and they don’t always prioritize a sustainable pace. If nobody educates them on how an investment in software quality – doing things like giving teams time to experiment and learn — pays off in the long run, they just keep imposing unrealistic deadlines while their software teams burn out.

Change is hard. Even when management is receptive, maybe not all team members are willing to try something new. It often takes only one naysayer to kill an effort to move away from “the way we’ve always done it”. Companies that are strapped for money may not be able to see how an investment in learning pays off.

I once worked in a company with a “hero culture” where the person who fixed the problem that brought down the website was lauded and rewarded – so why try to prevent production problems from happening? Even after leading a successful agile project to meet an “impossible” deadline, I couldn’t affect change in that culture.

Life is Challenging

If your management doesn’t support you learning how to improve the way you deliver your software, you have to do it on your own time. As you learn, you can be a change agent and try to help your team improve.

But that takes time, and we all have many demands on our time. Some people have to work two or more jobs to support their family. Some people must spend much of each day caring for a family member. Others have health issues that limit their activities. Perhaps they can’t afford to go to conferences. Perhaps they can’t take an evening off to go to a meetup or watch online video courses. There are so many reasons people aren’t able to learn on their own time.

So, how do we promote adoption of modern software delivery principles and practices?

I don’t have any easy answers, but I’d like to start more conversations about this. I think we can raise awareness that there are better ways to work, that it’s possible to make our customers happier while enjoying our work a lot more. Here are some ideas:

  • If you can, make time to educate yourself with the many resources available to us these days: online courses, webinars, blogs, books, articles, podcasts, local meetups, conferences.
  • Share what you learn with your teammates. Help them learn about different types of testing. Try small experiments together to improve.
  • When you meet other software professionals, for example in a social situation, encourage them to join you at your local tech meetup
  • Write about your own experiences, share them at meetups and conferences, to show others that improvements are doable and effective (if public speaking scares the pants off you, check out the SpeakEasy mentoring program)
  • Contribute to scholarship programs that help people attend conferences and access online content such as webinars and videos of conference talks

We have new people joining our profession all the time. What ideas do you have to help them embrace 2018 rather than 1988?

modern testing culture lisa crispin

About Lisa Crispin: Lisa is a tester who enjoys sharing her experiences and learning from others. She works as a Testing Advocate at mabl, is a co-author of Agile Testing: A Practical Guide for Testers and Agile Teams as well More Agile Testing: Learning Journeys for the Whole Team, she’s a frequent conference attendee and organizer, and she is an avid donkey lover. To find out more, find her on Twitter or LinkedIn.

The Top 5 Things We Learned in Our Webinar with Bas Dijkstra

By Leave a Comment

Recently, Bas Dijsktra of On Test Automation joined the CrossBrowserTesting team in the webinar “Lighten Up: Methods for Making Your Automation Suite Leaner ”.

Bas’s presentation was jam-packed with important lessons and actionable takeaways. To get the most out of it, you should take the time to watch the full webinar at the bottom of this post.

But we’re also rounding up a few of the most important lessons we learned from Bas. Here are our top five:

  1. There’s such thing as too much automation – With the introduction of tools like Selenium that allow us to automate at the UI level across different browsers, it can be tempting to automate anything we know how to write a test for. However, automation isn’t always the answer, and if you’re automating too much, it may be more of a problem than a solution when your automation suite becomes bloated. This can mean slow tests, shallow feedback, and high maintenance costs that reverse the benefits of automation. Additionally, when your suite is too heavy with automation, it no longer helps to shorten your feedback loop, and will likely add time instead. In order to get the most value out of your automation suite, Bas says that it’s important to understand where you should be testing at the UI level, when you should be testing at a lower level, and when to skip automation altogether.
  2. We have some unrealistic expectations around automation – Automation should never be the end goal of our testing. Bas advocates that we stop trying to automate everything and start focusing on automating the right things. Prioritize the business value rather than automating just to automate. Understand that not everything testers do can be automated, and not all test should be automated. Know the limitations of automation and your testing tool, and familiarize yourself with the tests that could be taken out of your suite to lighten your load to improve your feedback loop.
  3. Automation requires resources – One of the reasons why it may not be worth it to automate is return on investment, or ROI. Automation requires resources to write, maintain, run, and analyze tests. Bas says that this is too often forgotten when teams strive for “complete automation”. You have to decide whether it’s worth it to expend this energy and resources on automation, or if it would be best spent somewhere else. For example, maybe it would be more valuable to do an exploratory test of your application and do a deeper dive into areas you can’t assess with automation. Ask, “Are we comfortable not having the information provided by this test?” If you are, it might be a reason to redirect your resources away from automation.
  4. Not everything should be tested from the end user perspective – UI testing is great to test the end user perspective, but it’s not always the best method. You should automate as close to the implementation as possible and only do user interface-driven end-to-end automation when verifying end-user journeys. Ask, “where is the logic that we’re verifying actually implemented?” Often times you can run a test in the UI, but it won’t give you the feedback you’re looking for. In his example, Bas shows there are instances where testing at a lower level will give you more accurate test results.
  5. You might not need to run your automation suite as often as you think – UI tests can take a long time to run, and with multiple browsers and operating systems, it can take even longer. While tools like CrossBrowserTesting allow you access to a seemingly unlimited amount of browsers, that doesn’t mean you have to test in them all every time. Bas encourages you to decide which browsers you need to test and how often by looking at which your users are on and which are most problematic. If you think there’s value in testing in a browser or operating system but not adding it to your regression suite, CrossBrowserTesting lets you look at those same configurations in on-demand so you can test manually, as well. Bas also suggests moving tests out of the build pipeline and into an overnight job, so you’re not spending as much time running your suite.

Conclusion

As teams become better acquainted with automation, they have certain choices to make about what to automate, how to automate it, and why it should be automated. Unfortunately, sometimes we don’t ask these questions first and end up with a bloated test automation suite. Bas Dijkstra gives valuable advice for considering will actually set you up for success and give you more valuable test results.

To watch “Lighten Up: Methods for Making Your Automation Suite Leaner ” play the video below.

3 Easy Tutorials for Getting Started with Selenium

By Leave a Comment

selenium tutorial easy beginners webdriver with examplesselenium tutorial easy beginners webdriver with examples

You’ve learned your first programming language. You’ve familiarized yourself with Selenium. You’ve got a great Selenium testing tool and a great team to support you. You know what to automate and what not to. But where do you start with your first test?

No matter how prepared you are, getting started with Selenium can be intimidating. But by starting small and getting some of those repetitive tests out of the way, you can get your bearings while building a maintainable and reusable automation suite.

If you work in ecommerce, there are a few test cases you might come across regularly: sign-up, login, and sign out. The good news is that these user stories are fairly easy to automate, and once you understand how to set them up, it’ll be easier to start building your own tests in the future.

SignUp

The signup process will differ from site to site, but there are a few patterns that will tend to repeat themselves. For example, many will have multiple pages with “next” buttons, dropdown menus, and form fields to fill out.

Let’s go through what a test case would look like if you were automating a signup for Gmail.

First, we want to import WebDriver as well as the “Select” command we’ll be using:

from selenium import webdriver
from selenium.webdriver.support.select import Select
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait

Next, we need to instantiate an instance of ChromeDriver, the guy who will be driving our browser:

driver = webdriver.Chrome()

Now we want to define a method, so you can reuse this test, and navigate to Gmail.com where we’ll click “Create an Account” using the element id:

def site_signup():
driver.get(“https://gmail.com”)
driver.find_element_by_id(“Get Gmail”).click()

Clicking should take us to a new tab, then we’ll want wait for the page to load completely so that the form to create your Google Account appears:

WebDriverWait(driver, 10).until(EC.title_contains("Create Your Account"))

Here you find multiple form fields including First name, Last name, Username, Password, and Confirm password. This script will fill all these fields and click next:

driver.find_element_by_id (“firstName”).send_keys(“Mark”)
driver.find_element_by_id (“lastName”).send_keys(“Zuckerberg”)
driver.find_element_by_id (“username”).send_keys(“MZuckerberg”)
driver.find_element_by_id (“password”).send_keys(“Harvard2004”)
driver.find_element_by_id (“confirmpassword”).send_keys(“Harvard2004”)
driver.find_element_by_id (“Next”).click()

Again, this will take to to a new page where you’ll want to wait until it shows all the form fields for “Create Your Account”. It will then request you fill out more form fields. These will have drop down boxes:

def droppable()


select_fr = Select(driver.find_element_by_id("Month")) select_fr.select_by_index(May)
select_fr = Select(driver.find_element_by_id("Day")) select_fr.select_by_index(14)
select_fr = Select(driver.find_element_by_id("Year")) select_fr.select_by_index(1984)
select_fr = Select(driver.find_element_by_id("Gender")) select_fr.select_by_index(Male)

driver.find_element_by_id(“Next”).click()

From here, you agree to terms and conditions, which means you have to scroll down until you see the “I agree” button:

element = driver.find_element_by_id('I agree') element.location_once_scrolled_into_view

driver.find_element_by_id(“I agree”).click()

That’s it! If you follow these steps, it should create a Gmail account.

Log In

Once you get some of those trickier commands out of the way with sign up, these test cases should be easy. As we mentioned, you’ll see how you build on your previous learning to create new tests, such as with this login that uses find element by id, send keys, and click that we just used.

This is what a login with Facebook might look like (we hope that Mark doesn’t actually use the same password for Gmail and Facebook…and we also hope it’s more secure than Harvard2004):

def site_login():
driver.get (“facebook.com”)
driver.find_element_by_id(“ID”).send_keys(“MZuckerberg”)
driver.find_element_by_id (“ID”).send_keys(“Harvard2004”)
driver.find_element_by_id(“submit”).click()

LogOut

Logouts are usually pretty simple as well, but you will probably have to find settings or menu through a drop down. This is what the test would look like if you wanted to log out of Facebook:

def site_logout():
driver.get (“facebook.com”)
driver.find_element_by_id(“userNavigationLabel”).click()
driver.find_element_by_partial_link_text(“Log Out”).click()

You can see here that we switched up the way we located the element (by name instead of id) because of the nature of the dropdown window. Keep in mind there are multiple ways to locate an element if one doesn’t work out, including:

  • ID
  • Name
  • XPath
  • Link Text
  • Partial Link Text
  • Tag Name
  • Class Name
  • CSS Selector

Conclusion

Even if you don’t own a site that includes sign up, login, or log out, practicing tests similar to these will teach you some of the foundations of Selenium, popular commands, and good practices so you can continue to build and improve your Selenium test automation strategy.

To find more common commands that you may run into when building out your Selenium tests, check out our full Selenium toolkit, which includes more Selenium tutorials.

Balancing Your Automation Strategy with Manual Testing

By Leave a Comment

balance between manual and automationbalance between manual and automation

You’ve heard it a million times before — you can’t automate everything. But if your organization is just starting into the swing of automation, they’re probably thinking about all the ways they can automate. So how do you make sure you maintain the proper balance of manual testing and automation?

Whether you have an entire QA team whose roles are split between automated and manual testing, or there is one tester whose jobs is to do both, striking the right balance will be the difference between a testing strategy that provides value to your team and one that gives you more frustration than feedback.

By understanding how to best support your manual strategy with automation, you can ensure you’re getting the best of both world.

Know the Limitations of Automation

No matter how skilled you are in automation, how many Selenium commands you know, or what testing frameworks you use, some things simply cannot be automated.

For example, CAPTCHAs are often very popular with ecommerce sites. However, they’re inherently impossible to automate. This means that any test case that includes a CAPTCHA can’t be automated, and manual testing has to step in.

This, of course, isn’t the only case where you won’t be able to use automation. There are plenty of instances that it will be possible to automate, but it will be so complex to script, that you’re better off doing it manually.

Assess the limitations and challenges of test automation so you know where to step in with manual testing.

Solidify Your Regression Suite

The bottom line for automation is that you want to automate tests that are repetitive and time-consuming, so think about which tests you find yourself going through the most often. Solidifying your regression suite will help you determine which tests you want to cover after every integration or change to code.

Keep in mind that this will change over time as your application changes, but it’s helpful to have a suite that ensures basic functionality.

Additionally, deciding which environments you want to test on will be key to understanding which browsers, devices, and operating systems will be included in your regression suite.

Don’t Sweat the Small Stuff

Small test cases are best for automation and give you tests that are reusable and maintainable. The results of your automation are dependent on the state of your application, which means the smaller and more specific the test is, the more resistant it will be to changes in the UI. This is why unit and integration tests are ideal for automation.

But longer, more complex test cases and entire user journeys are often better left for manual testing. This gives you more room to explore the application and requires less time for scripting. Moreover, it means that if the UI of the application changes, you won’t have to go back into your script and figure out where it needs to be adjusted.

What’s the Risk?

When you’re thinking about whether or not to automate, risk should be a leading factor for consideration. However, while there are many methods for evaluating which tests are the most at risk, identifying these doesn’t mean you want to leave all other test cases that aren’t high risk out completely. In fact, these are perfect for manual testing.

When you considering risk, there are usually three tiers that you sort test cases into — high, medium, and low. As we discussed, most high-risk test cases will probably be included in your automation suite.

But, if you’re finding medium and low-risk cases that you still find valuable to test in order to provide feedback, then they might be something you want to run through manually.

Is It Providing Value?

Automation is a great way to speed up testing, but it also takes time for coding and maintenance, and that may not be worth it if you implement a test that isn’t adding value.

Say you run a test suite every day, and it passes 100 percent of the time. Is it worth it to keep automating if your results never change?

Just because a test was once automated doesn’t mean it should stay that way. Automation requires making changes, updates, and edits as the application code changes.

If a test is no longer providing value through automation, it might be better to take it back to manual testing to see if there are new parts to explore.

Communicate With Developers

Developers can’t tell you what to automate and what not to automate, but they can give you insight into the state of the application.

Keep open lines of communication with developers and other team members so that you are clear on what’s being changed in the application during development cycles. Depending on whether integrations are minor changes to the code, completely new features, or a revamped UI will make a difference in whether the test that follows will be part of your automation suite or should be tested manually.

To get additional insight, it may be helpful to pair with developers when going through the application to get a more in-depth look at what has changed and what areas or functions need to be tested.

Exploratory Testing

Exploratory testing is crucial to visually and functionally verifying new features and changes to the UI. Automation frees you up to test parts that you wouldn’t normally get to, so take it as a challenge to find a new bug or test for a different persona.

Human observation, curiosity, and creativity are paramount to being a tester. Automation can only take you so far when it comes to finding new bugs. That’s why manual testing, and especially exploratory testing, will always have a place in the QA process.

Visual Testing

Functional automated testing is not the only thing you have to worry about. Oftentimes, a functional test will pass because your application is working correctly, but it could still be visually inconsistent across browsers. Not only could this affect your layout and design, it could also affect the way your users are able to interact with your application.

In addition to functional testing with Selenium, you should incorporate automated screenshot testing into your workflow. This will help you account for visual cross-compatibility issues that you may have missed with functional testing by allowing you to compare screenshots of your web application side-by-side in different browsers.

Increasing Test Coverage

Automation will take care of repeated sequences, but in order to increase test coverage, you want to use manual testing to look at areas that have not yet been tested.

In this way, automation can be used as a tool to gradually help you increase test coverage over time. Test high-level functionality with automation, then fill in the gaps with manual testing. Look at edge cases and try negative testing to go beyond pass and fail results to gain deeper insight into your application.

This is also a good time to figure out what new tests you would like to add into your automation suite, so you can continue to increase coverage. Evaluate manual test cases to determine whether they would be valuable in regression, and assess which tests may be missing from the suite already.

Considering ROI

At the end of the day, it’s best to consider return on investment when deciding whether or not to automate. Automating takes time to analyze, code, and maintain.

If the time it takes to automate isn’t worth the results that you’d get from the test, it may be something you want to do manually, or not at all.

 

Webinar | Lighten Up: Methods for Making Your Automation Suite Leaner

By Leave a Comment

Bas Dijkstra webinar leaner test automation

Selenium has been the testers’ tool of choice for automated functional UI testing, but it’s too often misused. This results in inaccurate results, slow tests, and shallow feedback — and no one wants that.

If you’re struggling to write Selenium tests that meet your goals, it might be time to take a step back and get a better understand of the test cases that are best for Selenium as well as those that are better suited for testing at a lower level, or perhaps not automated at all.

Test Consultant Bas Dijkstra of On Test Automation joins CrossBrowserTesting to provide his expertise on how to de-bloat you UI tests and get your automation suite into shape. In this webinar, Bas will arm you with solutions to tighten up your end-to-end automation and streamline your Selenium strategy.

In this webinar you will learn:

  • What makes a good Selenium test, and what doesn’t
  • What is better tested at the API level, or not automated at all
  • How to future-proof your Selenium tests for better end-to-end testing
  • Live Q&A with Bas

Register Now

Bas Dijkstra

More about Bas Dijkstra: Bas Dijkstra a Test Automation and Service Virtualization Consultant at On Test Automation who’s always looking for more intelligent ways to use tools to improve test processes and software quality. He’s been in the field for over 10 years, designing and developing test automation and service virtualization solutions that enhance and improve test teams and test processes. You can read more about him on LinkedIn or Twitter.

Selenium Interview Questions You Should be Prepared to Answer

By Leave a Comment

Selenium interview questions

Software development teams are constantly shifting, which means testers’ roles have to evolve along with them. As we’ve seen more teams go towards Agile, DevOps, and CI/CD, testers are encouraged to concurrently take on skills that better support these workflows and help with faster testing.

One of those skills is Selenium. As more software testers take on test automation and begin to shift left, Selenium testing is one of the most popular strategies, and a comprehensive knowledge of the tool is foundational to creating tests that provide the team with productive feedback and insight. While Selenium is a tool that can benefit from continuous learning, there are certain things that interviewers will want to make sure you know when you’re applying for you’re next role in automation.

To better prepare for that upcoming interview, here are the top Selenium interview questions you can expect to be asked.

Explain the different Selenium components.

It’s important to be able to prove you know the full breadth of Selenium beyond writing a script. Selenium has an entire suite of tools that are used for a different capability. Review the history of Selenium to fully understand the differences between IDE, RC, WebDriver, and Grid.

Do you know the difference between Selenium 1 and 2? How about the programming languages and browsers it supports? Knowing the tool and rounding out your understanding of how it works will be something that employers value.

What’s your experience with Selenium?

Be able to talk about the different test cases that you use Selenium for. What worked well in your last role, and how do you identify a good case for automation? Is there a time that you automated something you shouldn’t have, and what did you learn from it? What automated tests have you cut out of your suite because they weren’t valuable? How many tests did you automate? Do you have experience using a tool to run Selenium in the cloud?

Giving an in-depth look into the breadth of your experience will help your interviewer understand your qualifications.

What are the advantages of Selenium? What are the most common Selenium challenges?

There are plenty of reasons that people gravitate towards Selenium. Have an answer for why people would choose it over other test automation tools and what it can help accomplish.

However, Selenium also has its challenges. Flakiness, dynamic content, reporting, mobile testing, and captcha are a few things that may come to mind when you think of these limitations. Know which ones have solutions and how to implement them, and know which mean testing is better of done manually.

What should Selenium be used for? What shouldn’t it be used for?

Going off of the last question, you should have a comprehensive knowledge of which tests should be automated and which are better off done manually.

This is not necessarily subject to Selenium as you should be able to give an overview of the right times to automate, how to decide by creating a risk analysis, and how to prioritize your automation.

When do you use Waits and Sleeps?

Knowing the proper use of explicit wait, implicit waits, and sleeps is key to reducing flakiness in your tests, which is undoubtedly something your future employer will care about — no one wants false positives and negatives.

Be able to talk about the difference between implicit wait and explicit wait and sleep, when to use each one, and how.

How do you handle common commands?

To verify your basic knowledge of Selenium commands, it’s expected an interviewer might ask you how to:

  • Launch a browser
  • Click
  • Scroll
  • Drag and drop
  • Send keys
  • Find an element
  • Close and quit
  • Handle pop-ups
  • Navigate back and forth
  • Switching tabs
  • Minimize and maximize windows

Check out our Selenium toolkit for tutorials on these commands and more.

What is Page Object Model and Page Factory?

To get more readable maintainable Selenium tests, you need to have a structure that likely involves design patterns including Page Object Model and Page Factory Object.

You should be able to speak to what they are, the differences between them, why they’re important, and how you use them.

What’s your process for reporting?

Because Selenium has limited reporting capabilities, testers’ method of reporting will differ, and your future employer will likely be interested in getting a better understanding of how you report your Selenium tests.

How would you automate this?

In order to prove your technical capabilities, you may be asked to write a test case. This not only helps the interviewer understand whether you’re able to accurately use Selenium, it also helps them understand your process.

Straight From the Manager’s Mouth

These previous questions are common in the test automation space, but to give you a more well-rounded look at what you could be asked, we also asked a few others might ask for Selenium interview questions:

“There are a few questions I ask our candidates. In most cases, they are a bit general, however it will give us an overall picture of what to expect from your future employee. I value the logical and intuitive thinking of a candidate, as I understand that in tandem with their knowledge, this will make them perfect team players and valuable members of our company.

These questions help me assume if a candidate follows the best practices in work with Selenium. It also helps me to understand how much expertise the candidate has and if he or she is ready for further learning and improving as a QA professional.

  1. In your opinion, why we should we choose this tool, and what are its advantages among others similar tools? What is the fundamental benefit of Selenium?
  2. What are the limitations of Selenium? For what purpose it is not the best choice?
  3. What type of tests have you automated?
  4. What are some differences between CSS selector and XPath based locators? Explain how one is better than another and why. What are the testing process particularities while working with CSS selectors and Selenium?
  5. What is a hybrid framework in Selenium?
  6. What is Page Object Model and Page Factory in Selenium? Provide an example in which case we can use them.”

–QArea Software Development Company

 

“1. How many tools are there in the Selenium tool suite? Tell me more about each tool.
2. We use WebDriver’s APIs to write our tests. What do you think about calling WebDriver’s APIs directly inside the test case?

These first two questions are easy questions to start the interview. There is no need to continue if they fail these.

3. What are the solutions to eliminate bad test design problems when using WebDriver? Compare and contrast your solutions.

I expect the interviewee to show deep knowledge on industry best practices on test designs such as Page Object or BDD.

4. What is your preferred control locating technique? And Why? Compare.

Interviewee should have good knowledge about capturing UI controls, and ability to determine what is the best locator for different situation, knowledge about different locator techniques.

5. Do you have any experience integrating your Selenium tests with any CI/CD tools? If Yes, Explain your experience.

This question helps to identify the superstar candidates. One element is that in order to successfully answer this, they will have to have had to have some real experience working with CI/CD tools to explain the answer in detail.”

-Thanh Viet Le, Automation Project Lead & Framework Architect, LogiGear

 

“1. What are some common exceptions you have encountered in Selenium and what are the steps you took to handle those exceptions?

This question is asked to understand the candidate’s knowledge and experience of Selenium exceptions and handling. This enables maintainable automation and we want to ensure these basics are understood by the software engineers.

2. What would the optimal Selenium automation framework look like?

This question is asked to understand the candidate’s in-depth knowledge of automation testing and the automation process, as well as gauge the candidate’s technical knowledge of how Selenium frameworks are architected.

3. Can you provide a few examples of major challenges you have experienced when automating test cases with Selenium and how you went about resolving them?

This question is asked to get an understanding of the types of complex test cases the candidate has automated with Selenium and the strategies the candidate used to resolve the testing challenges.

4. What are some major innovations or enhancements you have created in your Selenium automation framework?

This question is asked to identify the candidates out of the box thinking and assess the candidate’s self-driven qualities. It allows us to gain insight into how a Selenium automation engineer thinks in regards to ensuring the frameworks are maintainable, robust, and portable.

5. Can you describe some changes that have been made in Selenium 3.0 from Selenium 2 and the reason for those changes?

This question is asked to learn about candidate’s complete knowledge of the Selenium testing framework and the candidate’s enthusiasm/curiosity about product updates/new versions. It also gives us a clear idea if they stay up-to-date on enhancements and best practices.”

-Vivek Kumar, Head of Automation, QASource Automation Testing Services

What are we missing? Leave the Selenium interview questions you’ve asked or answered in the comments.

What Robotic Restaurants Can Teach Us About Test Automation

By Leave a Comment

robotic restaurants test automation

To most people, going out to eat doesn’t evoke the image of robots. But to the four founders of this restaurant, robot chefs are the only viable solution.

In this automated kitchen, robots begin cooking your meal in temperature controlled pans as soon as you submit your order. The bots a programmed to spin the ingredients and spices around until everything’s perfectly cooked. Then, it dumps your meal into a compostable bowl.

While the restaurant may seem overrun by restaurants when you look around, there are more humans involved than you think.

Once in the bowl, a person takes it and adds yogurts, sauces, and dry toppings, then hands you your finished product. Behind the scenes, the Culinary Director and Executive Chef are the masterminds behind the delicious recipes, making sure the finished meals have the right balance of flavor. Additionally, there are people prepping the ingredients. Of course, there are also the four engineers behind the robots making sure that everything runs smoothly.

So what does this have to do with software testing?

In the shift from manual to automated testing, we often get ahead of ourselves in trying to automate as much as possible, or even attempting to automate everything.

But just like a fast-casual restaurant cannot operate on robotics alone, neither can we rely on automation to fulfill 100 percent of our testing. There are certain qualities that testers possess that we can’t automate.

So as we continue to explore the ways that automation can aid in our testing efforts, here are a few takeaways we can gain from robotic restaurants:

  • A Good Recipe is Key – Just as we need a chef to create the recipes for the robotic sous chef to follow, there needs to be the first test for automation to follow. Whether this is done through record & replay or by writing out a script, the quality of the first test is crucial to designing a suite that will provide insightful results. Just like a robot can’t follow a recipe it isn’t yet programmed to make and can’t create one on its own, a test automation tool can only test what it is told. Additionally, every bowl is only as good as the recipe, just as every regression test is only as good as the first test or script.
  • Robots Don’t Have an Eye for Detail – Just like humans are the final step in garnishing your meal at a robotic restaurant, human testers have an eye for detail that is critical in testing. There are many soft skills that software testers have — empathy, creativity, curiosity, and knowledge of the software, for example — that are impossible to teach to an automation system. This is why it’s important to have real people that can perform exploratory tests on new features, layouts, or other noticeable application updates, so they can bring unique perception and understanding to provide valuable insight. Not to mention the customer service aspect — communication is important that there’s someone on the team advocating for testing as well as the end users and who’s able to talk to other stakeholders.
  • There is Risk Associated with Automation – Automation usually accounts for “normal” situations but doesn’t consider edge cases or stress. What happens if a customer has an issue with their bowl? What happens if a concert gets out, and the influx of customers is 10 times what it normally is? What happens if you run out of utensils? These are the instances where automation can no longer solve the problem, and a human needs to step in. Automation can’t go outside of what it’s told, which means there will always be risk involved when expected situations arise.
  • Automation is Best for Repeated Actions – A McKinsey report said the food industry is most susceptible to automation because there are so many predictable physical activities involved. Similarly, during code changes and new integrations, it’s important to repeat some of the same tests run previously to make sure they work. This is where automation becomes a good candidate for testing — regressions that have programmable repetition that helps to reduce human error. Rather than have someone perform the same actions over and over, having a machine take over the tester has more time to add new tests is what will result faster productivity in less time.
  • Cut Costs Without Cutting Jobs – Because of reduced labor costs, the robotic kitchen’s bowls start at $7.50 — around $3 less than comparable options. If you’re getting to the point where your solo tester doesn’t have time to run all the regression tests they need across browsers in addition to new exploratory tests, introducing an automation solution can alleviate some of those time-consuming tasks and give you a higher ROI. However, if you implement automation, there needs to be a human behind that as well. It’s also important to have someone who understands automation and the tools in order to properly execute and maintain tests.

The Biggest Takeaway

You can’t automate everything. Though you can add significant automation to a project, there will always be a place for human testers. Additionally, automation should be used as a tool to support manual work rather than replace it.

Interested in hearing more about how test automation is like a robot restaurant. Listen to Episode 6, Manual vs Automated…Restaurants? of SmartBear’s podcast series “The Good, the Bad, and the Buggy” with an interview with Paul Grizzaffi.

 

Join Over 200,000 Testers & Developers And Start Testing Today

Learn more Free Trial