Manual Testing Archives | CrossBrowserTesting.com

Discovering Exploratory Testing

August 22, 2017 By Alex McPeak Leave a Comment

Despite being fairly unstructured, scriptless, and independent of a pass/fail status, exploratory testing is a very important and strategic part of QA. In fact, its focus on discovery and learning in software is proven to be highly valuable when it comes to testing an application for the first time or looking into a new feature from the last integration.

By identifying problems, questions, ideas, and praise through this method of manual testing, exploratory testing helps start a conversation between internal teams and produce improved software.

What is Exploratory Testing?

Exploratory testing is a scriptless testing practice that emphasizes learning and discovery in software and aims to improve quality.

While test cases are not created in advance, exploratory testing still has a very strategic process and requires documentation. Test design and execution are done simultaneously in an ad-hoc manner, although exploratory testing and ad-hoc testing are not the same (we’ll get to that later).

Since exploratory testing requires a human to look at software, evaluate its functionality, and think of new ideas, it’s a manual practice rather than an automated one. It also depends on the skills and knowledge of an independent tester to be able to evaluate where software has issues or could use improvements.

Exploratory testing is often best for:

New websites
New feature
Unexplored areas
Different devices
UI testing

Afterwards, these insights are discussed with the rest of the testing and development team members to make changes, fix bugs, or add new features. Additionally, results can be used to raise questions about what’s working well and what’s not to implement in other areas of the software.

Session-Based Testing

Session-based testing is one way to provide more structure in exploratory testing. While you can’t go into it with a script or a definitive test plan, having certain elements in mind like objectives, heuristics and oracles, focus, risks, and follow-up will make sure the process is valuable.

Session-based testing will often include a technique called mind mapping to identify some of these elements in a visual layout. This also helps write better documentation to share with teams after testing is finished.

Establishing a Definition

James Bach and Michael Bolton are attributed as the testers responsible for calling attention to the value of exploratory testing.

However, Cem Kaner coined the term in 1983, defining it as “a style of software testing that emphasizes the personal freedom and responsibility of the individual tester to continually optimize the quality of his/her work by treating test-related learning, test design, test execution, and test result interpretation as mutually supportive activities that run in parallel throughout the project.”

Today, some people suggest that exploratory testing should just be called “testing” since all testing has an element of exploration to it, but we still think that it deserves a place as its own method according to the original definition.

Exploratory Testing vs. Other Testing

The parameters of exploratory testing are often confused or misunderstood. Comparing and contrasting exploratory testing to other types of testing can help clarify its purpose.

Scripted Testing – Scripted testing and exploratory testing are posed as opposites, which helps better understand their definitions. As mentioned, exploratory testing is scriptless and more random, whereas scripted testing follows a set and defined plan written by the tester. This also means that you can’t test something that isn’t planned ahead with scripted testing, whereas exploratory testing encourages testers to discover new regions as they go. This allows testers to act more as the end user and evaluate areas they come across rather than predefined spaces.

Ad-hoc Testing – Exploratory testing and ad-hoc testing are probably compared most often, but they’re two distinct methods of testing. For one, ad-hoc testing doesn’t require documentation, which is an essential part of exploratory testing. Similarly, it’s not well-integrated into team collaboration and has no set goals, objectives, or focus. Though exploratory testing is meant to be done in an ad-hoc style, it also has more set expectations than a being a random test. Additionally, many divert the two because exploratory testing could be said to require more skill, whereas anyone could perform an ad-hoc test.

Can You Automate Exploratory Testing?

Not really. Exploratory testing inherently depends on the individual tester to contribute their skills, knowledge, and insight to new or unevaluated components in an application — something that cannot be done with automation.

Thinking about this in terms of the testing vs checking debate, exploratory testing would be part of “testing” while automated testing is a method of “checking,” meaning that you could not automate it because you can only use automation to check — a tester has to test since a machine cannot provide independent thought.

Tips for Exploratory Testing

Have a loose plan – Going into a test with a focus is important in order to come out of the process with valuable feedback on certain objectives. At the same time, the nature of the exploratory test is to address certain issues as they arise and consider those as well. While testers should have a general plan, they should also allow room to improvise and consider unfamiliar elements.
Think like an end user – Exploratory testing is an ideal way to advocate as a customer that may be using a website. While developers are familiar with the code they build, it might be more difficult them to think outside the box and use software in a way it’s not optimized for. The job of the tester in this phase is to recognize these areas of weakness and bring them to the attention of the rest of the team.
Leverage your skills – While exploratory tests are designed and changed as the tester goes along, it doesn’t mean that just anyone can do it correctly. In fact, the results of the test will largely depend on a tester’s skill level. Leveraging a tester’s knowledge of software and previous testing practices will be the most efficient way to find and address problems.

The Unexpected Value of a Professional Developer Drunk Testing Your Web Design

May 31, 2017 By Alex McPeak Leave a Comment

It’s not often you stumble upon a service where you’re willing to pay a professional to work while intoxicated. In fact, getting drunk on the job is usually frowned upon. But when Richard Littauer founded The User is Drunk, he seemed to have found just the right niche where this was not only found acceptable, but people were willing to pay a little extra for it.

Operating under the tagline “Your website should be so simple, a drunk person could use it.” Littauer’s homegrown side-hustle took off as designers and businesses invested in a UX pro that would review their sites with honest commentary and a new perspective.

At CrossBrowserTesting, we believe web design should be optimized for every user, no matter what browser, device, or operating system. Littauer’s project suggests that it’s important to consider the universal user in another way — drunk or sober.

Though Littauer no longer regularly drunk reviews websites (for money, anyway), he still remains opinionated on what separates a good web design from a bad one. In order to answer some of our burning questions, Littauer gave us an exclusive on some of the things he learned and experienced while testing under the influence.

What’s your background in development and UI/UX?

I grew up with computers and had a half dozen dumpster-dived desktops in my basement. So, naturally, I did a university degree in Linguistics. After enjoying coding for my senior thesis and being the web developer for some societies in uni and for my own personal sites, I took a Masters in Computational Linguistics, with some UX courses which lead through a long, tangled route to me working as a front end developer for a couple of years in New York and San Francisco.

When I started the User Is Drunk, I mainly had front end experience, not UX, and was working as a freelancer on a project with the MIT Media Lab as a designer and developer. There’s no easy path that shows how I got from A to B, and I’m not entirely sure where B is, anyway.

So, how did you decide to start getting drunk and testing people’s websites for them?

Personal experience. Basically, I had come back from a nice night (I assume, I’ve since forgotten. I think it was a date.) and my friend asked me to jump on Google Hangouts and review his site for him. The feedback was brutally honest, and kind of useful because of that, and I remember saying “I should make a thing out of this.”

Drunk advice is different than normal advice; it’s not caged, it’s emotional bull**** but not normally buzzword bull****, and it is not good at error recognition or correction at all. It’s niche. And niche things are good.

I let the idea stew for a couple of months, and then, at a hackathon, decided to just run with it one day. The rest is history – it went viral, and there I was, reviewing people’s sites for money.

When and where have you primarily done this?

I did the majority of the reviews while traveling with Hacker Paradise through in Bali and Thailand. But I’ve done reviews all over the world – I’m a digital nomad, so it’s anyone’s guess where I am going to be at any given moment. I remember doing reviews in Tokyo, Berlin, Boston…

I don’t drink alone. That’s a quick way to hate yourself. So, normally, I invite some friends out for some drinks, and then slip away for twenty minutes, and come back and buy everyone a round. It seems to work pretty well. I’ve gotten better at planning to do little work the next day, too.

What do you do when you aren’t critiquing people’s websites?

Live my life. Recently, that means I run a company I founded – maintainer.io – which helps people maintain their open source software, and consults with businesses on their open source strategy. I am a community manager as well as a developer, so this came out of that, realizing that a lot of what maintainers do can be done as a service.

Sometimes I debate getting a lot of buzz words I’ve used to describe my work printed on hats, and hanging them all on my wall.

What’s your approach — do you wing it or are you usually looking at something specific?

I wing it. Sometimes clients have questions for me, but mostly I just go. It’s very hard to plan drunk. There have been many times where I’ve failed to turn on my microphone, for instance, so I am largely better off just clicking around and following my gut.

How do you think drinking allows you to look at websites differently? How do you think web designers benefit from getting a drunk user to look at their site from a UX perspective?

Basically, it means that I am honest, or at least that there is no filter. There’s what alcohol does; it inhibits your own ability to self-reflect and realize what you are saying and doing. This kind of reaction can’t be faked, and it’s one of the reasons I’ve never pretended to be drunk (the other reason being because that just wouldn’t be right).

The advice or complaints you give while drunk haven’t gone through some review board, haven’t been cleaned up – it’s what me, a real person, actually thinks about your site at a given time. That’s invaluable.

Couldn’t just anyone get drunk and click through their own website?

Sure. But they don’t have the experience I do, and they may not be used to developing and designing websites. Also, at this point, I’m the most experienced drunk UX auditor in the world. I think that counts for something. But if you’re up for it, go have a good time. Drink responsibly.

What are the top 3 worst things websites tend to do in the eyes of an inebriated user?

Dark UX patterns are very, very clear when you’re drunk. A lot of websites are jerks – no, I don’t want to sign up for your newsletter. No, I don’t want this pop-up to happen every time.
Another major thing would be too much text. Reading walls of texts isn’t really my thing, and probably isn’t other people’s either. On the other hand, Pam, the mom from The User Is My Mom, loves reading walls of texts. She has a longer attention span. Age matters.
The final worst thing would be red and green colors everywhere. I’m a bit colorblind; not horribly, but if you point to a cardinal in a Christmas tree, it’ll take me a while to find it. A lot of sites don’t cater to people with this disability, although it is fairly common. That gets annoying, fast.

Have you ever drunk tested in different browsers or devices? Do you think there would be any value in testing responsive design under the influence?

I’ve done a few apps, and a few private sessions. There is some value, there. Frankly, though, I’m not an app developer, and I’ve always felt that my advice on websites is more pertinent. There is definitely value in responsive design for drunk people. You have no idea where your users will be.

How did people respond to the results? Do a lot of people change their web design after your reviews?

It’s all across the board. I’ve gotten a few people who were full of praise, and changed a lot of things, working my tips into their sites. I’ve had people change their onboarding process after, especially.

I’ve also had a fair amount of people who never got back to me. Those are not fun. Drunk testing is very hard to standardize – sometimes I’m nice, sometimes I’m not. That’s how it goes.

What’s your drink of choice?

Uisge beatha. Talisker, 12 year, or a Laphroaig, quarter-cask. But normally it’s just a local beer. I’ve gone off stouts, wheat beers, and red wine with cigars as a result of some bad nights. That’s OK with me.

What advice do you have for designers who want to cater to the user experience, whether those users are drunk or sober?

Get on Twitter and Designer News. This is the golden age of learning – never stop.

And please don’t use red and green colors too much.

Any last thoughts you’d like to add?

While the User is Drunk is fun and stupid, it’s also fun and stupid. This is not my main job, at all. Currently, I’m working on consulting for companies wishing to do better open source, and helping individual coders and teams who have too much code to maintain – check out my business at maintainer.io.

I say this because I think a lot of people are under the illusion I only do the User Is Drunk, which couldn’t be further from the truth. I limit my reviews to around once a month now. Livers can’t be fixed by resetting the router. Drink responsibly, folks.

Choosing Between Simulators, Emulators, or Real Devices for Testing

March 20, 2017 By Alex McPeak Leave a Comment

Should you use simulators, emulators, or real devices for testing?

One of the initial choices you have to make when you begin to build a browser testing strategy is what environment those tests are going to be run against. When it comes to choosing the mobile devices used for testing, you have three options — simulators, emulators, or real devices.

While developers evaluate the pros and cons of these choices, the answer to a classic testing dilemma may be more straightforward than you think.

Emulators and Simulators

Emulators and simulators are both virtual devices that are used for the same basic purpose — they mimic the functionality of real devices and imitate a user’s actions to recreate operational behaviors.

The difference, respectively, is that one emulates (replicates or reproduces) real mobile device software, hardware, and the OS in order to test and debug applications within another software/hardware platform, while the other simulates (imitates or mimics) the internal behavior of a device, but does not emulate hardware or work on the OS.

The best way to exemplify this is thinking about a flight simulation. By using a simulator, you are mimicking the operation of the plane’s mechanics, journey, and surrounding environment through a computer-generated program that shows you what the procedure might look like. If there were such thing as a flight emulator, on the other hand, the device could theoretically take you to a destination the way a plane could.

In this instance, if you are deciding between emulation and simulation, emulation is your best bet as it’s more accurate and more suitable for debugging. However, emulators and simulators are often used and referenced interchangeably since they perform similar functions.

Additionally, when you think about the flight simulator example, you can see that both instances have degrees of variation from boarding a real plane, just as using simulators and emulators varies from using a real device. In real life, there are many factors that can affect the results that would not be considered in a simulation or emulation.

We’ve found that the use of emulators and simulators is most appropriate in the earlier stages of testing, where you may not require 100 percent accuracy but rather want a fast test that gives you a general idea of how a web application is performing.

As you continue in your testing process, you will likely find that real devices give you more precise application and allow you to observe more particular aspects of a web application.

Real Devices

The advantage of real devices is in the name. Testing on real, physical devices means that you’re testing the same way that your users are operating their devices.

While simulators and emulators will come close to imitation, it’s impossible to perform exactly the same as a real device. This is because emulators can’t account for every single environmental factor, feature or user action performed on a real device. For example, emulators cannot simulate high traffic volume in the way that a real device will experience, which would possibly affect the results of performance or functional testing.

They also are known to give false negative/and positive testing results, which can be problematic in an advanced testing process and can negatively impact developers’ progress, ROI and bottom line goals. For this reason, testing on real devices more accurate, concise and user-specific, and they become essential for performance, reliability, operability, sanity, and regression testing.

Testing on real devices comes at a higher cost, however. Oftentimes, testers may resort to emulators and simulators because they’re a less expensive option compared to building a device lab for all the platforms you require. However, hosting real devices in a third-party cloud is a simple answer to cutting costs while obtaining access to a broad range of devices.

This will also allow you to perform faster testing, since you can run them in parallel. In comparison, owning and operating a device lab means that you can only run one test at a time as you go through different phones, desktops, and operating systems.

At the end of the day, real devices are necessary because testing on them will show a larger range of issues that your users are experiencing.

In Conclusion…

There’s a place and a time for emulators and simulators as well as real devices. Deciding between the two is a choice that is highly dependent on where you are in your development process. Additionally, because both options have their pros and cons, many developers will decide to us both practices for testing.

These are the main points to consider when choosing between emulators/simulators and real devices:

Most developers agree that some combination of simulation/emulation and real devices is best for testing.
Emulators are a good option in the development process because they’re a faster option, but often are less accurate due to lack of human observation that is critical in testing
Real device testing can be done through a physical device lab, VMs or a third-party cloud
Hosting real device testing in a secure cloud is often less expensive, while it allows you to test more devices at the same time

Getting Started with Manual Testing

March 17, 2017 By Alex McPeak Leave a Comment

Manual testing is a must

What exactly is Manual Testing?

When any tester, developer or QA team begins a functional or performance testing process for the first time, they’ll likely start with manual testing. Regardless of the size of the company or web application, manual testing is an essential component of any testing strategy because it gives deep insight into the user experience.

This is because manual testing is done completely by human hand, without the assistance of automation tools or scripts of any kind. For this reason, manual testing is the ideal way to assess the details that are relevant to the end-user including how the application functions, how well visual elements render, and other GUI qualities.

Manual testing is specifically valuable in exploratory testing or other test cases that are only run once or twice and ensures that you discover bugs and other issues early on in the development cycle. These tests are an optimal environment for incorporating human observation without an urgent need for the speed and in-depth accuracy that’s characteristic of automated testing.

Manual testing also allows testers to build test cases, which are needed for automation so that you can repeat those actions. No matter how much time you’re willing to put into manual testing, it’s the first step to creating a script that can be run in parallel in other browsers, or many times over.

For this reason, it’s crucial to understand some of the stages and techniques before executing manual tests.

Types of Software Tests

These stages of testing are sequential, starting with the smallest component of the application and progressing to make sure that all parts are working together.

Unit

Unit testing is at the bottom of the testing pyramid because, as the smallest testable part of an application, it focuses on one individual function within the entire application. Although this type of test is better performed in an automated manner – it’s still the base of our pyramid and most plentiful type of tests out there.

Clicking a button to make sure it takes you to the correct destination would be a unit test. For example, making sure that the “Next page” button will actually take to you the next page.

Integration

Integration testing is the next step after unit testing and is the process of integrating multiple units to test them together. This lets testers observes how different components of the software harmonize to ensure functionality.

An example of an integrated test could be your applications log-in process. Now, you’re testing whether entering a username and password and clicking login will bring you to the Welcome screen. These types of tests match two or more unit tests together.

It is important to have an inverse amount of tests as you go down the testing pyramid. This allows for easy debugging and error locating, speeding up both testing and development. Have unit tests along with integration and other test lets you diagnose the exact location of bugs.

System

A system test refers to testing completely integrated software in order to evaluate that system meets the requirements of the other hardware or software systems that it’ll come in contact with.

This is actually a series of multiple tests including end-to-end scenario testing, verifying inputs and output and testing the user experience. There are also multiple types of system testing such as stress, regression and functional testing, the uses of which will differ from tester to tester depending on available time and resources.

Again, all the components that would be tested in a system test have already passed integration testing. System testing after is necessary for testing the design and behavior of the software.

Acceptance

Acceptance testing or User Acceptance testing is the beta testing of the software performed by actual end users. This, of course, is to bring the product into a real life environment to see how well it measures up to a user’s standards in case the tester misses an essential user function, specification or request.

As you can see, manual testing requires a thorough and sequential completion of these stages in order to properly measure a software’s functionality at different points in the testing process in order to target or fix various areas.

Black Box vs White Box Testing

Black box testing and white box testing are two techniques of testing that can be applied to the above stages.

Black box testing is where the internal structure of the software being tested isn’t known to the test, while the internal structure is known within white box testing.

Black box testing is the most common for testers because they don’t have to be expert in coding, making it easier and simpler to execute. It’s also quicker to do because testers only have to look at the GUIs that users are interacting with and looking at.

However, black box testing has restricted coverage in this way and can only evaluate a percentage of every possible test case or scenario, rather it just considers inputs and outputs.

In comparison, white box testing is a little more in-depth and looks at the programmatic structure of the system. In this way, it allows the developer or tester to spot errors and bugs quicker and more accurately and provides a more thorough insight.

Keeping Manual in Your Testing Strategy

While the testing environment is largely moving to automated testing as today’s developers focus on a strategy that embodies agile CI, there will always be a very specific and important place for manual testing. Those hoping to move towards 100 percent automation will find that their results will fall short from complete and efficient web development.

Instead, getting to know your code through the methods and techniques of manual testing will prove to be invaluable when it comes to present and future web testing practices.

Launching A Instant Test With The Bookmarklet

February 2, 2017 By Daniel Giordano Leave a Comment

Our Instant Test Bookmarklet allows you to quickly jump into a Live Testing session or take screenshots of the web page you are currently on at the time.

A bookmarklet is like a bookmark, but instead of loading a specific page, it injects JavaScript into the current page in your browser for added functionality. Our bookmarklet can be found at the Test Center, and quickly installed by being dragged into your bookmark bar.

Starting The Bookmarklet

To begin using the bookmarklet, simply click the bookmark while on the page you’d like to test. You’ll be presented with the following app on your web page:

From here you can jump right into a Live Testing session by clicking one of the browsers presented as a “Quick Launch Browser”.

Adding A Browser

To add a new browser, simply click Add and select the operating system, browser, and resolution. When you click Add, we’ll include that as a “Quick Launch Browser” as long as you are logged in.

Removing A Browser

To remove a browser, click Edit, and the X in the upper right-hand corner. In this view you can also change the resolutions of each configuration.

Running Screenshots

To run Automated Screenshots, select Shots on the left side navigation menu, right below Live. You can choose from popular or saved lists of browsers and quickly launch into our app’s UI.

Installing the CBT Bookmarklet

Known Issues:

Bookmarklet layout issues: The bookmarklet injects javascript into the current page you are viewing. CSS styles on these pages can impact the appearance of the bookmarklet. If you notice a site that it is not correct on, contact us and we will try to correct it.