CrossBrowserTesting.com

A Design, Development, Testing Blog

The Junction of Browser Testing and Test Management

By Leave a Comment

Software development – and subsequently software testing – is happening at speeds which we’ve never seen before. With many tests being set up and deployed automatically in order to keep up with pace and maintain quality and diligence throughout the testing process, keeping track of those tests has become even more paramount.  

This is the junction of browser (and app) testing and test management. Test management is crucial to ensure collaboration and traceability throughout the testing lifecycle, especially given the increase in volume and complexity of these automated tests.  

Unifying test automation and test management capabilities can be done using SmartBear’s industry leading test automation software CrossBrowserTesting, and Zephyr for Jira, the most widely used test management tool to work natively inside Jira.  

The CrossBrowserTesting and Zephyr for Jira integration allows teams to easily manage all of their manual and automated tests. Users and testers can quickly navigate to the respective Jira card to view results of each test, analyze pictures and video recordings of the test, and add any notes that may have come up — all within that native Jira experience.  

By including test management in your Agile process, you’ll have much more control and visibility into your team’s efforts. During Sprint and Release meetings, you can make sure tests are properly executed, while giving your testers the ability to quickly change or re-prioritize tests accordingly.  

Want to learn more about where browser testing fits in the testing pyramid and how to include test management in your Agile process, both with the power and support of award-winning SmartBear tools? Check out our On-demand session to learn more about both tools and how they work together!

Next Steps for Teams Who Want to Move to Agile Project Management

By Leave a Comment

agile project management development

It seems that the companies we all aspire to be more like are moving quicker and releasing faster than ever before. So, how can the rest of us strive to follow in the footsteps of software giants that release innovative software on the hour? While there may be no single, clearly-defined answer, we do know that many are seeing success with agile project management and development methods.

By focusing on feedback from stakeholders throughout the development process, not only is agile found to be a faster development method, it’s also more effective in the way that it allows and encourages changing requirements.

However, as waterfall is the traditional methodology for software development, teams that are stuck in their old ways might find the transition to agile hard to wrap their head around, so here are 7 tips for teams who want to take the next steps and move to agile project management.

  1. Identify what or who is missing to make agile work. In order to move to agile, you first have to recognize that it will be process your entire team will have to get on board with and fully embrace in order to make it work. Identify the roles and people you need on your team to make a successful transition. It might be helpful to onboard a project manager, scrum master, or agile coach to help adjust the current workflow. Look at your current team, as well, to understand where there are silos and barriers to communication. Teams that don’t communicate will find it hard to be successful. Agile project management works best when this is cross-departmental collaboration, so it’s important to work from the inside out.
  2. Prepare to shift left. Agile will require more testing earlier in the software development lifecycle. Because each phase of the SDLC is done in short iterations, rather than long incremental ones like with waterfall, testing shouldn’t wait until the end when new features are fully developed. With a focus on continuous integration and continuous delivery that comes with agile development, continuous testing will be the key to fast feedback and iterative changes that are needed to release high-quality software. Evaluate whether you will need to introduce test automation, additional QA team members, new tools, or something else to help you reach maximum test coverage.
  3. Poke holes in what you’re doing now. Look at your current planning process, and outline how your strategy will have to change in agile development to allow for more feedback and flexibility. Where are the bottlenecks in your current process, and what prevents your team from delivering high-quality software or meeting short deadlines? This will be crucial to shifting to a new way of thinking and transforming your team’s workflow. Familiarize yourself with the different agile methodologies such as Scrum, Kanban, and Extreme Programming. While these all follow agile principles, they achieve it in different ways. Based on the makeup of your team and the objectives you discussed, it will affect the way you carry out agile project management.
  4. Get on board with sprints and standup. No matter which methodology you choose, a few things will probably be the same, and they’ll help you properly implement agile project management in your organization if you take the time to do them correctly. Sprints will be the duration of you planning — usually they’re two weeks long but can be anywhere from one to four — and standups are where you talk about what you’ll be doing in each sprint. Take advantage of these opportunities for strategic planning and collaboration with the entire team.
  5. Backlog issues and tasks. Identify the projects that currently challenge your team and what you’d like prioritize when you transition to agile. Create a backlog of tasks, gaps, objectives that need to be addressed so you can start ranking them on most to least important and having conversations about future goals. Discuss what is needed from each team member, how it will be accomplished, and what information is currently needed to make it as “done”. Once you begin to organize your team and the work that needs to be done, you can start building out detailed tasks and sprints.
  6. Don’t skip your retrospective. As you adjust to agile and the many changes that come along, you’ll be constantly learning about what’s working well and what isn’t, as well as what individuals are successful with and what they’re struggling with. Ensure that everyone has an equal say in each sprint, so it’s not just leaders and managers giving their two centrs. Focus on how you can improve in each sprint. However, don’t just wait until retrospectives to collect feedback from your team — encourage communication and checkins throguhout the sprint.
  7. Be open to change. Requirements change, tasks get de-prioritized, clients and customers change their mind, and feedback will alter the sprint process. The nice thing about agile project management is that it encourages these adjustments. Embrace agile and understand that these frequent changes are improvements that will help your team achieve continuous integration, testing, and delivery. If you feel like you’re getting lost or confused, always resort back to the agile manifesto to realign your team with the principles behind agile.

Finding your groove with agile won’t happen overnight, but there are certain steps your team can take to make the transition more seamless. For more, take a hint from what agile teams can learn from Netflix.

 

The Spectrum of Testing Practice: Why Are Some Companies Stuck in the 90’s?

By Leave a Comment

modern testing culture lisa crispin

Those of us who are passionate about delivering valuable, high-quality software to our customers frequently and at a sustainable pace are living in exciting times.

Many are embracing “modern testing” principles. We’re acquiring new skills such as how to help non-testing teammates learn to test, how to analyze production use data, and how to use that data for testing. Testing is at the heart of DevOps culture, providing new opportunities for testers. We have amazing tools to help us with activities such as regression test automation and learning from production monitoring, logging, and observing.

At the same time, I still encounter many companies who are doing testing the way most people approached it 20 or more years ago. They have siloed testing/QA teams who don’t collaborate with development teams, operation teams, or even product and design teams. They have no automated regression tests or are struggling mightily to get traction on that automation. They’re doing only manual regression testing, working from written scripts, and no exploratory testing.

Why??!!!

I’ve seen leading agile practitioners scratch their heads and wonder – after two decades of sharing agile values, principles, and practices with the world, why are so many people not using them?

Teams that do use a whole team approach to testing and quality are successful at improving their processes and their product. The State of DevOps report shows correlations between the use of modern testing approaches and high team performance. So why isn’t everyone trying to use what we’ve seen work well for 20 years now?

I have no actual evidence as to why this is, but I have some unscientific theories which I’d like to share. I’d love to hear your theories too.

Lots of Newbies

The number of new software professionals is growing fast. The Bureau of Labor Statistics predicted a 24% increase in software developers alone from 2016 to 2026. Despite having heard “testing is dead” for the past 15 or so years, I see more and more testing conferences with more and more people attending them, so I surmise our profession is also growing fast.

Universities are still generally poor at teaching modern development and testing approaches, so people come out of school without expertise in agile or DevOps values, principles, and practices. They certainly don’t learn much about testing in university. So, they have to somehow discover all this once they’re on the job. If they join a company whose software process is still stuck in the 90’s, doing poor waterfall at best, they’re unlikely to be exposed to modern testing.

Culture is Hard to Change

In my experience, it’s extremely difficult to change an established company culture, especially in a large organization. Even big enterprise companies who “go agile” often transition from role-based silos to having dozens of siloed Scrum teams that don’t talk to each other. All too often, an IT organization transitions to “agile” and either leaves the testers on their own test/QA team or sticks them in a cross-functional delivery team with no training or support to figure out how they should now work with people in other roles.

Large companies often have a complicated power structure. Upper managers may be more interested in protecting their domain than in delivering better software to their customers more frequently – and they don’t always prioritize a sustainable pace. If nobody educates them on how an investment in software quality – doing things like giving teams time to experiment and learn — pays off in the long run, they just keep imposing unrealistic deadlines while their software teams burn out.

Change is hard. Even when management is receptive, maybe not all team members are willing to try something new. It often takes only one naysayer to kill an effort to move away from “the way we’ve always done it”. Companies that are strapped for money may not be able to see how an investment in learning pays off.

I once worked in a company with a “hero culture” where the person who fixed the problem that brought down the website was lauded and rewarded – so why try to prevent production problems from happening? Even after leading a successful agile project to meet an “impossible” deadline, I couldn’t affect change in that culture.

Life is Challenging

If your management doesn’t support you learning how to improve the way you deliver your software, you have to do it on your own time. As you learn, you can be a change agent and try to help your team improve.

But that takes time, and we all have many demands on our time. Some people have to work two or more jobs to support their family. Some people must spend much of each day caring for a family member. Others have health issues that limit their activities. Perhaps they can’t afford to go to conferences. Perhaps they can’t take an evening off to go to a meetup or watch online video courses. There are so many reasons people aren’t able to learn on their own time.

So, how do we promote adoption of modern software delivery principles and practices?

I don’t have any easy answers, but I’d like to start more conversations about this. I think we can raise awareness that there are better ways to work, that it’s possible to make our customers happier while enjoying our work a lot more. Here are some ideas:

  • If you can, make time to educate yourself with the many resources available to us these days: online courses, webinars, blogs, books, articles, podcasts, local meetups, conferences.
  • Share what you learn with your teammates. Help them learn about different types of testing. Try small experiments together to improve.
  • When you meet other software professionals, for example in a social situation, encourage them to join you at your local tech meetup
  • Write about your own experiences, share them at meetups and conferences, to show others that improvements are doable and effective (if public speaking scares the pants off you, check out the SpeakEasy mentoring program)
  • Contribute to scholarship programs that help people attend conferences and access online content such as webinars and videos of conference talks

We have new people joining our profession all the time. What ideas do you have to help them embrace 2018 rather than 1988?

modern testing culture lisa crispin

About Lisa Crispin: Lisa is a tester who enjoys sharing her experiences and learning from others. She works as a Testing Advocate at mabl, is a co-author of Agile Testing: A Practical Guide for Testers and Agile Teams as well More Agile Testing: Learning Journeys for the Whole Team, she’s a frequent conference attendee and organizer, and she is an avid donkey lover. To find out more, find her on Twitter or LinkedIn.

Challenges of Being the Team’s Sole Tester

By Leave a Comment

lone software tester agile

PART ONE

Introduction

Until relatively recently, the chances are that if you were a tester on a project, you’d be one of a number of such people. You’d have other members of the team to try ideas out, share the workload, cover for you when you’re away.

With the recent drive towards agile, we’re seeing the makeup of the team change dramatically. Projects can typically be supported by much smaller teams working permanently to evolve the product. This can often result in there only being a single tester on a team.

What are the challenges being the sole tester on such a project? How can you work with these constraints? This has been the subject of a series of workshops with fellow testers within my company, which I’m excited to share the outcome with you …

The Iron Triangle

Before we get underway, it’s useful to revisit the following principle within project management which we found underpinned many of our conversations. It’s useful for thinking about the constraints we’re working within on a project, especially in agile.

Image result for scope cost schedule quality

In the iron triangle gives us the idea that the “quality” of a project is defined by three attributes of your project – cost, scope, schedule (or time).

You might have heard the adage “cost, scope, schedule … pick two”. However, ideally on a project, there should be only one cast iron attribute – what management consultant Johanna Rothman calls “your project driver” in her book Manage It.

Within any project you can only really have only one attribute which is fixed – it could be “we need this done by X” (schedule) or “there is only Y budget for this” (cost). The skill of a manager is to work with this constraint and plan around what can be done with the other two attributes to achieve this goal.

Within traditional test management, there are a lot of parallels for applying this same theory to test planning. Within this dynamic the fields are,

  • Scope – how much testing you’d like to achieve
  • Cost (or rather typically resources) – having more testers allow you to execute more activity
  • Schedule or timeframe – how long you have to do things

It should be obvious that if you have a large scope, and a short timeframe, one solution would be to have more testers on it. Although of course in the real world, there are constraints as to how much this can be pushed, and good test management revolves around knowing and pragmatically working within these constraints.

Another solution, of course, is less testers, but it means that it takes longer to get through everything you’d like. Great for the test budget, but typically people like developers need to be paid to be on call to fix the bugs and the bugs are found later in the cycle, so developers need to be available longer.

Finally, if you find yourself in a situation where your available people and schedule are fixed, the only thing to do it to prioritise your scope as it’s the only thing you have control of.

Understanding this dynamic and the trade-offs is important because it was a core part of the discussions that were held, together with ways they could be handled and occasionally hacked.

Under pressure

A common initial feeling of someone stepping into the role of a sole tester was that of feeling under pressure.

Especially in an agile project, the timeframe is set by the sprint duration and your testing team size (although this can be “hacked” as we’ll discuss later).

Just back in 2013, one of our projects would have an annual release, which would involve a two-month testing window and would keep our test team of six busy.

Fast forward to 2018, and we’re now working in agile teams where we are creating deliverable code in a two-week sprint window using only two testers.

A key enabler in this was adopting a robust automated testing framework, which was easy to maintain with changes in the system under test. Such a suite did not grow overnight – and required a lot of work between testers and developers to build the right thing from a framework perspective, as well as to work through a prioritised list of useful automated tests to have in place. In working out idea scenarios and prioritisation, testers found themselves well-placed to lead these discussions. Over time, this suite was able to carry the functional regression load.

Automated testing helped, however, it didn’t eliminate the testing role. But testers found that their role did change dramatically. Most manual testing effort now focused on testing new or changed functionality in-depth during a sprint, as well as helping out with increasing ownership on test scenario selection for the automated suite (as well as shock-horror, learning to code their own tests).

In teams which are still undergoing a level of “forming” – a term used to describe those that have relatively new team members, some of whom were relatively new to working in an agile team – it was quite common for the sole tester to feel initially like they were the “point of blame”. If something gets out into production, the inevitable uncomfortable question can be asked of “why didn’t you test that?”

We shared a few of our experiences looking for general themes. Part of the problem that we were acutely aware of was time, and it’s not always possible to test everything you want to.

In many examples of a release where a defect had been undetected, manual testing had always occurred. Typically though, something was missed, or it was not imagined that a particular scenario could have been capable of causing an issue.

It’s worth taking a moment to think about how this was addressed in “classic” waterfall projects. A test lead would create a plan of what’s to be covered in consultation with many people on the project, but especially using the requirements. From this, they would build up a series of scenarios to be covered and make estimations around the resources and timescale.

However, on these classic projects, this was not the end of the story. It was the tester’s job to produce the best schedule they could, but it was known that this would not be perfect on the first draft. This was why such emphasis was put on the importance of reviewing – firstly by peer testers to see if enough testing heuristic variation has been employed, but also by a wider team such as project managers, customers, developers.

The aim with reviews was to find gaps in the plan and address them. This allowed the final testing scheme to be the most robust scheme of testing possible. This could come from developers saying, “we’re also making changes in this area” or our customers stating there’s an expectation that “most people will…”.

Within agile, it can be easy to forget that this level of contribution is still required. It needs to occur, however, it’s in a more informal, often verbal manner.
Within my colleagues, there is a general consensus that the tester becomes more responsible for facilitating a discussion around testing, much closer to what some organisations will call “a quality coach”.

A core tool for having these conversations is the use of mind maps, which the group has been using with success since 2013. A mind map allows the author to show for a particular feature, all the different variations, and the factors that they’re planning to follow in a one-page diagram.

When done well, they’re intuitive to read and can even be posted in common areas for people to look at. Their brevity helps get people to read them — “I haven’t had time to read that thirty-page document you’ve sent yet” is a frequent complaint in IT.

Even with a mind map in place, there is a natural tendency for the rest of the team to rubber stamp things. A sample conversation might go like this:

Tester: Did you have anything to add to the test mind map I sent out?

Team member: Uh … I guess it’s okay?

We all have a tendency to say something along the lines of “I guess so” for something we’ve not properly read. It’s important to still follow up with a brief conversation about what’s in your coverage – this can be individually with each team member, but often it’s better with the whole team. Just after stand-up can be a great time for this to occur.

If a member of the team notices there’s a mistake about the approach or some items that are missing, it’s expected for them to provide that feedback. Likewise, if the developer does more change than initially anticipated, there’s an expectation for them to tell the tester what they might also want to consider.

Often what you’ll read in agile literature about a “whole team approach” is essentially this: the whole team takes responsibility to give feedback whether it’s about how a story is defined, how a feature is being developed, or how testing is being planned.

A good indicator of when a team has made this mind shift is the use in retrospective of “we” instead of “you” – “WE missed this, WE need to fix this”. Teams where this happens have a much more positive dynamic. It’s important that this applies not just to testing.

Other examples include when a developer builds exactly what was on the story card, but not what was actually wanted (“we failed to elaborate”), when a story turns out much bigger than first thought (“we failed to estimate”) etc.

That said though, agile does not mean the breakdown of individual responsibility. A clear part of the tester’s role is to set clear expectations for the team of what they can do, how much effort it will take, and how you’re approaching it. But there needs to be team input to fine tune this to deliver the best value.

Mainly testing will revolve around changes to a product, for which the rest of your team are your first “go-tos” as fellow subject matter experts on the item. Occasionally as a tester though, you will find the value to consult with another peer tester – and there is an expectation that testers who are part of the same organisation but in other teams can be approached to be asked for their advice and thoughts on a test approach. Within our company, there is an expectation that all testers make some time in their schedule to support each other in this way. This, in many ways, echoes the “chapter” part of the Spotify model, with testing being it’s its own chapter of specialists spread across multiple teams/squads who provide test discipline expertise.

Reaching out to other testers like this is important; it creates a sense of community and the opportunity to knowledge share across your organisation.

Waterfall into agile won’t go…

There have been some “agile-hybrid” projects where there has been an expectation of set numbers of people being able to perform a set volume of testing in a set time (sprint). This can sometimes be problematic as the tester involved in execution hasn’t been involved in setting the expectation of what volume of tests are likely. And hence, it can feel like working against an arbitrary measure not based in reality.

In such a situation, it’s like being given an iron triangle where someone has given you “here’s your schedule, here’s your resources … so you need to fit in this much scope”. When faced with so many tests to run, it obviously helps to have them prioritised so that you’re always running the most important test next. When three areas are fixed, what suffers is the quality – it gets squished.

On projects where test scripting was not mandated by contract, there was always a preference for use of exploratory testing – this being because it allowed the manual tester to focus their time on test execution with very little wastage, meaning more tests could be run, which helped reduce the risk.

Summing up for now …

There was so much material, we had to split it up. So far we’ve taken a dip in, looking at how teams found themselves evolving to a whole team responsibility to quality.
Next time we’ll look at how testers found their voice, and some of the key skills and approaches my colleagues found increasingly pivotal in their day-to-day role.

Thank you to Janet Gregory for reviewing, editing, and donating her expertise for this article.

Top Takeaways from Agile Testing Days USA

By Leave a Comment

Agile testing days usa

Agile Testing Days came to the US for the first time this year, and if we do say so ourselves, they chose one of the best city for the venue — Boston!

SmartBear was one of the event sponsors, so luckily we were able to sit in on some of the amazing speakers, and we learned a few things along the way about Agile testing. Here are just a few of the top takeaways from Agile Testing Days USA.

How Agile Are You?

If you’re at Agile TD, your team is probably working in an Agile fashion or looking to become more Agile. But Selena Delesie noticed something in her time working with different companies — many were saying they were Agile without actually working in an Agile manner.

Many times she found that people wanted to control everything and strictly plan out their sprints, which didn’t lead to the flexibility that Agile demands. Additionally, she found that many people were no longer engaged in their work and were not present with themselves and their coworkers.

“Stop doing Agile; stop treating it as a process,” said Selena. “Start being Agile. Doing and being are very different things”

Selena challenged the audience to reconsider what Agile means to them and think about whether their work processes were actually Agile.

Going back to the Agile Manifesto — specifically thinking about individuals and interactions, customer collaboration, working software, and responding to change — and even the definition of “agile” are good places to start. Agile is supposed to mean that you’re able to think on your feet, be active, and lively but many companies have lost that, and agile has become a heavier process than Waterfall.

Selena says that having more process and control does not help them do their best work, it makes them mediocre. She especially emphasized connecting with people and building relationships. Making the effort to have conversations with people, listen to them, and show compassion can make all the difference.

It’s especially important for management to instill these values and be transparent in order for an Agile organization to work effectively. Selena suggested to stop having meetings behind closed doors and instead, show trust in your team. Let go of feeling the need to control things and make more of an effort to discover and learn.

 

Look Beyond the Tool

Test automation is has proven essential for Agile teams, which means you need certain test automation tools. But when implementing automation into your workflow, it’s important to consider a few factors besides just which tools you’re using. In “Automation for the People”, Christin Wiedemann suggested that you additionally think about the “what, why, and who” of automation.

Before even starting with automation, you should first determine why you’re automating and how it’s benefitting the rest of your team.

Christin notes that automation is much more than running script using a tool — you should expand the scope of what you define as automation to understand what you want to automate. This is the best way to avoid waste, whether it’s waste of time, waste of effort, waste of money, or waste of trust in automation.

Oftentimes, automation is used to take over repetitive tasks to eliminate human error that comes with doing things over and over, but a lot of people start automating without specific intention.

Saving time isn’t always the answer because you might end up replacing the time it takes to execute a test with coding and maintenance. The most important thing is to have a reason for automation and set goals.

You also want to think about who will be automating. This isn’t limited to thinking about who will be writing the scripts and running them. Testing is a team effort, and everyone has different skills and ways to add value, which means those people should be contributing to automation efforts. “At the end of the day, we all want to contribute to the same objective,” says Christin.

Additionally, Christin emphasizes the importance of seeing automated and manual testing as supporting practices rather than contrasting them against each other. Meeting your objectives will require a mix of both manual and automated testing, and she says that it’s important to remember that the results are more telling than the method of achieving them.

By broadening our definition of automation to think beyond tools, we can get a better understand of test automation to further the success of testing in our teams.

Musicians and Testers Aren’t so Different

Have you ever thought about how much test automation has in common with music history? If not, Angie Jones’ talk “Owning Our Narrative” gives a lot to think about.

Angie took us through the evolution of music, starting with Thomas Edison’s phonograph in 1877 and explaining how that idea developed to lead way to new recording instruments like the player piano and jukebox, then more recently to the cassettes, CDs, and MP3s that we know today.

As these changes took place, people began to develop new preferences when it came to music. Rather than being the social aspect it once was, music was now being sorted in genres. Record labels were recording electronically to get better sound, and automated machines were able to play one song instead of making people buy a whole album.

With all this innovation, musicians were forced to make adjustments and change the way they made music to meet the requirements of the different ways it was recorded and played. However, while the role of the musician changed in many ways, the skills remained the same.

If we think about the timeline of test automation, it’s very similar. We’ve found new ways to write and record tests, which has required testers to learn new skills as we embrace agility and shift left, but at the core, the role of the tester also remains the same.

Angie advocates that testers need to find a way to stay true to this role no matter the changes that are going on around us. Musicians were upset about automated machines taking jobs while everyone else was excited, and it was only by adapting to the new environment of music they were able to be successful.

She encourages us not to make the same mistakes as musicians. Embracing industry changes and making it work to your advantage will be far more productive than resisting it.

Angie notes that sometimes it can be difficult to convince others the value of our role in the changing landscape, but the more conscious testers are to those changes, the more prepared they are to own their narrative.

BONUS: Strategize Test Automation

We love Angie Jones so much, we wanted to make sure we invited her to SmartBear while she was in Boston. Luckily Angie accepted our invitation and taught our attendees about “Which Tests Should I Automate” at our Ministry of Testing meetup.

In this interactive session, Angie gave us a look at how she would perform a risk analysis of different test cases at Twitter to decide which tests should be automated, which tests you might want to automate, and which you would skip.

Many times we make the decision of automation off a gut reaction without considering the different aspects of each test — which are the most user critical? Which break the most often? Which would take the longest to fix?

By evaluating these factors against each other, we can make better decisions when it comes to test automation.

See the video below to watch the lesson for yourself.

Choosing Between Speed vs Quality in Software Development

By Leave a Comment

speed vs quality

Speed and quality — they’re debatably the two most sought-after exploits to participants in the software development lifecycle. Unfortunately, having one often comes at the cost of the other, leaving professionals to make a decision on which is more important.

Of course, different businesses may put a different emphasis on one or the other. For example, an e-commerce company might be more interested in maintaining consistent quality on their web application so customers will complete a purchase from their site and have a good enough experience to come back. Additionally, companies in high-stakes industries such as medical or financial recognize the unparalleled importance of quality over speed with their stakeholders.

Meanwhile, a large social network like Facebook may be more interested in releasing new features to keep their users engaged and interested than they are invested in worrying about quality. We can assume this because Facebook doesn’t even have dedicated QA or Testing team — they depend on developers and users to test new features since “social media is nonessential.”

In light of many organizations adopting processes such as Agile and DevOps, it may seem like speed has become a priority as teams take a more iterative approach to implementing new features and getting them to market more quickly.

But, it also depends on who you ask. If you consult a tester, they might say that these methodologies allow for more opportunities to intermittently assess quality and make way for Continuous Testing, while others might say that focus on speed often means overlooking those same concerns.

The moral of the story is that speed versus quality will be an eternal debate. Some will think one is more important, and some the other, with no one-size-fits-all answer.

Maybe the question should not be which do we prefer, but how we can do both more often. How can we combine the speed that development and management strive for with the quality our testers and QA engineers know is critical? How can we make Agile work for us in a way that is both effective and efficient?

Test automation exhibits the best of both worlds in speed and quality. Repeating scripts allows anyone to run tests in regression without having to manually repeat the same steps over and over so that more time can be spent exploring new features rather than checking old ones. However, automated testing can be time-consuming in itself. Even just considering all the different browsers and devices you have to test on, running these tests one after another in different environments is still tedious.

Parallel testing is an increasingly popular way many software teams are finding a way to strike the balance between speed and quality by taking automated testing a step further. By leveraging a Selenium grid, either with an in-house device lab or a cloud service like CrossBrowserTesting, developers and testers are able to run their scripts at the same time instead of one after the other, allowing teams to test across different environments in a fraction of the time it normally takes.

Cumbersome and tedious tasks like unit testing, smoke testing, regression testing, and cross-browser testing can be cut from taking weeks to days, days to hours, and hours to minutes with parallel testing. This not only makes the process faster to affect quicker deployments, but it also increases the range of coverage of your tests. Yes, this means that speed and quality are being addressed at the same time.

It’s revolutionary. Teams that have experienced this phenomenon know that life will never be the same again. They no longer have to waste their time and effort worrying about speed and quality (or arguing which is superior) but instead, they can focus on their jobs — building new features, creating new test cases, and delivering great software to customers.

So perhaps the challenge is to stop pursuing the debate of speed vs. quality and start embracing the idea of speed and quality. How can we find more ways to combine these ideas in order to work towards one common goal? How can we utilize faster web testing that doesn’t compromise quality, but enhance it?

5 Lessons Agile Teams Can Learn From Netflix

By Leave a Comment

Emerging from the days of video rentals and cable came a new way to digest as many movies and television shows as we want, when we want — Netflix.

There’s no doubt that the binge-watching business is booming. Netflix enjoys 109.25 million customers and subscribers consume more than 140 million hours of content every day, which means it doesn’t look like they’re going the ways of Blockbuster anytime soon.

So why do streaming services experience such high success compared to our previous viewing options? Why do people spend 120 minutes per day on Netflix, but only 30 – 60 minutes per week watching regular television? If you consider that Netflix is operating in an Agile fashion, it makes sense.

That’s not to say that the development team does or does not follow an Agile methodology, but it has to do more with the way that Netflix releases content. Similar to Agile development, it seems like Netflix also strives to focus on fast feedback, iterative changes, and cross-collaboration.

By taking a closer look at the way the company operates behind the scenes (or should we say, behind the screens), Agile development teams can learn quite a bit from Netflix. If you’re still not convinced, here are a few ways Netflix echoes Agile development and can teach teams to be more successful when building, testing, and delivering software.

  1. Do Better Than a Pilot Episode – Have you ever noticed that Netflix doesn’t have pilot episodes? That’s because a pilot episode is essentially a big test to see whether the network and the network’s audiences like the first episode enough to keep watching more. But what happens if they don’t? A lot of time and money wasted. Instead, Netflix pulls the perfect “balance of intuition and analytics” during production so that they know shows will be successful. House of Cards was greenlighted based on deep data analytics that told Netflix it would be well-received. In fact, Netflix Originals have a 35% higher success rate than new TV shows released on-network. Don’t wait until the end of development to test, and don’t let your users find faults for you. Integrate testing into every step of the software development lifecycle so you can be sure you’re delivering something your customers will love.
  2. Integrate Feedback – As per the manifesto, one of the key components of a successful Agile team is allowing for fast feedback, and just as importantly, implementing it in your next release. You don’t have to tell Netflix twice. Take House of Cards again — amid recent allegations about Kevin Spacey, there was a decision to be made about keeping him on the cast or canceling the show, both of which the company knew would agitate viewers. Instead, they decided to continue production with Robin Wright as the lead, showing that integrating feedback is a win-win. Whether that feedback is from testers while the application is in production or from customers after the software is delivered, incorporating feedback maintains continuous growth and improvement.
  3. Be Compatible with Your Customers – Netflix supports 900 different devices — that’s almost as many as CrossBrowserTesting has in the cloud. Between laptops, smartphones, and tablets, not to mention different browsers, operating systems and screen sizes, consistency is key. Netflix has a huge customer base, which means they probably have a lot of diversified device usage. If they didn’t make the application accessible to each one of those, they simply would not enjoy the amount of success they do today. Cross-browser testing is a no-brainer, so be sure to not to skip it if you really want to release a high-quality application for every user.
  4. Release Often – One of the major advantages of Agile is that it allows teams to release software more often instead of having longer release cycles and only delivering every few months or even years. Netflix has found that releasing new movies and shows every on a weekly basis keeps customers excited and intrigued, so they don’t get bored by the same selections over and over. That means once bingers are done with Stranger Things, they still have the next season of Black Mirror to look forward to. It’s no secret that consumer expectations and demands these days are high — they constantly want their hands on the next best thing. But at the end of the day, it’s pretty straightforward. If people want to consume your product, give them the means that allows them to do that as much as they want with Continuous Integration and Delivery.
  5. Embrace Detailed User Stories – A user story describes a software feature as a customer would see it. The goal of Agile is to bring user stories from ideation to deployment, considering who the user is, what they want to accomplish, and how they accomplish it with that feature. Netflix takes this to the next level by understanding every user story with advanced personalization. In fact, 75% of Netflix views are a result of their recommendation engine. By creating different trailers and artwork for content based on viewers previous movie and show choices, they’re able to more precisely communicate recommendations with people based on their interests and behavior. Take it from Netflix and get familiar with user personas and the customer journeys that take place throughout your application to better plan throughout development.

Whether you stream shows or surf TV channels, there’s a lot we can learn from the media mogul. As Agile teams embrace speed and quality throughout testing and development, Netflix provides the blueprint for success from production to breakout deployment.

Running Automated Testing Software in an Agile Environment

By Leave a Comment

Claire Mackerras Automated Testing Agile Environment

There is no doubt that automated software testing is critical for maintaining agility and establishing continuous improvement. As more teams adopt both automated testing and Agile development, it becomes clear they are best practiced together.

Any testing that is being performed for more than two or three times in a short span of time can be automated with paid and/or open source tools. Execution of unit tests, functional tests, continuous integration, and continuous deployment are common ways of utilizing automation for the greatest benefit.

Automating software testing is especially preferred in an Agile environment because it accomplishes the following tasks:

  • Identifies test productivity problems
  • Swiftly responds to the testers
  • Helps testers discuss and examine solutions
  • Implements technology to improve test processes
  • Assess automation possibilities for upcoming projects

Benefits of Automated Testing in an Agile Environment

Automation technology accounts for repeated testing and delivers extensive functionality and coverage through the software testing life cycle. If automation software is correctly executed, it provides the following benefits:

  • Improved Software Quality – Consistent, repetitive and extensive automation tests helps to deliver high-quality software
  • Improved Documentation – Automated tests produces instant documented quality proofs, comprises extensive flop/pass consequences, brings down testing manpower and time
  • Bring down testing manpower and time – As automated testing is quicker than manual testing, as these are labor-intensive, and can do verification of thousands to millions of test permutations from minutes to hours.
  • Cheaper – It also helps bring a remarkable reduction in the expenses related to the production of best-in-class software.

Automated Testing Trends

Agile environment automated testing

Organizations have largely merged automated testing and Agile development due to major trends that have arisen in the last few years.

Let’s take a brief look at some of the trends as follows:

1. Mobile Testing

Mobile testing will continue to remain at the top of the list for automation trends, owing to the remarkable shift in the adoption of mobile technology. Mobile application testing will indisputably have a major impact on the software testing industry with functionality, compatibility, performance, usability and security testing as major areas of focus.

In the next few years, mobile test automation will match web test automation if the market for mobile continues to rise at the same pace. There’s a great possibility in the open-source framework, mobile development, cross-browser testing, and app markets to expand for the mobile test automation landscape.

2. Big Data Testing

With an increase in Big Data analytics, quality management for Big Data implementations will emerge. Applications, networks, and devices will generate a huge source of unstructured data, ultimately leading to faulty and costly decisions.

Testing quality ahead of product release will continue to remain critical. With this type of testing, businesses will aim to get hard numbers and statistical indicators on whether the application behavior will meet the demand of customers or not.

3. Testing with TDD & BDD

Enterprises are likely to adopt TDD and BDD to deliver applications as per the customer needs, without any defects or delay. By engaging in this technology, teams can get an actual idea of what is required.

As a result, enterprises are better off in terms of cost and time since the longer it takes to identify bugs, the more it costs to fix them.

4. DevOps

Alliance among IT professionals, developers and operation engineers has increased since the emergence of DevOps. More and more organizations are merging the role of testers with operational engineers and developers due to overlapping competencies in the continuous Agile cycles.

Experts predict that testers, developers, and operation engineers will similarly need to utilize the same tools to increase the collaboration. This technique will promote better communication and feedback among teams to achieve Agile development, testing, and delivery.

6. Micro Testing

Enterprises financing in Micro Services make it easier to test an independent piece of functionality without altering other applications.

This trend will inevitably rise in the coming years, as it certifies testing complex applications which comprise small, independent processes and architectures.

The Verdict

Automated Testing is considered by many software testers as one of the most forthcoming trends practiced in the overall software development lifecycle today. With the help of automated testing, one can get more effective and efficient testing results in an Agile environment.

About the author: Claire Mackerras is a Senior Quality Assurance Engineer & Editor at BugRaptors. BugRaptors is a CMMi5 certified company with extensive experience as a third party testing vendor for tools in Software Testing & QA. Claire enjoys creating, uncovering, and disseminating new and interesting perspectives on technology and software testing.

What’s the True Cost of a Software Bug?

By Leave a Comment

The Most Costly and Expensive Software Bug Screw Ups

Test early and test often — that’s what everyone says will help you avoid the high cost of deploying a software bug. But a bug might not seem like a big deal without a dollar sign attributed to it. Why invest in testing if you can just fix your mistake after? What’s the true cost of a software bug?

While there’s no set cost you can ascribe to a software bug found after the product release because it’s highly dependant on the organization’s size, customers, employees, and debugging resources, we can look at a few statistics and examples that show just how damaging it can be.

Starting with the big picture, software failures cost the worldwide economy $1.1 trillion in 2016. These failures were found at 363 companies, affected 4.4 billion customers, and caused more than 315 years of lost time. Most of these incidents were avoidable, but the software was simply pushed to production without proper QA.

Those may be stunning numbers, but you could still be thinking it doesn’t mean much to your specific organization, so let’s look at it on a smaller scale.

IBM found that the cost to fix an error found after product release was 4 to 5 times higher than if it’s uncovered during the design phase, and up to 100 more expensive than if it’s identified in the maintenance phase.

The costs go up as the bug moves through the SDLC. For example, IBM estimates that if a bug costs $100 to fix in Gathering Requirements phase, it would be $1,500 in QA testing phase, and $10,000 once in Production

And software bug expenses aren’t just related to the impact on internal operations. When a company announces a software failure, they lose an average of $2.3 B of shareholder value just on the first day.

The fact of the matter is that a software bug can affect everything from indirect costs like customer loyalty and brand reputation to more direct losses in business revenue and wasted time.

Companies Who Learned the Hard Way

You might think a software bug can only be so bad, or that any bug big enough to cause significant financial distress will be a complex mathematical error. In reality, some of the most detrimental bugs had minor mistakes that developers just didn’t catch because they weren’t used to putting the software through proper testing.

The following are companies who suffered monumental losses — some that almost put them out of business — due to easy errors, new software, or a bad line of code.

NASA – The Mars Climate Orbiter was a robotic space probe that NASA Launched in 1998 to study the climate and atmosphere of Mars. All was well until communication got cut-off a year into the mission when the Orbiter got lost in space. When dealing with spacecraft engineering, there’s a lot that could go wrong — it’s literally rocket science. However, the error could have been an easy fix if caught on time. The $125 million spacecraft presumably orbited too close to Mars’ surface and disintegrated because the engineering team failed to convert their measurements from U.S. units to metric. However, that’s not the only time a programming blunder destroyed a NASA rocket.

Knight Capital Group – As one of the biggest American financial services firms, Knight had a lot to lose, and in 2012 they lost $440 million of it. All it took was thirty minutes to drop 75 percent of their shares after an automated set of orders was sent out all at once to nearly 150 stocks instead of over a period of days like it was supposed to. The function, known as “power peg,” made stocks move up and down, lowering the stock price of all the affected companies. The loss was almost four times the company’s profit and would have put them into bankruptcy if a group of investors hadn’t come to the rescue.

AT&T – In January of 1990, AT&T failed to meet the basic needs of their customers as thousands attempted to call their friends, family, and airports. Long distance calling was down for nine hours, and customers sparked outrage after experiencing a total of 75 million missed phone calls and 200,000 lost airline reservations. Initially, AT&T thought they were being hacked, but the issue was due to a software bug. AT&T had just recently “updated” code, but the change of speed ended up being detrimental to a system that couldn’t keep up. AT&T lost an estimated $60 million from long distances charges that didn’t go through after being down less than a full day in an error that was the result of one line of buggy code.

EDS Child Support – In 2004, a computer system managed to overpay 1.9 million people, underpay 700,000, accumulate $7 billion in uncollected child support, backlog 239,000 cases, and get 36,000 new cases “stuck” in the system. This happened after a software company introduced a complex new IT system to the UK’s Child Support Agency. Meanwhile, the Department for Work and Pensions was restructuring their entire agency. The software was incompatible as the systems clashed, and as a result, the oversight cost taxpayers $1 billion.

Apple – For many consumers, Apple can do no wrong, but when they replaced Google Maps with Apple Maps on iOS 6 in 2012, they managed to put a pause on customer’s endless adoration. The update was nothing more than a product of corporate politics — Apple no longer wanted to be tied to Google’s application, so they made their own version. However, in Apple’s haste to release the latest and greatest map app, they made some unforgivable lapses in navigation warranting the #ios6apocalypse hashtag. Just a few of the many Apple Maps fails include erased cities, buildings disappearing, flattened landmarks, duplicated islands, warped graphics, and false location data. While an exact financial loss was never attributed to this lack of judgment, it made customers question whether Apple prioritized their needs over their money as they considered switching to the very competitor Apple was trying to combat.

An Argument for Agile and Continuous Testing

As technology evolves, so do our development methodologies and knowledge of testing practices. Getting stuck in the old age of software development means that crucial processes go ignored.

For example, while a Waterfall methodology used to be the go-to back in the day, high functioning software teams have realized that following Agile practices better support changes made throughout development. Agile’s incremental approach allows more feedback, flexibility, and of course, testing so that every time a feature, fix, or function is added or changed in code, it’s checked for bugs.

Since teams have adopted Agile and Continuous Integration, thus adopting continuous testing, software is higher quality after a product release because bugs and inconsistencies are caught earlier and fixed easier. In turn, organizations have found that this helps avoid preventable bugs, save time and money, and retain their customers’ trust.

How to Prevent Your Next Software Bug

While we can’t expect to test everything and go our entire lives deploying a product that’s 100% error free, we can make strides to safeguard software as best we can. Many of the worst bugs that we encounter are simple mistakes that could have been caught with a more thorough quality analysis.

The moral of the story is to test your software. If you don’t, it’ll cost you.

For more proof of why software testing is so important, check out Why Testing Matters and what the top QA professionals have to say.

Establishing Your Scrum Team’s Definition of Done

By 1 Comment

Kanban scrum definition of done

When talking about building culture in the software development process, deciding on a team’s Definition of “Done” has become a major component in the shift to Agile — most commonly in Scrum — that ensures every player works together toward the same end goals.

For example, if a developer is working on a certain user story and tells the product owner they’re done but it hasn’t been tested, this will backup product release once the mistake has been realized, or potentially affect reputation if a buggy app is deployed without being caught.

While one person may be finished with their tasks, that does not mean the project is done as a whole, and saying so can limit progress if it’s not done in all parts and stages. Teams that do not consider this groundwork often find that it severely impacts productivity.

The Definition of Done for Scrum Teams

When Agile strives for iterative development, fast feedback, and adapting requirements, having limited communication will delay a team’s acceleration and pace when pushing a project or feature set to market.

Maybe you feel factors like testing, reviewing, documenting, releasing, or even collaboration should go without saying, but many team members will consider tasks done based on whether or not they fulfilled their individual role.

Because Scrum has lightweight management overhead and bi-weekly check-ins (Sprints), having a Definition of Done becomes even more important to your team’s efficiency. Nothing kills team chemistry and collaboration quicker than having tasks done “half-way” only because another teammate thought they only had to scope out a project, and not actually supply mocks or a rough draft.

Coming up with a Definition of Done is about the bigger picture organizational process of your team. By looking at projects as a collaborative effort instead of dividing responsibilities among different departments or one leader, you encourage the communication and transparency that’s necessary for Agile.

This is why creating a Definition of Done should be a starting point for Scrum teams. Your organization should essentially have a checklist of specific criteria and requirements that need to be fulfilled before the user story is done. However, the definition will likely differ from organization to organization based on the size of the teams, different fulfilled roles, and different project requirements.

Criteria to Consider When Defining Done

  • Prioritizing tasks – Having an order and process in place should be a prerequisite for defining done. This means having a set expectation for every time a task moves throughout your Kanban flow, whether something is selected for development or in code review. Also, this includes deciding whether a Definition of Done should just be attributed to user stories, or features and epics as well.
  • Including testing – Testing is often the most overlooked step. Is more manual or automated testing needed for this particular project? Are you using regression tests throughout development or just at the very end? Have you tested across multiple browsers? What processes do you have in place for analyzing risk and evaluating security?
  • Involving every role – If your developers have a different Definition of Done than your testers, for example, projects aren’t going to be done by everyone’s standard. This also includes product owners, project managers, scrum masters, designers, automation engineers, QA, or any other role. Furthermore, the definition should not be prescribed by one team member, but instead agreed upon by everyone.
  • Coming up with a timeline – How many iterations are you striving to make throughout the day? How many are you committing to at the end of the day? What are the goals during your sprints? How do you keep track of this and what happens if a task is not completed in time?
  • Ask what makes the process complete – Is it done when the product goes through a last round of testing, is it done after a last once-over by the product owner, or is it when it’s released to the end user? Having an end point is just as important as having a designated place to start.
  • Document the definition – Have the information easily accessible for people to reference, even if it’s printed out as a checklist. This way, everyone can double-check before they move a task into “done.” Additionally, in the same way, that Agile is adaptable, your Definition of Done should be, too. Make sure to return to the document and reassess processes that need to be adjusted according to changing requirements.

How do you decide when a project is “done”? Share your thoughts in our comments!