CrossBrowserTesting.com

A Design, Development, Testing Blog

Challenges of Being the Team’s Sole Tester

By Leave a Comment

lone software tester agile

PART ONE

Introduction

Until relatively recently, the chances are that if you were a tester on a project, you’d be one of a number of such people. You’d have other members of the team to try ideas out, share the workload, cover for you when you’re away.

With the recent drive towards agile, we’re seeing the makeup of the team change dramatically. Projects can typically be supported by much smaller teams working permanently to evolve the product. This can often result in there only being a single tester on a team.

What are the challenges being the sole tester on such a project? How can you work with these constraints? This has been the subject of a series of workshops with fellow testers within my company, which I’m excited to share the outcome with you …

The Iron Triangle

Before we get underway, it’s useful to revisit the following principle within project management which we found underpinned many of our conversations. It’s useful for thinking about the constraints we’re working within on a project, especially in agile.

Image result for scope cost schedule quality

In the iron triangle gives us the idea that the “quality” of a project is defined by three attributes of your project – cost, scope, schedule (or time).

You might have heard the adage “cost, scope, schedule … pick two”. However, ideally on a project, there should be only one cast iron attribute – what management consultant Johanna Rothman calls “your project driver” in her book Manage It.

Within any project you can only really have only one attribute which is fixed – it could be “we need this done by X” (schedule) or “there is only Y budget for this” (cost). The skill of a manager is to work with this constraint and plan around what can be done with the other two attributes to achieve this goal.

Within traditional test management, there are a lot of parallels for applying this same theory to test planning. Within this dynamic the fields are,

  • Scope – how much testing you’d like to achieve
  • Cost (or rather typically resources) – having more testers allow you to execute more activity
  • Schedule or timeframe – how long you have to do things

It should be obvious that if you have a large scope, and a short timeframe, one solution would be to have more testers on it. Although of course in the real world, there are constraints as to how much this can be pushed, and good test management revolves around knowing and pragmatically working within these constraints.

Another solution, of course, is less testers, but it means that it takes longer to get through everything you’d like. Great for the test budget, but typically people like developers need to be paid to be on call to fix the bugs and the bugs are found later in the cycle, so developers need to be available longer.

Finally, if you find yourself in a situation where your available people and schedule are fixed, the only thing to do it to prioritise your scope as it’s the only thing you have control of.

Understanding this dynamic and the trade-offs is important because it was a core part of the discussions that were held, together with ways they could be handled and occasionally hacked.

Under pressure

A common initial feeling of someone stepping into the role of a sole tester was that of feeling under pressure.

Especially in an agile project, the timeframe is set by the sprint duration and your testing team size (although this can be “hacked” as we’ll discuss later).

Just back in 2013, one of our projects would have an annual release, which would involve a two-month testing window and would keep our test team of six busy.

Fast forward to 2018, and we’re now working in agile teams where we are creating deliverable code in a two-week sprint window using only two testers.

A key enabler in this was adopting a robust automated testing framework, which was easy to maintain with changes in the system under test. Such a suite did not grow overnight – and required a lot of work between testers and developers to build the right thing from a framework perspective, as well as to work through a prioritised list of useful automated tests to have in place. In working out idea scenarios and prioritisation, testers found themselves well-placed to lead these discussions. Over time, this suite was able to carry the functional regression load.

Automated testing helped, however, it didn’t eliminate the testing role. But testers found that their role did change dramatically. Most manual testing effort now focused on testing new or changed functionality in-depth during a sprint, as well as helping out with increasing ownership on test scenario selection for the automated suite (as well as shock-horror, learning to code their own tests).

In teams which are still undergoing a level of “forming” – a term used to describe those that have relatively new team members, some of whom were relatively new to working in an agile team – it was quite common for the sole tester to feel initially like they were the “point of blame”. If something gets out into production, the inevitable uncomfortable question can be asked of “why didn’t you test that?”

We shared a few of our experiences looking for general themes. Part of the problem that we were acutely aware of was time, and it’s not always possible to test everything you want to.

In many examples of a release where a defect had been undetected, manual testing had always occurred. Typically though, something was missed, or it was not imagined that a particular scenario could have been capable of causing an issue.

It’s worth taking a moment to think about how this was addressed in “classic” waterfall projects. A test lead would create a plan of what’s to be covered in consultation with many people on the project, but especially using the requirements. From this, they would build up a series of scenarios to be covered and make estimations around the resources and timescale.

However, on these classic projects, this was not the end of the story. It was the tester’s job to produce the best schedule they could, but it was known that this would not be perfect on the first draft. This was why such emphasis was put on the importance of reviewing – firstly by peer testers to see if enough testing heuristic variation has been employed, but also by a wider team such as project managers, customers, developers.

The aim with reviews was to find gaps in the plan and address them. This allowed the final testing scheme to be the most robust scheme of testing possible. This could come from developers saying, “we’re also making changes in this area” or our customers stating there’s an expectation that “most people will…”.

Within agile, it can be easy to forget that this level of contribution is still required. It needs to occur, however, it’s in a more informal, often verbal manner.
Within my colleagues, there is a general consensus that the tester becomes more responsible for facilitating a discussion around testing, much closer to what some organisations will call “a quality coach”.

A core tool for having these conversations is the use of mind maps, which the group has been using with success since 2013. A mind map allows the author to show for a particular feature, all the different variations, and the factors that they’re planning to follow in a one-page diagram.

When done well, they’re intuitive to read and can even be posted in common areas for people to look at. Their brevity helps get people to read them — “I haven’t had time to read that thirty-page document you’ve sent yet” is a frequent complaint in IT.

Even with a mind map in place, there is a natural tendency for the rest of the team to rubber stamp things. A sample conversation might go like this:

Tester: Did you have anything to add to the test mind map I sent out?

Team member: Uh … I guess it’s okay?

We all have a tendency to say something along the lines of “I guess so” for something we’ve not properly read. It’s important to still follow up with a brief conversation about what’s in your coverage – this can be individually with each team member, but often it’s better with the whole team. Just after stand-up can be a great time for this to occur.

If a member of the team notices there’s a mistake about the approach or some items that are missing, it’s expected for them to provide that feedback. Likewise, if the developer does more change than initially anticipated, there’s an expectation for them to tell the tester what they might also want to consider.

Often what you’ll read in agile literature about a “whole team approach” is essentially this: the whole team takes responsibility to give feedback whether it’s about how a story is defined, how a feature is being developed, or how testing is being planned.

A good indicator of when a team has made this mind shift is the use in retrospective of “we” instead of “you” – “WE missed this, WE need to fix this”. Teams where this happens have a much more positive dynamic. It’s important that this applies not just to testing.

Other examples include when a developer builds exactly what was on the story card, but not what was actually wanted (“we failed to elaborate”), when a story turns out much bigger than first thought (“we failed to estimate”) etc.

That said though, agile does not mean the breakdown of individual responsibility. A clear part of the tester’s role is to set clear expectations for the team of what they can do, how much effort it will take, and how you’re approaching it. But there needs to be team input to fine tune this to deliver the best value.

Mainly testing will revolve around changes to a product, for which the rest of your team are your first “go-tos” as fellow subject matter experts on the item. Occasionally as a tester though, you will find the value to consult with another peer tester – and there is an expectation that testers who are part of the same organisation but in other teams can be approached to be asked for their advice and thoughts on a test approach. Within our company, there is an expectation that all testers make some time in their schedule to support each other in this way. This, in many ways, echoes the “chapter” part of the Spotify model, with testing being it’s its own chapter of specialists spread across multiple teams/squads who provide test discipline expertise.

Reaching out to other testers like this is important; it creates a sense of community and the opportunity to knowledge share across your organisation.

Waterfall into agile won’t go…

There have been some “agile-hybrid” projects where there has been an expectation of set numbers of people being able to perform a set volume of testing in a set time (sprint). This can sometimes be problematic as the tester involved in execution hasn’t been involved in setting the expectation of what volume of tests are likely. And hence, it can feel like working against an arbitrary measure not based in reality.

In such a situation, it’s like being given an iron triangle where someone has given you “here’s your schedule, here’s your resources … so you need to fit in this much scope”. When faced with so many tests to run, it obviously helps to have them prioritised so that you’re always running the most important test next. When three areas are fixed, what suffers is the quality – it gets squished.

On projects where test scripting was not mandated by contract, there was always a preference for use of exploratory testing – this being because it allowed the manual tester to focus their time on test execution with very little wastage, meaning more tests could be run, which helped reduce the risk.

Summing up for now …

There was so much material, we had to split it up. So far we’ve taken a dip in, looking at how teams found themselves evolving to a whole team responsibility to quality.
Next time we’ll look at how testers found their voice, and some of the key skills and approaches my colleagues found increasingly pivotal in their day-to-day role.

Thank you to Janet Gregory for reviewing, editing, and donating her expertise for this article.

Top Takeaways from Agile Testing Days USA

By Leave a Comment

Agile testing days usa

Agile Testing Days came to the US for the first time this year, and if we do say so ourselves, they chose one of the best city for the venue — Boston!

SmartBear was one of the event sponsors, so luckily we were able to sit in on some of the amazing speakers, and we learned a few things along the way about Agile testing. Here are just a few of the top takeaways from Agile Testing Days USA.

How Agile Are You?

If you’re at Agile TD, your team is probably working in an Agile fashion or looking to become more Agile. But Selena Delesie noticed something in her time working with different companies — many were saying they were Agile without actually working in an Agile manner.

Many times she found that people wanted to control everything and strictly plan out their sprints, which didn’t lead to the flexibility that Agile demands. Additionally, she found that many people were no longer engaged in their work and were not present with themselves and their coworkers.

“Stop doing Agile; stop treating it as a process,” said Selena. “Start being Agile. Doing and being are very different things”

Selena challenged the audience to reconsider what Agile means to them and think about whether their work processes were actually Agile.

Going back to the Agile Manifesto — specifically thinking about individuals and interactions, customer collaboration, working software, and responding to change — and even the definition of “agile” are good places to start. Agile is supposed to mean that you’re able to think on your feet, be active, and lively but many companies have lost that, and agile has become a heavier process than Waterfall.

Selena says that having more process and control does not help them do their best work, it makes them mediocre. She especially emphasized connecting with people and building relationships. Making the effort to have conversations with people, listen to them, and show compassion can make all the difference.

It’s especially important for management to instill these values and be transparent in order for an Agile organization to work effectively. Selena suggested to stop having meetings behind closed doors and instead, show trust in your team. Let go of feeling the need to control things and make more of an effort to discover and learn.

 

Look Beyond the Tool

Test automation is has proven essential for Agile teams, which means you need certain test automation tools. But when implementing automation into your workflow, it’s important to consider a few factors besides just which tools you’re using. In “Automation for the People”, Christin Wiedemann suggested that you additionally think about the “what, why, and who” of automation.

Before even starting with automation, you should first determine why you’re automating and how it’s benefitting the rest of your team.

Christin notes that automation is much more than running script using a tool — you should expand the scope of what you define as automation to understand what you want to automate. This is the best way to avoid waste, whether it’s waste of time, waste of effort, waste of money, or waste of trust in automation.

Oftentimes, automation is used to take over repetitive tasks to eliminate human error that comes with doing things over and over, but a lot of people start automating without specific intention.

Saving time isn’t always the answer because you might end up replacing the time it takes to execute a test with coding and maintenance. The most important thing is to have a reason for automation and set goals.

You also want to think about who will be automating. This isn’t limited to thinking about who will be writing the scripts and running them. Testing is a team effort, and everyone has different skills and ways to add value, which means those people should be contributing to automation efforts. “At the end of the day, we all want to contribute to the same objective,” says Christin.

Additionally, Christin emphasizes the importance of seeing automated and manual testing as supporting practices rather than contrasting them against each other. Meeting your objectives will require a mix of both manual and automated testing, and she says that it’s important to remember that the results are more telling than the method of achieving them.

By broadening our definition of automation to think beyond tools, we can get a better understand of test automation to further the success of testing in our teams.

Musicians and Testers Aren’t so Different

Have you ever thought about how much test automation has in common with music history? If not, Angie Jones’ talk “Owning Our Narrative” gives a lot to think about.

Angie took us through the evolution of music, starting with Thomas Edison’s phonograph in 1877 and explaining how that idea developed to lead way to new recording instruments like the player piano and jukebox, then more recently to the cassettes, CDs, and MP3s that we know today.

As these changes took place, people began to develop new preferences when it came to music. Rather than being the social aspect it once was, music was now being sorted in genres. Record labels were recording electronically to get better sound, and automated machines were able to play one song instead of making people buy a whole album.

With all this innovation, musicians were forced to make adjustments and change the way they made music to meet the requirements of the different ways it was recorded and played. However, while the role of the musician changed in many ways, the skills remained the same.

If we think about the timeline of test automation, it’s very similar. We’ve found new ways to write and record tests, which has required testers to learn new skills as we embrace agility and shift left, but at the core, the role of the tester also remains the same.

Angie advocates that testers need to find a way to stay true to this role no matter the changes that are going on around us. Musicians were upset about automated machines taking jobs while everyone else was excited, and it was only by adapting to the new environment of music they were able to be successful.

She encourages us not to make the same mistakes as musicians. Embracing industry changes and making it work to your advantage will be far more productive than resisting it.

Angie notes that sometimes it can be difficult to convince others the value of our role in the changing landscape, but the more conscious testers are to those changes, the more prepared they are to own their narrative.

BONUS: Strategize Test Automation

We love Angie Jones so much, we wanted to make sure we invited her to SmartBear while she was in Boston. Luckily Angie accepted our invitation and taught our attendees about “Which Tests Should I Automate” at our Ministry of Testing meetup.

In this interactive session, Angie gave us a look at how she would perform a risk analysis of different test cases at Twitter to decide which tests should be automated, which tests you might want to automate, and which you would skip.

Many times we make the decision of automation off a gut reaction without considering the different aspects of each test — which are the most user critical? Which break the most often? Which would take the longest to fix?

By evaluating these factors against each other, we can make better decisions when it comes to test automation.

See the video below to watch the lesson for yourself.

Choosing Between Speed vs Quality in Software Development

By Leave a Comment

speed vs quality

Speed and quality — they’re debatably the two most sought-after exploits to participants in the software development lifecycle. Unfortunately, having one often comes at the cost of the other, leaving professionals to make a decision on which is more important.

Of course, different businesses may put a different emphasis on one or the other. For example, an e-commerce company might be more interested in maintaining consistent quality on their web application so customers will complete a purchase from their site and have a good enough experience to come back. Additionally, companies in high-stakes industries such as medical or financial recognize the unparalleled importance of quality over speed with their stakeholders.

Meanwhile, a large social network like Facebook may be more interested in releasing new features to keep their users engaged and interested than they are invested in worrying about quality. We can assume this because Facebook doesn’t even have dedicated QA or Testing team — they depend on developers and users to test new features since “social media is nonessential.”

In light of many organizations adopting processes such as Agile and DevOps, it may seem like speed has become a priority as teams take a more iterative approach to implementing new features and getting them to market more quickly.

But, it also depends on who you ask. If you consult a tester, they might say that these methodologies allow for more opportunities to intermittently assess quality and make way for Continuous Testing, while others might say that focus on speed often means overlooking those same concerns.

The moral of the story is that speed versus quality will be an eternal debate. Some will think one is more important, and some the other, with no one-size-fits-all answer.

Maybe the question should not be which do we prefer, but how we can do both more often. How can we combine the speed that development and management strive for with the quality our testers and QA engineers know is critical? How can we make Agile work for us in a way that is both effective and efficient?

Test automation exhibits the best of both worlds in speed and quality. Repeating scripts allows anyone to run tests in regression without having to manually repeat the same steps over and over so that more time can be spent exploring new features rather than checking old ones. However, automated testing can be time-consuming in itself. Even just considering all the different browsers and devices you have to test on, running these tests one after another in different environments is still tedious.

Parallel testing is an increasingly popular way many software teams are finding a way to strike the balance between speed and quality by taking automated testing a step further. By leveraging a Selenium grid, either with an in-house device lab or a cloud service like CrossBrowserTesting, developers and testers are able to run their scripts at the same time instead of one after the other, allowing teams to test across different environments in a fraction of the time it normally takes.

Cumbersome and tedious tasks like unit testing, smoke testing, regression testing, and cross-browser testing can be cut from taking weeks to days, days to hours, and hours to minutes with parallel testing. This not only makes the process faster to affect quicker deployments, but it also increases the range of coverage of your tests. Yes, this means that speed and quality are being addressed at the same time.

It’s revolutionary. Teams that have experienced this phenomenon know that life will never be the same again. They no longer have to waste their time and effort worrying about speed and quality (or arguing which is superior) but instead, they can focus on their jobs — building new features, creating new test cases, and delivering great software to customers.

So perhaps the challenge is to stop pursuing the debate of speed vs. quality and start embracing the idea of speed and quality. How can we find more ways to combine these ideas in order to work towards one common goal? How can we utilize faster web testing that doesn’t compromise quality, but enhance it?

5 Lessons Agile Teams Can Learn From Netflix

By Leave a Comment

Emerging from the days of video rentals and cable came a new way to digest as many movies and television shows as we want, when we want — Netflix.

There’s no doubt that the binge-watching business is booming. Netflix enjoys 109.25 million customers and subscribers consume more than 140 million hours of content every day, which means it doesn’t look like they’re going the ways of Blockbuster anytime soon.

So why do streaming services experience such high success compared to our previous viewing options? Why do people spend 120 minutes per day on Netflix, but only 30 – 60 minutes per week watching regular television? If you consider that Netflix is operating in an Agile fashion, it makes sense.

That’s not to say that the development team does or does not follow an Agile methodology, but it has to do more with the way that Netflix releases content. Similar to Agile development, it seems like Netflix also strives to focus on fast feedback, iterative changes, and cross-collaboration.

By taking a closer look at the way the company operates behind the scenes (or should we say, behind the screens), Agile development teams can learn quite a bit from Netflix. If you’re still not convinced, here are a few ways Netflix echoes Agile development and can teach teams to be more successful when building, testing, and delivering software.

  1. Do Better Than a Pilot Episode – Have you ever noticed that Netflix doesn’t have pilot episodes? That’s because a pilot episode is essentially a big test to see whether the network and the network’s audiences like the first episode enough to keep watching more. But what happens if they don’t? A lot of time and money wasted. Instead, Netflix pulls the perfect “balance of intuition and analytics” during production so that they know shows will be successful. House of Cards was greenlighted based on deep data analytics that told Netflix it would be well-received. In fact, Netflix Originals have a 35% higher success rate than new TV shows released on-network. Don’t wait until the end of development to test, and don’t let your users find faults for you. Integrate testing into every step of the software development lifecycle so you can be sure you’re delivering something your customers will love.
  2. Integrate Feedback – As per the manifesto, one of the key components of a successful Agile team is allowing for fast feedback, and just as importantly, implementing it in your next release. You don’t have to tell Netflix twice. Take House of Cards again — amid recent allegations about Kevin Spacey, there was a decision to be made about keeping him on the cast or canceling the show, both of which the company knew would agitate viewers. Instead, they decided to continue production with Robin Wright as the lead, showing that integrating feedback is a win-win. Whether that feedback is from testers while the application is in production or from customers after the software is delivered, incorporating feedback maintains continuous growth and improvement.
  3. Be Compatible with Your Customers – Netflix supports 900 different devices — that’s almost as many as CrossBrowserTesting has in the cloud. Between laptops, smartphones, and tablets, not to mention different browsers, operating systems and screen sizes, consistency is key. Netflix has a huge customer base, which means they probably have a lot of diversified device usage. If they didn’t make the application accessible to each one of those, they simply would not enjoy the amount of success they do today. Cross-browser testing is a no-brainer, so be sure to not to skip it if you really want to release a high-quality application for every user.
  4. Release Often – One of the major advantages of Agile is that it allows teams to release software more often instead of having longer release cycles and only delivering every few months or even years. Netflix has found that releasing new movies and shows every on a weekly basis keeps customers excited and intrigued, so they don’t get bored by the same selections over and over. That means once bingers are done with Stranger Things, they still have the next season of Black Mirror to look forward to. It’s no secret that consumer expectations and demands these days are high — they constantly want their hands on the next best thing. But at the end of the day, it’s pretty straightforward. If people want to consume your product, give them the means that allows them to do that as much as they want with Continuous Integration and Delivery.
  5. Embrace Detailed User Stories – A user story describes a software feature as a customer would see it. The goal of Agile is to bring user stories from ideation to deployment, considering who the user is, what they want to accomplish, and how they accomplish it with that feature. Netflix takes this to the next level by understanding every user story with advanced personalization. In fact, 75% of Netflix views are a result of their recommendation engine. By creating different trailers and artwork for content based on viewers previous movie and show choices, they’re able to more precisely communicate recommendations with people based on their interests and behavior. Take it from Netflix and get familiar with user personas and the customer journeys that take place throughout your application to better plan throughout development.

Whether you stream shows or surf TV channels, there’s a lot we can learn from the media mogul. As Agile teams embrace speed and quality throughout testing and development, Netflix provides the blueprint for success from production to breakout deployment.

Running Automated Testing Software in an Agile Environment

By Leave a Comment

Claire Mackerras Automated Testing Agile Environment

There is no doubt that automated software testing is critical for maintaining agility and establishing continuous improvement. As more teams adopt both automated testing and Agile development, it becomes clear they are best practiced together.

Any testing that is being performed for more than two or three times in a short span of time can be automated with paid and/or open source tools. Execution of unit tests, functional tests, continuous integration, and continuous deployment are common ways of utilizing automation for the greatest benefit.

Automating software testing is especially preferred in an Agile environment because it accomplishes the following tasks:

  • Identifies test productivity problems
  • Swiftly responds to the testers
  • Helps testers discuss and examine solutions
  • Implements technology to improve test processes
  • Assess automation possibilities for upcoming projects

Benefits of Automated Testing in an Agile Environment

Automation technology accounts for repeated testing and delivers extensive functionality and coverage through the software testing life cycle. If automation software is correctly executed, it provides the following benefits:

  • Improved Software Quality – Consistent, repetitive and extensive automation tests helps to deliver high-quality software
  • Improved Documentation – Automated tests produces instant documented quality proofs, comprises extensive flop/pass consequences, brings down testing manpower and time
  • Bring down testing manpower and time – As automated testing is quicker than manual testing, as these are labor-intensive, and can do verification of thousands to millions of test permutations from minutes to hours.
  • Cheaper – It also helps bring a remarkable reduction in the expenses related to the production of best-in-class software.

Automated Testing Trends

Agile environment automated testing

Organizations have largely merged automated testing and Agile development due to major trends that have arisen in the last few years.

Let’s take a brief look at some of the trends as follows:

1. Mobile Testing

Mobile testing will continue to remain at the top of the list for automation trends, owing to the remarkable shift in the adoption of mobile technology. Mobile application testing will indisputably have a major impact on the software testing industry with functionality, compatibility, performance, usability and security testing as major areas of focus.

In the next few years, mobile test automation will match web test automation if the market for mobile continues to rise at the same pace. There’s a great possibility in the open-source framework, mobile development, cross-browser testing, and app markets to expand for the mobile test automation landscape.

2. Big Data Testing

With an increase in Big Data analytics, quality management for Big Data implementations will emerge. Applications, networks, and devices will generate a huge source of unstructured data, ultimately leading to faulty and costly decisions.

Testing quality ahead of product release will continue to remain critical. With this type of testing, businesses will aim to get hard numbers and statistical indicators on whether the application behavior will meet the demand of customers or not.

3. Testing with TDD & BDD

Enterprises are likely to adopt TDD and BDD to deliver applications as per the customer needs, without any defects or delay. By engaging in this technology, teams can get an actual idea of what is required.

As a result, enterprises are better off in terms of cost and time since the longer it takes to identify bugs, the more it costs to fix them.

4. DevOps

Alliance among IT professionals, developers and operation engineers has increased since the emergence of DevOps. More and more organizations are merging the role of testers with operational engineers and developers due to overlapping competencies in the continuous Agile cycles.

Experts predict that testers, developers, and operation engineers will similarly need to utilize the same tools to increase the collaboration. This technique will promote better communication and feedback among teams to achieve Agile development, testing, and delivery.

6. Micro Testing

Enterprises financing in Micro Services make it easier to test an independent piece of functionality without altering other applications.

This trend will inevitably rise in the coming years, as it certifies testing complex applications which comprise small, independent processes and architectures.

The Verdict

Automated Testing is considered by many software testers as one of the most forthcoming trends practiced in the overall software development lifecycle today. With the help of automated testing, one can get more effective and efficient testing results in an Agile environment.

About the author: Claire Mackerras is a Senior Quality Assurance Engineer & Editor at BugRaptors. BugRaptors is a CMMi5 certified company with extensive experience as a third party testing vendor for tools in Software Testing & QA. Claire enjoys creating, uncovering, and disseminating new and interesting perspectives on technology and software testing.

What’s the True Cost of a Software Bug?

By Leave a Comment

The Most Costly and Expensive Software Bug Screw Ups

Test early and test often — that’s what everyone says will help you avoid the high cost of deploying a software bug. But a bug might not seem like a big deal without a dollar sign attributed to it. Why invest in testing if you can just fix your mistake after? What’s the true cost of a software bug?

While there’s no set cost you can ascribe to a software bug found after the product release because it’s highly dependant on the organization’s size, customers, employees, and debugging resources, we can look at a few statistics and examples that show just how damaging it can be.

Starting with the big picture, software failures cost the worldwide economy $1.1 trillion in 2016. These failures were found at 363 companies, affected 4.4 billion customers, and caused more than 315 years of lost time. Most of these incidents were avoidable, but the software was simply pushed to production without proper QA.

Those may be stunning numbers, but you could still be thinking it doesn’t mean much to your specific organization, so let’s look at it on a smaller scale.

IBM found that the cost to fix an error found after product release was 4 to 5 times higher than if it’s uncovered during the design phase, and up to 100 more expensive than if it’s identified in the maintenance phase.

The costs go up as the bug moves through the SDLC. For example, IBM estimates that if a bug costs $100 to fix in Gathering Requirements phase, it would be $1,500 in QA testing phase, and $10,000 once in Production

And software bug expenses aren’t just related to the impact on internal operations. When a company announces a software failure, they lose an average of $2.3 B of shareholder value just on the first day.

The fact of the matter is that a software bug can affect everything from indirect costs like customer loyalty and brand reputation to more direct losses in business revenue and wasted time.

Companies Who Learned the Hard Way

You might think a software bug can only be so bad, or that any bug big enough to cause significant financial distress will be a complex mathematical error. In reality, some of the most detrimental bugs had minor mistakes that developers just didn’t catch because they weren’t used to putting the software through proper testing.

The following are companies who suffered monumental losses — some that almost put them out of business — due to easy errors, new software, or a bad line of code.

NASA – The Mars Climate Orbiter was a robotic space probe that NASA Launched in 1998 to study the climate and atmosphere of Mars. All was well until communication got cut-off a year into the mission when the Orbiter got lost in space. When dealing with spacecraft engineering, there’s a lot that could go wrong — it’s literally rocket science. However, the error could have been an easy fix if caught on time. The $125 million spacecraft presumably orbited too close to Mars’ surface and disintegrated because the engineering team failed to convert their measurements from U.S. units to metric. However, that’s not the only time a programming blunder destroyed a NASA rocket.

Knight Capital Group – As one of the biggest American financial services firms, Knight had a lot to lose, and in 2012 they lost $440 million of it. All it took was thirty minutes to drop 75 percent of their shares after an automated set of orders was sent out all at once to nearly 150 stocks instead of over a period of days like it was supposed to. The function, known as “power peg,” made stocks move up and down, lowering the stock price of all the affected companies. The loss was almost four times the company’s profit and would have put them into bankruptcy if a group of investors hadn’t come to the rescue.

AT&T – In January of 1990, AT&T failed to meet the basic needs of their customers as thousands attempted to call their friends, family, and airports. Long distance calling was down for nine hours, and customers sparked outrage after experiencing a total of 75 million missed phone calls and 200,000 lost airline reservations. Initially, AT&T thought they were being hacked, but the issue was due to a software bug. AT&T had just recently “updated” code, but the change of speed ended up being detrimental to a system that couldn’t keep up. AT&T lost an estimated $60 million from long distances charges that didn’t go through after being down less than a full day in an error that was the result of one line of buggy code.

EDS Child Support – In 2004, a computer system managed to overpay 1.9 million people, underpay 700,000, accumulate $7 billion in uncollected child support, backlog 239,000 cases, and get 36,000 new cases “stuck” in the system. This happened after a software company introduced a complex new IT system to the UK’s Child Support Agency. Meanwhile, the Department for Work and Pensions was restructuring their entire agency. The software was incompatible as the systems clashed, and as a result, the oversight cost taxpayers $1 billion.

Apple – For many consumers, Apple can do no wrong, but when they replaced Google Maps with Apple Maps on iOS 6 in 2012, they managed to put a pause on customer’s endless adoration. The update was nothing more than a product of corporate politics — Apple no longer wanted to be tied to Google’s application, so they made their own version. However, in Apple’s haste to release the latest and greatest map app, they made some unforgivable lapses in navigation warranting the #ios6apocalypse hashtag. Just a few of the many Apple Maps fails include erased cities, buildings disappearing, flattened landmarks, duplicated islands, warped graphics, and false location data. While an exact financial loss was never attributed to this lack of judgment, it made customers question whether Apple prioritized their needs over their money as they considered switching to the very competitor Apple was trying to combat.

An Argument for Agile and Continuous Testing

As technology evolves, so do our development methodologies and knowledge of testing practices. Getting stuck in the old age of software development means that crucial processes go ignored.

For example, while a Waterfall methodology used to be the go-to back in the day, high functioning software teams have realized that following Agile practices better support changes made throughout development. Agile’s incremental approach allows more feedback, flexibility, and of course, testing so that every time a feature, fix, or function is added or changed in code, it’s checked for bugs.

Since teams have adopted Agile and Continuous Integration, software is higher quality after a product release because bugs and inconsistencies are caught earlier and fixed easier. In turn, organizations have found that this helps avoid preventable bugs, save time and money, and retain their customers’ trust.

How to Prevent Your Next Software Bug

While we can’t expect to test everything and go our entire lives deploying a product that’s 100% error free, we can make strides to safeguard software as best we can. Many of the worst bugs that we encounter are simple mistakes that could have been caught with a more thorough quality analysis.

The moral of the story is to test your software. If you don’t, it’ll cost you.

For more proof of why software testing is so important, check out Why Testing Matters and what the top QA professionals have to say.

Establishing Your Scrum Team’s Definition of Done

By 1 Comment

Kanban scrum definition of done

When talking about building culture in the software development process, deciding on a team’s Definition of “Done” has become a major component in the shift to Agile — most commonly in Scrum — that ensures every player works together toward the same end goals.

For example, if a developer is working on a certain user story and tells the product owner they’re done but it hasn’t been tested, this will backup product release once the mistake has been realized, or potentially affect reputation if a buggy app is deployed without being caught.

While one person may be finished with their tasks, that does not mean the project is done as a whole, and saying so can limit progress if it’s not done in all parts and stages. Teams that do not consider this groundwork often find that it severely impacts productivity.

The Definition of Done for Scrum Teams

When Agile strives for iterative development, fast feedback, and adapting requirements, having limited communication will delay a team’s acceleration and pace when pushing a project or feature set to market.

Maybe you feel factors like testing, reviewing, documenting, releasing, or even collaboration should go without saying, but many team members will consider tasks done based on whether or not they fulfilled their individual role.

Because Scrum has lightweight management overhead and bi-weekly check-ins (Sprints), having a Definition of Done becomes even more important to your team’s efficiency. Nothing kills team chemistry and collaboration quicker than having tasks done “half-way” only because another teammate thought they only had to scope out a project, and not actually supply mocks or a rough draft.

Coming up with a Definition of Done is about the bigger picture organizational process of your team. By looking at projects as a collaborative effort instead of dividing responsibilities among different departments or one leader, you encourage the communication and transparency that’s necessary for Agile.

This is why creating a Definition of Done should be a starting point for Scrum teams. Your organization should essentially have a checklist of specific criteria and requirements that need to be fulfilled before the user story is done. However, the definition will likely differ from organization to organization based on the size of the teams, different fulfilled roles, and different project requirements.

Criteria to Consider When Defining Done

  • Prioritizing tasks – Having an order and process in place should be a prerequisite for defining done. This means having a set expectation for every time a task moves throughout your Kanban flow, whether something is selected for development or in code review. Also, this includes deciding whether a Definition of Done should just be attributed to user stories, or features and epics as well.
  • Including testing – Testing is often the most overlooked step. Is more manual or automated testing needed for this particular project? Are you using regression tests throughout development or just at the very end? Have you tested across multiple browsers? What processes do you have in place for analyzing risk and evaluating security?
  • Involving every role – If your developers have a different Definition of Done than your testers, for example, projects aren’t going to be done by everyone’s standard. This also includes product owners, project managers, scrum masters, designers, automation engineers, QA, or any other role. Furthermore, the definition should not be prescribed by one team member, but instead agreed upon by everyone.
  • Coming up with a timeline – How many iterations are you striving to make throughout the day? How many are you committing to at the end of the day? What are the goals during your sprints? How do you keep track of this and what happens if a task is not completed in time?
  • Ask what makes the process complete – Is it done when the product goes through a last round of testing, is it done after a last once-over by the product owner, or is it when it’s released to the end user? Having an end point is just as important as having a designated place to start.
  • Document the definition – Have the information easily accessible for people to reference, even if it’s printed out as a checklist. This way, everyone can double-check before they move a task into “done.” Additionally, in the same way, that Agile is adaptable, your Definition of Done should be, too. Make sure to return to the document and reassess processes that need to be adjusted according to changing requirements.

How do you decide when a project is “done”? Share your thoughts in our comments!

What is DevOps, Really?

By Leave a Comment

what is devops testing

DevOps: fundamentally, it’s the collaboration between Development and IT Operations. Okay, but what does that mean exactly? What is DevOps, really? Well, it depends on who you ask.

While DevOps is certainly the buzzword of the software industry, with popularity comes criticism, and many people seem to disagree on what its definition truly is. After extensive investigation, these are a few of the themes we’ve found most people seem to be able to agree upon.

The Age of Agile

Maybe a few years ago siloed contributors could get away with creating high-performing deployments when they were only releasing features every few months, but amid today’s frequent and faced-paced release cycles, having isolated departments just doesn’t fly. This, of course, is because of the massive shift toward Agile development.

Luckily, if you know a thing or two about Agile, you have a basic foundation of what DevOps strives for. In fact, in the Agile Admin, Ernest Mueller says that the best way to understand DevOps is to equate its definition in reference to the philosophies of the Agile Manifesto.

That is to say, if you want to build new features, get fast feedback, ship often, and ensure quality, you have to have a certain infrastructure to make that happen. This includes having values, methods, tools, and teams working in harmony.

We’d like to think that DevOps is pretty just a common sense approach to successful Agile —
“Since the Agile methodology was only implemented during the development phase, the operations department was left behind, leading to deployments piling up faster than it was possible to release them,” noted DevOpsGuys.

By bridging the gap between two departments that instinctively should work together but weren’t, deploying rapidly and efficiently became a more achievable goal.

The Dev + Ops Alliance

Maybe you haven’t noticed, but culture is really big deal in the software development and IT landscape.

It seems like the most successful organizations are putting culture at the forefront, and it makes sense. With so many moving parts and processes, having everyone on the same page is the only way to be productive and proficient.

When we say culture is this scenario, we’re not talking about open offices, holiday parties, and company perks. In DevOps, the talk of culture is more of an encompassment of communication, transparency, collaboration, and respect.

This also means ending the stereotype of Development vs Operations. While the skill sets and day-to-day focus of those in Development and Operations may differ, they should complement each other to act as a united front. There should be an investment in innovation and problem-solving across the board.

Additionally, the more developers are willing to improve their own operational knowledge and the more Ops is willing to learn about best practices in development, the more cohesive the whole workflow will be.

At the end of the day, DevOps is all about being able to work well together. The effect of that, of course, is more frequent integrations, faster deployments, and better results.

Defining What is DevOps

We always come back to this description:

“DevOps is the practice of operations and development engineers participating together in the entire service lifecycle, from design through the development process to production support.”

Like DevOps, Mueller’s explanation is broad. It leaves room for interpretation without limiting teams or defining what a role does or does not do, because that really depends on a lot of things.

People are never going to agree on one version of DevOps because it’s not consistent. You can think about it as a culture shift, an answer to Agile and Continuous Delivery, or simply just an effective approach to the software development lifecycle.

Some may say there’s no such thing as a DevOps team, while some may agree it exists and be tempted to leave the term behind forever because of ambiguity and popularity. However, if a team calls itself DevOps and produces great software, there’s probably something we can learn from it.

Maybe the more we try to define DevOps, the more complicated and controversial it’s going to become, and we should simply focus on the principles and the processes that are working — unifying siloed departments, collaborating often, and leveraging a diverse array of strengths.

What’s your definition of “DevOps”? Tell us what you think in the comments!

The Difference Between Agile vs Waterfall Development

By Leave a Comment

Agile vs waterfall development methodology
Waterfall and Agile are the two central methods of organizing software development. While Waterfall used to be the reigning method, Agile has become increasingly popular since “The Agile Manifesto” to keep up with consumers’ standards in today’s face-paced technological environment by ensuring continuous integration and continuous delivery.

To obtain a better understanding of Agile vs Waterfall, we’ll look at the differences between the two methodologies as well as which one is best for your development and testing teams.

Waterfall

In the most basic sense of the definition, Waterfall is a non-iterative or sequential design process.

With Waterfall, there are several distinct stages. These include requirements and analysis, design, development, testing, deployment, and maintenance that are completed in a linear flow where each stage must be finished before the next one can begin.

Those who still use waterfall might prefer the method because it requires a clear and concise plan from the beginning, ensuring that it’s easier to estimate timetables and associated costs.

Additionally, since the project has a more defined design, there’s usually a clear vision between customers and developers. To someone using a Waterfall methodology, a successful product always matches the initial plan without any surprises.

However, it’s rare that a customer won’t change their mind about a single feature or function throughout the entire development process. Agile supporters argue that Waterfall is extremely limiting in this way because it can’t keep up with changing customer needs.

The issue with Waterfall is that once a stage is completed, it’s almost impossible to go back and make changes. Consequently, it’s inflexibility does not serve as a practical option for teams that strive for continuous integration and delivery.

Not to mention, the high dependency on initial requirements leaves little room for error. If a requirement needs to be adjusted for any reason, you’ll likely have to start over from the beginning.

Agile

Agile is certainly a newer methodology that came to being by seventeen developers who created the “Manifesto for Agile Software Development” during a trip to a Utah ski resort in 2011.

Ultimately, they agreed that Waterfall wasn’t meeting their needs and came up with this alternative methodology that is known today as Agile to better support change throughout the development process.

Like Waterfall, Agile also follows a process, but its incremental approach allows more flexibility, feedback, and testing. The process includes a sprint planning session with smaller tasks and testing at every increment rather than just at the end before product launch.

This makes Agile the ideal environment for Continuous Integration, which requires team members to integrate their work frequently. In fact, CI was created for Agile development so teams could solve issues as they come instead of trying to predict and solve every issue up front.

Compared to Waterfall, developers find this often generates higher quality software because bugs and inconsistencies are caught earlier and fixed easier.

Agile also allows more communication between customers, developers, managers, testers throughout the entire process to ensure the product is moving in the right direction and adhering to the customer’s specifications.

Additionally, Agile can better accommodate the evolving requirements of customers because feedback is given simultaneously with development so that additions, changes, or complete feature removals can be easily adapted and tested without having to start over.

This ensures that as a customer’s vision changes over the time of development, it can be easily accounted for to build a truly successful product.

Tips for Transitioning from Waterfall to Agile

Perhaps the Waterfall methodology you’ve been using is working exceptionally well for your team and customers. If that’s the case, then keep doing what you’re doing. However, if you feel Agile may better suit the needs of your stakeholders, consider these points during your transition.

  • Culture and communication – One of the best things about an Agile process is that it enforces more departmental collaboration. However, this won’t come naturally, and the transition will likely require a culture change. It’s important to have that all the necessary team members to accommodate an Agile process, while transparent communication will become increasingly crucial for successful deployment.
  • Digging deeper into agile – As Agile is more of an umbrella term, deciding which specific Agile process you’d like to delve into is essential. The Agile methodologies include scrum, lean and kanban, extreme programming (XP), crystal, FDD, and DSDM. Learning more about the methods of each will allow you to tailor Agile to your projects.
  • Transition your customers, too – Your internal team isn’t the only one who needs to be updated. One of the benefits of Agile is constant customer feedback, but this means you’ll need to keep them in the loop. Make sure they are invested and committed to a more agile process and know what it requires, as they’ll probably need to be more involved in development.
  • Prioritize your tests – You can’t testing everything, especially in Agile. Executing a risk analysis will help your team understand the likelihood of bugs occurring and the impact features have on customers to determine how tests should be written and which ones should be included in the test suite. Assessing the priority of individual tests will ensure that the Agile process is an effective and efficient use of your team’s time.
  • Know that Agile is not a silver bullet – While Agile is becoming popular among software engineers being that it’s largely considered a more efficient organizational process, it will not solve all your problems and shouldn’t be expected to. Transitioning will likely require new hires and a shift in culture that may not always be smooth sailing, even if the ROI is optimal. Like anything else in the software industry, you need to examine the pros and cons of each method to determine which is best for your needs.

Do you follow a Waterfall or Agile methodology? Share your thoughts about them in the comments!

Join Over 200,000 Testers & Developers And Start Testing Today

Learn more Free Trial