Ashley Hunsberger is a Test Automation Architect at Blackboard, a virtual learning environment and course management system for top institutions worldwide.
Her Selenium Conference 2017 presentation, “Transformative Culture” reviewed how Blackboard established departmental goals, built teams that would ensure quality and success, and slashed execution times, while transforming the way the company thinks about software quality.
Not only was this presentation a favorite among conference attendees, it also serves as highly valuable information for any company that’s undergoing a culture change.
A Cause for Change
A year ago, it took an hour to run Blackboard’s 400 tests, and 370 were unreliable. So, not only was the time to execute high, but the builds were constantly failing and developers had stopped caring about them.
Hunsberger quickly realized that the root of the issues they were having centered around culture and the way different departments thought about quality and communicated with each other to deploy projects.
She wanted to make a company-wide change that would enable the development team to own testing and quality and hoped to get the entire organization to put more care into test suites and maintenance.
The very first step to transform the department was a simple one. They would not longer be referred to as QA, but instead Engineering Productivity.
Hunsberger also wanted to re-establish the department mission to “Reduce the time from concept to deliverable by providing our product development teams with the tools, practices, and support to increase their productivity while maintaining high-quality standards.”
She also wanted to clarify that Engineering Productivity would not take over writing tests for other teams, but instead be a resource to support the automation strategy.
The next move was to set goals including:
- Provide an easily maintainable and extensible framework that would enable scrum teams to add and remove tests.
- Enable the automatic and early detection of failures within the software under development.
- Prevent the source of detected failures from moving any further downstream.
- Accommodate all of this without impacting the engineer’s time.
Enter, Team Shangri-la
After realizing this transformation would only be possible with a proper leadership team to own the automation strategy, Hunsberger introduced Team Shangri-la, where she acted as the project owner alongside a scrum master, SETs, a feature engineer, and a CI/CD team member.
Team Shangri-la had to first establish their definition of “Done”, as Hunsberger found that in the past, a lot of teams were saying projects were done without them actually being testing, reviewed, documented, and released.
They also had to outline test suite definitions to get everyone speaking the same language:
- Goals – What was the intent for each suite?
- Trigger – What would trigger tests?
- Gates – What happens if tests fail?
- Requirements – What are the requirements for each suite?
Once the guidelines were determined, they had to work on containerizing environments for faster feedback and implementing project guardrails to provide support for nine different scrum teams.
Additionally, one prominent area of Shangri-la’s strategy was risk analysis, where they aimed to prioritize what tests would be executed and how they would be written.
By rating the likelihood of a bug occurring and what the impact on customers would be if the feature didn’t work, it became easier to evaluate and what areas need to be included in the test suite and which did not.
Charting and tracking the results also helped the team decide what kind of testing needed to be executed, as well as how the tests would be monitored and maintained.
Hunsberger and her teams saw the 33 stable tests and 370 unreliable tests that ran in an hour turn on its head to now run 165 stable tests in 30 minutes.
Along the way, Hunsberger learned that not everyone has the same idea about what’s critical, and you can’t test everything. These small realizations ultimately initiated a complete culture change that encouraged them to be better communicators so that they could establish goals and meet them.
By focusing departments to be a company-wide effort, Hunsberger was able to affect better testing to support development, optimize testing, and ensure uncompromised quality across Blackboard’s projects.