Improving team ownership of testing

If you are an engineering leader that strongly believes testing should be a team effort but have struggled to make it happen, keep reading. We have practical tips that go beyond the usual advice of writing unit tests, improving code reviews and aiming for better test coverage.

Driving developer engagement and ownership in testing

Since 2013, Qxf2 have been testing early stage products. That's our niche. We are QA consultants for startups. Our approach evolved from owning testing to figuring out how to get teams to own the testing and quality of their work. Along the way, we realized there is a lot more nuance and depth to making this mantra work. Many of these tips come from the mistakes we made along the way. Here are some tips on how to get your developers to own testing.

Key phases for building team testing ownership. Derived from Qxf2 experience guiding teams on testing strategy.

Fig 1. TL;DR: A summary of steps to improve developer ownership of testing

1. Prepare before asking your team to own testing

As a technical leader, below are a few points to consider before encouraging your team to successfully take ownership of testing. We have listed these points to draw your attention to them. However, we haven't provided specific solutions, as the best approach will depend on your leadership style.

Preparatory steps for engineering leaders to improve team ownership of testing

Fig 2. Prepare to improve team ownership of testing

  1. Accountability standards: We notice that teams fail to own testing when engineering leaders don't enforce accountability. You should have a plan for holding people accountable. While your approach will evolve, it is important to have some baseline ideas. Ideally, you have to consider common scenarios like first failure, repeat offenders, and differences between junior and senior engineers, etc.
  2. Insights before measurement: Relying on verbal updates about testing from developers can be risky. Instead, look for behavioral changes that deviate from the norm. This will help you know if your team is on the right path. For example, we have found that teams that take ownership of testing typically have a spike in bugs reported, post-mortems and discussions around testing. We also observe that they begin to flesh out tickets better, speak up more during retrospectives, mention testing in their standup updates, 1:1s, etc.
  3. Measuring progress: If you are looking for specific measurements, choose low cost, high impact techniques that your team will actually implement. Qxf2 is a fan of using the colour coded Attributes-Components-Capabilites model to capture the evolving quality of the product from a user's perspective.
  4. Internal support for testing: Make testing truly a company wide effort - not just limited to developers. Ensure you get support from other job roles like your customer success team, DevOps engineers, product owners, etc. Each could contribute to testing in their own ways that align with their job roles, interests and perspectives.
  5. A plan for critical moments: Testing is going to be put on the back burner in several scenarios. For example, you are working under a really tight deadline or you resume work on an older piece of software with no tests or there are big changes coming to your product, etc. Keep simple plans ready for such scenarios.
  6. Reassessment criteria: Setup some reassessment criteria for yourself. These criteria are for your own clarity and do not need to be shared with the team. We have included this tip here because we have seen too many executives persisting with plans that simply don't work, without reassessing them.

2. Implementing your plan to get everyone in the team to test

Once you have completed the preparation phase, it is time for planning and executing the implementation phase with your team. This phase involves putting your prepared strategies into action and ensuring that everyone is on board with the plan. Here are some critical components to focus on during the implementation process

Implementation steps to improve team testing ownership in Agile teams.

Fig 3. Create and implement a plan for stronger testing practices across your engineering team

  1. Identify what to test: Work with your team to identify which tests you will write. Do not fall into the trap of simply starting with the latest tickets or areas you plan to refactor next. Your testing time is limited and should be used where it has the largest impact. We recommend sitting with your team to answer questions like:
    • which areas routinely have client facing issues?
    • what areas are hard to test on a developer machine?
    • where do you expect the most incremental changes to happen?
    • any routinely brittle areas?
    A common mistake: Usually when a development team answers these questions, they try to solve the problem by changing code. Resist that instinct. Instead, invest your time to putting tests around these critical areas.
  2. Set realistic testing goals: This is a broad topic, but we will give you a brief overview of our perspective. First, here are some testing goals and approaches that tend to fail: focusing solely on the number of tests, overemphasizing one type of test (e.g.: unit tests), and writing tests only for the newest tickets. To simplify greatly, we believe most companies progress through these maturity levels in terms of test coverage. Organization maturity levels when it comes to testing. Qxf2 has provided QA consulting for several startups and observed this pattern.

    Fig 4. Evolution of test coverage at companies

    For early stage products, we recommend that you aim for 'multi-layer sparse' test coverage level. A 'multi-layer' defense means adding very few UI tests, some API tests and some regular exploratory testing to your existing unit tests. But it also means a whole lot more. There are other things for your developers to work on - like creating synthetic data for repeatable tests, virtualizing services to enable developers to test on their own machines, data quality checks, infrastructure tests, contract tests to decouple teams, alerting and monitoring, etc.
  3. Choose a testing stack: Modern testing stacks can be hairy and hard to select. To keep this section short, we will say that you minimally need to consider if the testing stack is being chosen for a team or multiple teams. Stay away from shiny new objects. When a tool is chosen, figure out if it was chosen because it was easy to get started with or because it has legs. Do not build frameworks before writing enough tests. Qxf2's testing stack as of 2023. We test early-stage products and need a versatile testing stack.

    Fig 5. Typical toolbelt that Qxf2 QA engineers work with (as of 2023)

    Well, we have so much experience on this topic that we could write a book filled with counter-intuitive insights. If you do need help at this stage, just write to [email protected] and we will be happy to share our expertise.
  4. Coaching your team on testing: Work on increasing your team's testing IQ. Unsurprisingly, developers rely heavily on writing code and automating tests. But they often spend time writing tests that do not matter. While that is better than no tests, you are not getting the most out of the time developers spent on testing. We find it effective to have continuing discussions around thinking critically about testing. We have a starter list of such topics that we customize to suit the teams we work with. Reach out to us if you need ideas on what topics to discuss.

3. Risks when you want your entire team to own quality

All strategies come with risks. We have a large list of risks. To keep things simple, we have listed the top 5 risks you need to be aware of when pursuing team ownership of your testing.

Risks when testing is a cumulative responsibility. When everyone is supposed to test, things slip!

Fig 6. Risks when you try to make testing a team responsibility

  1. Conflicts among developers: Conflicts are a natural part when introducing new responsibilities like testing. Your approach will depend on your leadership style, but it is crucial to be transparent and honest about the challenges and reasons behind the changes, as this builds trust. Encourage collaboration by fostering an environment where team members feel comfortable sharing their thoughts on testing and improving testability. This can prevent conflicts from escalating and turn them into opportunities for team growth.
  2. Unpleasant surprises: Unpleasant surprises can occur during and after testing. Quality of testing is inherently hard to verify, and a false sense of security can develop as more tests are added. Be wary if there are no negative updates, as this can be a red flag. On the other hand, team members vocalizing problems is a positive sign, indicating they are engaged and proactive in addressing issues.
  3. Excuses: Some team members may know how to avoid testing responsibilities or shift them onto junior team members. Be prepared for this and hold people accountable. Establish clear expectations and consistently enforce them to ensure everyone participates in the testing process.
  4. Bystander effect: The bystander effect occurs when everyone is responsible, leading to no single person taking accountability and leadership. To address this, acknowledge and reward proactive behavior, rotate responsibilities to ensure everyone is actively involved. This can prevent tasks from slipping through the cracks and ensure accountability across the board.
  5. Team composition: Team composition can pose challenges. Junior members might need guidance as some skills take time to learn. High attrition can disrupt team dynamics, making it hard to integrate testing when team members change frequently. In high-growth teams, it is essential to modify onboarding processes to include testing responsibilities from the outset. These changes in team structure and onboarding can help overcome challenges and smoothly integrate testing into the workflow.

4. Other testing options for early-stage products and teams

When things go bad, we see many companies try to tackle the problem by adding surface level testing. This is usually in the form of hiring a relatively inexperienced tester. The problem is much deeper and involves getting the team back on track to owning testing and quality. Qxf2 usually goes through a more thorough but quick 5-layer analysis to arrive at a suitable testing strategy. A lot of our work in such situation involves setting up critical defenses, improving testability, selecting the right technical stack for your team and then enabling your developers to write tests that matter. You could consult with us or hire a senior tester and give them the goal of getting your team to own the testing.

Note: Unlike many software testing providers, Qxf2 focuses solely on testing early-stage products. We do one thing and we do it well - that is provide testing services for startups. If you are a startup or working on an early-stage product that is gaining traction, we have several customized QA service offerings that you might not be aware of. The offerings are tailored to be front-loaded, fractional and involve options that are not usually available in the market. We encourage you to visit our homepage to learn more about our expertise in startup QA and how we can help you.

Closing thoughts

This article was written by Avinash Shetty and Arunkumar Muralidharan. Our understanding of getting teams to own testing has accumulated over the years. One article cannot capture our ongoing journey of learning and improving.

Testimonials for Qxf2's testing showing our evolution. Qxf2 provides QA consulting for startups and early-stage products.

Fig 7. Testimonials from Qxf2 clients showing our evolution in testing approach

We have had to leave out a LOT of what we know just to keep this article to a reasonable length. So if you want us to elaborate on anything in this article or want to share your own experience on the topic, please reach out to Avinash ([email protected]) or Arun ([email protected]).

paper cut