Modern testing for modern stacks

Read this article if you are an engineering leader (team lead/manager/director/VP/CTO) who is having a hard time getting your QA team to write automated tests. Your QA team is not the only reason you don't have automation. You, as engineering management, play an important role too.

Here is why you don't have test automation

Qxf2 has helped many startups with their QA needs. In this process, we end up talking with a lot of companies who are struggling to kick-start their test automation. Below are some of the main reasons why we feel companies still don't have QA automation and how they can be addressed.

1. You don't frame the introduction to automation well:

The existing QA feels threatened. The decision to introduce automation is done from a business point of view. The initiative to automate tests is not broken down into what it means for each QA engineer. The existing QA don't know how the change affects them and feel threatened. They may try to resist the change and things wouldn't move.

How can you get better: You can introduce the decision to automate tests by speaking to each individual on what they would gain. From our experience, the most popular benefits from a QA engineer's perspective happens to be learning a new skill set, using their existing technical skills, career progression and a break from their existing work. Once you (and not the QA manager!) have spoken to your QA engineers, pay attention to the first few automation-related tasks you assign to people. The first few tasks should help the QA engineer build momentum and confidence in their own technical skills. The difficulty of the initial tasks should depend on the employee's existing technical exposure, experience, attitude and ability.

2. You don't start with a useful broadstack test:

Instead, you pick a bad starting point, like developing a framework first. We have seen team overthink what framework and language to use. Since these take some time you slowly think it would take a larger effort to automate.

How can you get better: Instead, agree to be nimble and start with a broadstack test. Focus on making your broadstack test run with every build and get everyone into the habit of using the test. This way, you have some test which will be useful and people start using it. Once a single test is up and running, you can then spend time using it to evaluate the different options for frameworks and language.

This approach should hopefully correspond well with the way you develop your product. Whenever you build a new piece of software you start with an MVP and then continuously build it over time based on user feedback. Treat your test automation code also in a similar way. Start off with something useful and then develop it over time based on your products testing needs.

3. You overestimate the value of test automation:

We think VPs of engineering spend too much time talking about automation and not enough time talking about testing with their QA teams. We know you don't think of automation as a silver bullet but by talking about disproportionately, you give off the perception that you think automation can solve all your testing issues.

How can you get better: Mind the ratio of talk time you spend on testing versus automation. There are a lot of areas where testing is more valuable than automation and, where applicable, you could point out the specific instances out to build trust with your QA team. For example, you can occasionally ask your QA team about the most interesting bugs they found recently. Or you could ask your developers about interesting bugs they fixed and then try your best to understand how the QA team discovered the issue. Identify areas you cannot automate and instead rely on good QA engineers. A simple "nice-catch!" or "I'm glad someone is looking at this closely" goes a long way.

4. You don't give automation enough time to succeed:

You will slowly start noticing there are some bugs which your automation has failed to catch which your testers would have normally caught. This is because the tester would have noticed bugs other than what is mentioned in your test plan. You will also notice there is a lot of test automation maintenance effort involved in each sprint.

How can you get better: Have patience and give it enough time. Your automation tests and framework evolve over time. You can gradually reduce the maintenance effort once your framework evolves and testers are more comfortable automating stuff that is important.

5. You don't give enough development and DevOps support:

You feel your testing team is in charge of automation and don't give enough support. But all testers have gaps in their knowledge. It could be absurdly simple things like not knowing how to set up a cron job. Your testers will need some peripheral support to develop automation.

Eg: They may need some help in setting up Jenkins for Continuous Integration on a server.

How can you get better: As the head of engineering, you need to make it clear to your developers and DevOps teams that they need to treat QA asks as a priority. In our experience, the requests QA make are usually simple and not very time consuming. So, when your QA team begins to tackle test automation, you can ask explicitly your developer or DevOps to sit with the tester and help them out in setting things up.

6. You haven't staffed enough:

We work with startups and have observed this pattern often. The engineering priority is always shipping the next release. Your team would be busy working on getting the release out of the door. None of your testers would have time for automation and you don't know how to get it started.

How can you get better: This may be ok in some cases. But if you are convinced you need automation, you need to plan well and spend time on how to start automation. You may need to hire more people so that you are able to squeeze in some time for everyone to start automating some tests. You may also need to reserve one day of the week (or two days a sprint) for automation and nothing else. We find testers (at least in the beginning) do best when they have regular, undivided chunks of time to write automation. Don't worry about this becoming a habit.

7. Not enough business incentive to offset the retraining cost:

We have seen this in slightly older products that have found good product-market fit. Your existing QA does a great job testing your product and you barely hear of bugs from the field. You don't see any business incentive to train them to get some automation going.

How can you get better: This may be ok in some cases. It really depends on the business. But if you really want to transition your QA team from purely manual tests to automating more, the one thing you can do here is maybe to change your hiring pattern. Probably look for people who have some automation experience when hiring a new QA engineer.

These are some of the reasons why we feel organizations don't have test automation and what engineering management can try to increase the odds of establishing test automation. In case if you have any comments or suggestions, email Avinash ([email protected]) or Arun ([email protected]).

paper cut