Qxf2 reserves Round 2 of our three-round job interview solely to evaluate if the candidate can learn and apply new concepts quickly. During this round we use one of approximately fifteen exercises. This post goes into great detail on how an interviewer is supposed to conduct Round 2 when using the ‘Fork a repository and fix an error’ exercise. Interviewers can apply the general ideas in this post to the other exercises too. I wrote this post mainly for my colleagues. But if you regularly conduct QA interviews and are looking for fresh ideas, read on!

---

Contents

This post is organized as shown below

1. Background
2. What is the ‘Round 2’ interview?
3. Evaluation criteria
4. The exercise
5. How to conduct the interview

Warning: This is a long post because I am trying to condense a lot of tacit knowledge that we have accumulated into words. If you are an interviewer reading this, please try to simulate the interview in your imagination as you read along.

---

Background

As Qxf2 grows, more of my colleagues are going to be interviewing candidates. While this shared responsibility is good, I have noticed that our interviews have become inconsistent. The quality of the interview is heavily influenced by the interviewer. The interviewers are also not really taking the opportunity to implicitly give the candidate a glimpse of our work habits. I am taking time to document at least one Round 2 exercise in depth so it helps bring some consistency among interviewers.

---

The ‘Round 2’ interview

Qxf2 has interviewed hundreds of QA engineers. We consistently receive positive feedback from candidates about our second round. The round lasts 60-minutes and we describe it to candidates this way:

---

Evaluation criteria

Candidates might think that they need to complete the task in order to clear the round. But that is not true. As an interviewer, you should check if the candidate has a bias towards trying things out rather than just read references. You should also observe when the candidate asks for help – is it before trying a step, immediately after encountering an error or after having Googled the error, etc. Evaluate if the candidate is showing a sense of urgency based on their actions during the interview. The intentions behind the questions that the candidates asks should also be noted – are they stalling for time, are these genuine technical questions, are they indirectly complaining about the vague nature of the problem, etc. Take note of such stuff because they are critical indicators of whether a candidate can function well in a fully-remote company like Qxf2.

As you can imagine, this round often eliminates non-technical people who are too under-confident to even try things. Non-technical folks who at least try their best and are willing to fail and learn will clear this round comfortably.

---

The ‘Fork a repository and fix an error’ exercise

We want the candidate to fork a repository from GitHub and fix a simple Python error. That’s it! To do this, the candidate needs to sign up for a GitHub account, setup git bash (on Windows), setup Python 3.x, then follow part of a blog post to fork the repo and then finally follow this blog post that solves the error. As an interviewer, you are used to git and Python. So this exercise probably takes you no more than five minutes. But for someone with no background in git and Python, I think it is fair to give them about ~30 minutes.

Give this exercise to anyone who is not very technical and has nearly zero programming knowledge. The candidate should not have used at least one of git or Python. Do NOT give this to candidates who can code well or are technical.

---

Stage 1: Preparation

There are a few things you (the interviewer) needs to do to prepare for this round.

One, identify at least one item to either compliment the candidate or a question/comment that ties the candidate’s experience with yours. This shows the candidate that we did listen to them in the first round. Example: “I noticed you used HL7. I had a hard time learning the HL7 format. How did you go about learning HL7?”

Two, make sure https://qxf2trainer.pythonanywhere.com is up and running and that you have its credentials ready to share with the candidate. This is useful if the candidate wants help with installing Python, git or git bash.

Three, verify you have a timer plugin in your browser that is set to 30 minutes. You need to make sure you start the timer when the candidate starts sharing their screen.

Four, ensure that you have a bulleted agenda with a time and task breakdown for the round on VS Code (not notepad or IDLE or slides!) that you can show the candidate when setting the agenda for the meeting. This is a very important step that subconsciously communicates a few Qxf2 habits – we break tasks down, we time-box our tasks, our presentations are sparse, everyone here writes code, etc. See an example breakdown here but do not become lazy and copy-paste the example breakdown. Make your notes sound like you! Use your own language.

---

Stage 2: Problem statement

In this stage of the interview, make sure your candidate understands the task for the round. This stage takes between 5-10 minutes. Do NOT spend more than 10 minutes on this stage. As the interviewer, it is your duty to move the conversation along.

Here is my recommendation for how to go about this stage.

1. Introduce yourself and what you do at Qxf2
2. Remember the first thing you did in the preparation stage? Use that line now. Connecting with the candidate helps calm them down a bit.
3. Tell the candidate that you will spend the next 5 minutes outlining the structure of this interview.
4. Then, share your screen and go over the VS Code note that you prepared.
5. Before you give the candidate the exercise, please ensure they are not familiar with Python and git.
6. Confirm with the candidate that they have understood the exercise.
7. Paste the references and steps from your notes into the chat
8. Start the timer, stop sharing screens and ask the candidate to share screens

 

IMPORTANT: When you explain the exercise start with a one line summary that has technical jargon and then pause and wait for a reaction from the candidate. For example, you can say “In this exercise, you need to fork a repository, follow the setup instructions and then fix an error in our code. You will be interacting with git, git bash and Python.” … then pause. This is a critical moment when evaluating the candidate. You have finally introduced something technical that they do not know. Pay attention to the candidate and their body language and tone. Do they look physically uncomfortable? Are they silent? Is their body language defensive? Are they zoning out? There is really no incorrect reaction here. You need a lot more data to figure out if the candidate is uncomfortable trying something new. But observing their reaction is a good starting point.

---

Stage 3: Problem solving

This stage should last for 30 minutes or lesser. It should never exceed 30-minutes no matter what. During this stage, you are going to be mostly quiet and observant and answering questions. You are mostly reacting to what the candidate says. The only times you will proactively talk are to remind the candidate that they have 20, 10, 5 and 1 minute remaining. You should make calling out the time a habit. We want to emphasize that within Qxf2, we care about time-boxing.

---

Stage 3a: Start of problem solving

Give the candidate some silent time at the start. Usually, candidates will take 1-5 minutes to gather their thoughts and then ask questions. Many of the initial questions will be poor and just buying the candidate more time to think. That is ok. If they are silent, you remain silent.

Ten minutes into the problem solving stage, if the candidate has not even begun, nudge them to first get Python, git and git bash installed. Then handhold them a bit. We do want all our candidates leaving the interview thinking they at least did something.

Every time you answer a question, please tie it with at least one habit/scenario that happens for real at Qxf2. Example: if the candidate is hesitant to try out a vague instruction. Say something like “This is a very common situation in our daily work. In our target market of startups, it is normal to have sparse documentation. And given the timezone difference, we do not have people around to handhold us. We need to try things out for ourselves. Try out your first interpretation of the instruction. If that doesn’t work, you can try something else.”

Stage 3b: What to observe during problem solving

You are trying to put yourself in the candidate’s shoes and understand their thinking. Pay close attention to how the candidate uses their mouse, uses their keyboard, scrolls up and down a page, etc. Which sections of reading do they pause? Do their actions make it clear that they are used to Googling and installing software? If they installed software, did they seem careful or did they just zoom through the install steps without reading? Which steps are they stumbling or hesitating? Do their questions make any sense? Are they Googling well? On a Google Search results page, are they able to quickly pick out links that might help? Are they using the URLs you gave and trying to read them? If so, how are they reading them – sequentially or scanning? Are they blindly copy-pasting stuff without even making an attempt to read the pasted commands? Are they careful and double-checking their work? Did they use a search engine not named Google? If they used Chrome or Firefox, did you observe any technical add-ons? A combination of such small observations give us a better glimpse about the kind of colleague we will be working with.

Note: Scoring poorly on these observations does not eliminate a candidate. But a good score on these little observations give us some indications that the candidate is a good fit for us. For example, if a candidate uses a new tab to open a search result, you know they are used to Googling for harder problems, so hard that routinely Google’s first result does not always get them what they want.

Stage 3c: Good note-taking

All of the details that you noticed in stage 3b should be in your notes. Additionally, you should keep notes about the sequence and timing with which the candidate attacked the problem. For example, a candidate that spends 25 minutes reading and understanding stuff without trying is going to struggle in our environment. Please explicitly mention how far the candidate reached during the allotted 30 minutes. Add comments in your notes if the candidate took one approach and then decided to switch approaches when the first approach did not work out. How conscious and smooth was that switch? I also want to see if the candidate respected the time limit and tried to hurry up during the end. Paradoxically, candidates who get increasingly desperate near the time limit and try lower quality approaches are more likely to be a good fit for Qxf2. I think this is because it shows us that they are already in the habit of time-boxing their work.

Stage 3d: Ending the problem solving stage

Stop exactly at 30-minutes. Ending meetings on time is an important part of Qxf2 culture. So too is time-boxing. I want that conveyed through actions even in the interview stage. If a candidate asks for more time, say something like “That is ok. Most candidates do not finish this task. We just want to see how far you get and how you approach problems.”

Stage 3e: Example responses to common situations

Over the many interviews I have conducted, I have noticed that there are some situations that arise more often than others. Here are some responses you can use to handle these cases.

Some candidate will press you to spoon feed them. To resist, try some of the following counters: “What do you think should be done”, “Just do whatever you do at your job when you hit an error”, “Feel free to Google for the solution”, “I will definitely help but can you first try for a little longer?”

Some candidates will tell you stuff like “I am not used to this. So I might be slow” or something similar to ease their tension. You can reply with a non-commital reply acknowledging that you heard them. So a simple “I understand” or “That is fine, just do your best” is enough.

If the candidate finishes early, compliment them. Do not make them do more. Just note how much time they took and move on to the next stage.

If a candidate wants your feedback on how they did, your answer should always be “It was a mixed bag”. You can briefly explain what you think went well and what you think should have been better. Tie it to your work experience at Qxf2. Be genuine. Do NOT argue. Example: I thought you began poorly. You were looking for exact instructions. But once you realized that you were on your own, I think you did fine. I liked that you were trying/applying things as and when you were reading them. That is something we do at work on a daily basis.

IMPORTANT: If the candidate is truly clueless about the exercise, please switch to an easier exercise like reading code. It is better that the candidate gets at least something right rather than waste 30 full minutes doing nothing.

VERY IMPORTANT: Do not get patronizing or arrogant or use a bad tone. After all, every one of us struggles at work every day.

---

Stage 4: Ending the interview

In this stage, you are trying to read how the candidate felt about the challenge and their effort by asking clarifying questions. Ask the candidate to stop sharing screens and turn on their video again. First, remind them that this stage of the interview should take no more than 10-minutes. This is yet another instance where we are drilling in our habit of time-boxing.

Then, ask them exactly one question (and no other question!) – “So, how do you think you did?” and listen to the candidate. Record their answer in your notes. What we are trying to figure out here is if the candidate is showing a lack of self-awareness. While most people reply that they are pumped to do a hands-on exercise or feel happy that they exceeded their own expectations, you will sometimes come across candidates that are not very self-reflective.

Time permitting, tell them they can ask you any questions about working at Qxf2. You cannot answer administrative questions like CTC, etc. But you can give them a window into your day or work. You can answer these sort of questions without naming clients or the actual product. Just use generic terms like “I work with a company that provides monitoring for in-house patients …” rather than name your client or your product.

Some candidates will complain about the format. Listen to them and if applicable, tell them that we have been using this interview technique for years and most candidates do not see it the way they do.

At the end of the interview, remind the candidate that we will get back within 5 business days. Thank them for their time and hang up.

---

Stage 5: After the interview

Once you hang up, make sure to post your notes in our Interview Scheduler. The notes have to tell the reader how the candidate thought about things and how you actively probed them. It is not enough to just relate facts or answers verbatim. The notes should reflect an interaction between two human beings that were actively thinking and responding to each other. Your evaluation criteria will depend on whether you thought the candidate can learn and apply new concepts quickly. You should also talk about how much overlap the candidate’s habits have with Qxf2 habits. Make sure to back up your judgement either way. Hopefully, all the peripheral things that this post made you observe will guide you to make a better decision.

 

Thanks for reading!

---