Hiring: QA screening

This post is a small bonus chapter on top of the phone screening narrative.

So far, I've concentrated on pre-filtering developers, and mentioned test specialists very (too) briefly.

Let's take QA then, and ask again

What qualities are we looking for?

They are not that different from developers, with only coding making way for testing:

• Can they find bugs?
• Is there a large discrepancy between what they think they know and what they actually know? 
• Can we work together, i.e. is there a communication problem?

And going a little bit beyond the land of the utterly obvious, and zooming in upon bug finding:

• Is there relevant industry background (e.g. hardware appliances, or financial systems - depending on what we do)?
• Do they understand system interactions?
• How about intuition and ability to poke holes in software (including corner cases)?

The last one is the absolute must, so let's expand a bit:

Poking holes in software

Like with developers, here I tend to pose a simple scenario, and see how far we get. For the aesthetic sense of symmetry, I can ask to provide test cases for the same function that was brought up in the developer coding interview.

Now, the good thing is that with test cases the sky is the limit. It is possible to raise just the obvious ones, and yet, it is also possible to show full expertise while staying within the confines of the original task.

Let's take an example: prepare test cases for a function that reverses a string. If you like, you can take a small break here, and think what test cases you'd come up with.

The first checkpoint are the requirements. Note that I never mentioned what should be done when reversing empty or NULL strings: should the function return the same value, throw an exception or assert? It was very rarely that I was asked questions about edge cases, but on those rare instances it was a major bonus for the interviewee.

Now, let's move to the basic functional test cases; there's the very obvious ones:

abc→cba
aba→aba

and slightly less so:

a→a
bbb→bbb
cC→Cc

Nevertheless, that was still the boring bit. For more excitement, let's turn to speed complexity:

• Does the function perform quicker on palindromes?
• What is the speed on huge strings; how does it scale with size? 
• Does it perform better on multi-core machines?

Of course performance does not finish here, as we also have memory considerations: 

• Does the function swap characters in place of uses a new copy?
• What is the practical input size before we start swapping on typical hardware/VM?

However, we can go back to the functional tests, as we're not done there yet - we need to consider localization and special character sets:

• What is the behaviour on UTF-16, UTF-8, UTF-32?
• How does the function deal with special characters, such as newlines, and NULLs?

If we really want to be persistent about it, we can throw concurrency in, and consider what happens if we modify the string in parallel to calling it in a different thread (though you might argue that we cross into Software-Engineer-in-Test land here).

It's quite possible that we can squeeze out more, but if a candidate reached up to here, then they should already get the top prize: they know non-functional testing, exploratory testing, localisation, and concurrency (spoiler alert: never happened to me).

In any case, the intent is not to declare this exercise as the only way to filter candidates, but to demonstrate that with even the simplest task we can go far if need be. 

Do they understand system interactions?

Ability to recognise risky interactions is one of the main skills for test professionals. Ideally, I'd like to describe a system and ask what we should be worried about at a very high level. The problem, as always, is time - we can't devote more than 10 minutes, and that's not a lot of time to describe a system and discuss test strategy.

For this reason, I usually left this until face-to-face stage, but from time to time, if the previous stage went well, I'd describe a hypothetical system that relates to the person's previous experience. For example, if they have SaaS on their CV, I'd outline a licensing system that contains entitlements from different subscribers, and ask what typical faults they would explore. It's important not to overdo it, as such a Q&A can easily take up the rest of the call, but when it works well, it kills two proverbial birds with one stone: both establish SaaS credentials, and check ability to view systems from above.

CV/skillset discrepancy - still applies

Now, what I mentioned last time about CV/skillset discrepancy still applies. If someone scatters different skills in their CV, they're always fair game irrespective of what we are looking for. For this reason, I've been asking about automation, networking, security, Java and everything in between. As with developers, this works both ways - it can highlight important experience that could complement the team, or it can show CV inflation. Usually, I would not spend too much time on this: just a couple of questions per topic to do broad and shallow coverage.

In fact, everything else in the previous post also applies. We need to figure out if we can work together, so I might go for an open question that does not have a single answer, and see whether we can have a reasonable (but short) chat. To take an example at random: is it fine to concentrate on component testing only with microservices?

Also, this should be a non-confrontational experience for the interviewee, so I'll similarly present the role, the company, and let them ask any number of questions within reason.

Summary

Phone screening for QA candidates is not vastly dissimilar to developers, with the main accent shifted into ability to think of failure modes, and system interactions. At least one practical question during the call is an absolute must, and even the simplest test tasks can go very far.