Interview questions for QA managers

Which organization do you work for? How long have you worked there? Take a few minutes to read over what you just wrote and see if it makes sense to anyone else besides you?

What part can I explain to you better?

Couldn't agree with this more; especially:

>So echoing that scaling issue into your test documentation is a huge mistake.

and

> QA has a crappy reputation because it's slow, ponderous, and often not very effective. Maybe part of that is because people are updating overly-detailed docs all the time because other people have told them they should, rather than because they really need to.

I started my career in QA and moved over to Dev, then back to QA, and now back to Dev. One thing that frustrates me most as a Dev about QA folks that I've worked with is their insistence that the script is God, not the right functioning of the application to serve the customer/user. Often that comes down to these two issues.

Hmm. I find proposed answers to these questions somewhat... unexpected. Lacking, in fact. They focus so much on technical ability, and seem to ignore the human side of being a manager. Starting with the first one... "1) Let’s say you are the first QA manager joining our startup. What are the first three things you would do?" The kind of things I'd expect to hear here would never be of the kind 'write a test plan'. And I hardly believe anyone would be able to contribute a test plan right away, would they? I expect ability to write a test plan, or setup a process where it is created and managed, to be rather easy to check by just asking someone to do a test plan on an imagined scenario. The answer I would like much more would be: '1. meet the team', '2. understand the product, its vision, goals', '3. understand current dev/QA setup and code before I proceed'.

In other words, it seems to me that the first things to do should be to see where we are, connect with people rather than making things up. How can you propose first actions without at least basic understanding of where you are?

I like questions 5+ much better, but still these seem more suited for interviewing an experienced individual contributor, not a team manager.

How about setting up a team? How about hiring? How would you define and split tasks, assuming we agree on tasks? What kind of testing would you personally do, which would you delegate to QAs and which would you require to stay on the dev team? What would you do in situation X? (any difficult multi-optimization problem where people, technical and philosophical issues are to be considered)

I can imagine a great individual contributor to ace every single question on ths list, and then fail utterly when he has to manage even 2 people effectively.

I'm not aware of any specific reading... but I'll share a few more thoughts:

The key is to not make it a blame game. Every bug that is discovered on staging or production should (in theory) trigger a root cause analysis of some kind... because it means that one of the earlier processes failed.

If you have manual QA people, it's mostly just improving the QA plan and adding edge cases and domain knowledge, but it can also sometimes be that there are data bugs or integrations on production that are not on staging, slightly different on staging, etc.

With every bug there is an action plan for how to fix the bug, but that is separate from the knowledge that we gain about our system because the bug occurred. Maybe the fix is to get someone to manually clean up some dirty data that wound up on production due to someone forgetting to validate/clean input data ... that's great but from the QA perspective you might have learned that several of your steps failed.

So I think the main thing is considering bugs opportunities to learn about the system as a whole. The philosophy behind chaos monkey at NetFlix is that even a well tested, solid system needs to be resilient, so any opportunity to make your system stronger (regardless of the cause) is a good thing. In particular, any bug found before it hits production is a win overall.

I'd also add that it's important to let the knowledge flow back out of QA and into the product team, etc. QA people often end up becoming internal domain experts who catch lots of issues, but that is something that quickly exceeds what one person can remember/understand as a system scales, so organizational learning/practices pay off big.