Asking developers to do QA is broken – why anyone should own QA

Essentially yes. We are trying to move to a hybrid model where we can send excel workbooks to the customer for completion, and then directly import these into the system. This would get us ~80% of the way there.

One huge upside with configuration is that it can be copied really easily. If you have a model customer that is very similar to others, you can use it as a starting point and save 99% of the work effort.

The best outcomes are when people test what they are responsible for.

I think this article misses two important points on siloed qa teams

1. Qa doesn’t know what can be covered by unit or integration tests

2. Since they treat our code like a black box, they may create permutations of tests which which cover the same functionality

Maybe this is part of the draw of having a qa team. Feature coverage rather than code coverage. The downside is this can create a huge number of expensive to run manual tests which may be hitting the same code paths in functionally identical ways.

The tooling for automating manual tests of web apps is almost there: puppeteer, recording user inputs and network calls, replaying everything and diffing screenshots.

Since qa tests are tied to features and not code, There’s also the problem of having to run all qa tests even if you’re releasing minor code changes. My build tools are smart enough to return cached results for unit tests whose dependencies didn’t change, but there’s no equivalent for qa tests.

Communication is great until someone becomes unreasonable and doesn't want to do something. Trust but the chain-of-command must verify. Shouldn't need to, but it should be there as insurance.

Silver bullets. They don't exist.

Code review. Read the results of someone thinking through a process. Spot more than they will, simply by throwing more eyes at it. Actually fairly effective: getting a senior dev to cast even a lazy eye over everything gives more opportunities to discuss Why It's Done This Way and Why We Don't Do That and Why This Framework Sucks And How To Deal With It with specific concrete examples which the other dev is currently thinking about. But it's still easier to write the code yourself than review it, and things still get missed no matter how careful you try to be, so it's still just another layer.

Unit tests. They cover the stuff we think to check and actually encountered in the past (ie. regressions). Great for testing abstractions, not so great for testing features, since the latter typically rely on vast amounts of apparently-unrelated code.

Integration tests. Better for testing features than specific abstractions, and often the simplest ones will dredge up things when you update a library five years later and some subtle behaviour changed. Slow sanity checks fit here.

UI-first automation (inc. Selenium, etc). Code or no-code, it's glitchy as hell for any codebase not originally designed to support it; tends to get thrown out because tests which cry wolf every other day are worse than useless. Careful application to a few basics can smoke-test situations which otherwise pass internal application sanity checks, and systems built from the start to use it can benefit a lot.

Manual testing. Boring, mechanical, but the test plans require less active fiddling/maintenance because a link changed to a button or something. Best for exploratory find-new-edge-cases, but throwing a bunch of students at a huge test plan can sometimes deliver massive value for money/coffee/ramen. Humans can tell us when the instructions are 'slightly off' and carry on regardless, distinguishing the actual important breakage from a trivial 2px layout adjustment or a CSS classname change.

So that's the linear view. Let's go meta, and combine techniques for mutual reinforcement.

Code review benefits from local relevance and is hampered by action at a distance. Write static analysers which enforce relevant semantics sharing a lexical scope, ie. if two things are supposed to happen together ensure that they happen in the same function (at the same level of abstraction). Encourage relevant details to share not just a file diff, but a chunk. Kill dynamic scoping with fire.

Unit and Integration tests can be generated. Given a set of functions or types, ensure that they all fit some specific pattern. This is more powerful than leveraging the type system to enforce that pattern, because when one example needs to diverge you can just add a (commented) exception to the generative test instead of rearchitecting lots of code, ie. you can easily separate sharing behaviour from sharing code. Write tests which cover code not yet written, and force exceptions to a rule to be explicitly listed.

UI testing is rather hard to amplify because you need to reliably control that UI in abstractable ways, and make it easy to combine those operations. I honestly have no idea how to do this in any sane way for any codebase not constructed to enable it. If you're working on greenfield stuff, congratulations; some of us are working on stuff that's been ported forwards decade by decade... Actual practical solutions welcome!

That's my best shot at a 2D (triangular?) view: automated tests can enforce rules which simplify code review, etc. The goal is always to force errors up the page: find them as early as possible as cheaply as possible and as reliably as possible.

The machine can't check complex things without either missing stuff or crying wolf, but it can rigidly enforce simple rules which let humans spot the outliers more easily.

And it is amazing how reliable a system can become just by killing, mashing and burning all that low-hanging error-fruit.