Asking developers to do QA is broken – why anyone should own QA • Hacker News Board

Re: Asking developers to do QA is broken – why anyone should own QA

#91

Post by tpmx » Thu, Jul 01, 2021, 2:26 AM UTC

TameAntelope wrote: post #70

tpmx wrote: post #2

I think product owners should do (manual) QA. On their own, and when necessary for scalability reasons also via a small team under their direct control.

They don't have time, so they'd have to scale it, and once you scale it you basically are building a siloed QA team anyway.

Well, then they are doing it wrong.

Or maybe "my" process only scales so far...

It worked well for me at this scale: 30 developers, 10 QA people, 3 ux people, 2 product people "vs" 50 sales people.

There was also a separate delivery/consulting team of about 50 people which was important early on for revenue.

Re: Asking developers to do QA is broken – why anyone should own QA

#92

Post by salawat » Thu, Jul 01, 2021, 2:27 AM UTC

nullspace wrote: post #5

tpmx wrote: post #2

I think product owners should do (manual) QA. On their own, and when necessary for scalability reasons also via a small team under their direct control.

Yeah, this is the only thing that practically worked, in a complex product I used to work on. - If you ask the devs to do QA, you'll get no bugs other than the ones they already caught during testing and deployment. - If you have a mostly independent QA team, they will find somewhat silly/trivial bugs like the login page not working in an extreme edge case scenario. - However, when you ask your Product team to own QA…

>However, when you ask your Product team to own QA, you get the real good stuff - "Why does this feature not actually work well with this other feature when you combine the configuration" etc. It's great!

This works right up until you hit a complexity point product can't handle, or worse, you find out that product is building something completely different from the requirements SME's are giving.

Your QA group (or misguided product group) can do jack squat about badly translated reqs or the right questions not being asked, which in the presence of the best devs and QA's degenerates into building the wrong thing perfectly, which has to get redone over and over again. A good QA that's been around the block is usually pretty good at sussing out major communication defects, but it can be tough to pull out of.

It's a surreal class of miscommunication to behold, and as close a death spiral as you can end up in, because everybody day in day out is working hard and getting frustrated but no one's getting closer to the end goal.

Re: Asking developers to do QA is broken – why anyone should own QA

#93

Post by notmyfuture » Thu, Jul 01, 2021, 2:27 AM UTC

regularfry wrote: post #29

Devs doing QA worked fine for us. You need to have a team that actually cares about the product, and you need figurehead devs - not necessarily seniors or team leads, but charismatic people others will fall into line with - who model the behaviours you want. The problem with almost anything else is that it increases the number of hand-offs between groups of differently-aligned people on the route to production. If yo…

A person doing manual testing, which is a gate to deployment would certainly be hard to make work for deploys as fast as you're targeting. It may still be valuable to have someone separate to the devs testing your product (in production, not as a gate to release) looking for things your automated testing, customer feedback or metrics may have missed. Whether this would be valuable is really context-dependent.

Re: Asking developers to do QA is broken – why anyone should own QA

#94

Post by sethammons » Thu, Jul 01, 2021, 2:32 AM UTC

bilalq wrote: post #23

In a model where developers adhere to a devops philosophy and are product owners, there's no need for a split. Developers should not be measured only quantity of releases, but also on metrics related to availability, customer impact, latency, operational costs, etc. I'm not opposed to a model where non-technical roles are empowered to define test cases and contribute to approval workflows of a release pipeline, but t…

This is how I've worked for most of the last decade. Empowered devs that are accountable for their releases and supported by metrics related to customer experience want good quality releases. Ownership works.

Re: Asking developers to do QA is broken – why anyone should own QA

#95

Post by trynewideas » Thu, Jul 01, 2021, 2:33 AM UTC

From a couple different perspectives, docs and support also love (or should love) dedicated QA as a last line of defense against product changes that engineers don't consider worth documenting.

Every single release in an org with no dedicated QA and dev ownership of testing, I've seen something a dev has changed in the product that developers deemed weren't worth documenting, and every single release a user hits that thing and support is left not even knowing the change happened and looking like they aren't the product experts that sales told the user they were.

But aside from the dedicated viewpoint change of a QA analyst vs. an engineer, it's also a different relationship with other teams. Whether QA is part of engineering teams or siloed, it's usually a little easier for a tech writer or support engineer to pitch process improvements and regular communication to QA than developers. It's not about capacity, but value — adding a test to the product that confirms a code example in the documentation is accurate helps the product and the writer, so someone with a QA-first mindset doesn't have to think twice. But to a developer with a velocity-first mindset, it's busy work (shouldn't the tech writer own that check?) and maintenance load (isn't this another low-value test I have to manually update every release?).

Tests should be a form of docs for what's supported in the product. If a feature isn't tested, documenting it for users is a risk. If the tests don't confirm what's documented, it's a risk. If it's a risk, it's not the devs who'll deal with it first when it breaks, it's support. Having that healthy relationship between documentation, support, and testing is a force multiplier, and it's harder to build that relationship when development priorities like velocity conflict with testing priorities like coverage and accuracy.

Re: Asking developers to do QA is broken – why anyone should own QA

#96

Post by theptip » Thu, Jul 01, 2021, 2:56 AM UTC

xyzelement wrote: post #77

There's a little bit of an insane assumption in this pitch that PMs should be writing tests because they are the ones that "care a about quality." With my engineering hat on, I hate this idea because the message that devs don't care about quality (but PMs do) is not how I want to operate. With my PM hat on, I hate this because having the PM do test automation has got to be the lowest value activity you can ask the PM…

A problem with having engineers own the tests is that if the engineer misunderstands how the feature is supposed to work, then both the code and tests will be wrong, but because they are consistently wrong the tests will be green. So the product owner still needs to do ad hoc testing to verify that things work as expected.

I like the “three amigos” approach from BDD as a solution here.

The product owner writes the initial acceptance criteria, ie specifies the product. Then sits down with an engineer and tester to edit the ACs into sensible Gherkin so they can be executed as tests.

The tester goes off and builds the tests (implementing the Gherkin steps) and the developer builds the feature. At the end you bring them together and if the tests pass, you are quite confident that you met the requirements. (If the error is not obvious you all huddle and figure out where the misunderstanding was).

The important thing here is that everyone gets together and makes sure they share an understanding of the feature. You could skip the Gherkin and write the tests some other way. But having the product owner think through the process of specifying the acceptance criteria cleanly is a good exercise, I think. Now I don’t think that means they need to own the tests… but there is a case for it not being the engineer that’s implementing the feature, for sure.

Re: Asking developers to do QA is broken – why anyone should own QA

#97

Post by ukd1 » Thu, Jul 01, 2021, 3:02 AM UTC

sytse wrote: post #79

ukd1 wrote: post #78

Earlier quoted context omitted.

Some of QA is technical needlessly; we think that's a common issue with tooling today. QA today is just not accessible to folks that think about and manage change to the product, namely product folks. Why don't you think it should be part of product? Love that your department is called "Quality", implies more metrics-less-feeling before reading the doc.

Thanks Russell. In our experience a lot of QA improvements require automation which is more of a software engineering job than a product management one.

I think this is mostly true today, but something which we think of as broken and we're fixing. We believe it should be accessible to a wider audience outside of just engineering; there is a lot to do, but we're starting with UI automation.

Re: Asking developers to do QA is broken – why anyone should own QA

#98

Post by protomyth » Thu, Jul 01, 2021, 3:03 AM UTC

regularfry wrote: post #74

protomyth wrote: post #40

Dev sometimes aren't the best QA because they think like developers and not like end users. The mindset prevents you from doing things on the corners that end users do. Its like your instincts kick and a keep you safe without the railing where an end user might plow ahead thinking their path is ok and fall off the edge. Devs should do automated unit and functional tests, but after that, get some good QA that do not h…

This can absolutely be learned, enabled, and encouraged by the right guidance and, if necessary, training at the dev-team level. It's just part of the job.

I really don't think any Dev team will be as good at QA as a dedicated QA team. The mindset is very different and the context switch is hard. Checking your own work is also not the greatest of ideas.

Re: Asking developers to do QA is broken – why anyone should own QA

#99

Post by musingsole » Thu, Jul 01, 2021, 3:15 AM UTC

theptip wrote: post #96

xyzelement wrote: post #77

There's a little bit of an insane assumption in this pitch that PMs should be writing tests because they are the ones that "care a about quality." With my engineering hat on, I hate this idea because the message that devs don't care about quality (but PMs do) is not how I want to operate. With my PM hat on, I hate this because having the PM do test automation has got to be the lowest value activity you can ask the PM…

A problem with having engineers own the tests is that if the engineer misunderstands how the feature is supposed to work, then both the code and tests will be wrong, but because they are consistently wrong the tests will be green. So the product owner still needs to do ad hoc testing to verify that things work as expected. I like the “three amigos” approach from BDD as a solution here. The product owner writes the in…

> The important thing here is that everyone gets together and makes sure they share an understanding of the feature

So true. Communication is everything, but because it's hard it gets short circuited everywhere. Where I work, devs write their own tests. It's as terrible as you can imagine while still having a successful product. I once suggested we split to a designer/verifier model...and got the weirdest blank stares. I honestly think it's because no one I work with wants to talk to each other.

Re: Asking developers to do QA is broken – why anyone should own QA

#100

Post by edgyquant » Thu, Jul 01, 2021, 3:36 AM UTC

failwhaleshark wrote: post #46

MattGaiser wrote: post #33

Why can’t the QA people just live on the dev team? Why do they need to be siloed away or not exist? I had this in my past job. Have 1 QA per two developers, the QA sits with those two developers, and you are constantly telling them when a build is done and on staging. They write tests, do some manual checks, and then tell you how it is going relatively immediately. They also handle the problem of reproducing bugs and…

Terrible idea. The dev team manager is focused on growing new features in the next release. QA is focused on the excellent of the current release. Subordinating quality to new releases is the result. In a similar situation, devops supposedly addressed similar tensions of dev and ops.

I work at a startup so we have a small team. But for us it goes:

1. Develop some feature (me and the other engineer are full stack)

2. Send to other engineer for code review

3. Deploy to a staging site, with test data. Send to product girl to QA

If she finds a bug (or realizes her design doesn’t look as right in practice as she envisioned) she sends back with notes, we implement or fix them, and then start back at 1-2.

I think QA is different for every org but I believe there should be steps. At bigger orgs there should be QA for every feature developed (so each team should have a QA person) and then an integration QA of the current beta.