The best outcomes are when people test what they are responsible for.
What evidence do you have for this?
Asking developers to do QA is broken – why anyone should own QA
101–110 of 193 posts
Re: Asking developers to do QA is broken – why anyone should own QA
#102Many years ago I asked a developer which was a bigger deal: - Having an outage - Missing a deadline They answered: "oh having an outage is WAY worse". I then asked: "if that's the case, why do you push so hard to hit your deadlines with code you know and I both know is probably not ready?" They didn't really answer at the time but it eventually dawned on me what's happening: - odds of being yelled at if you miss a de…
This is the classic case of development velocity pitted against operational stability. Entirely different incentives, and when vested into the same role that role is bound to prioritize one incentive over the other. For this reason I think they must be separated at least by body if not by team. They certainly need to separated by different managers.
I tend to think QA is perhaps well situated alongside Ops("DevOps"), and very close to Product + Design.
Re: Asking developers to do QA is broken – why anyone should own QA
#103Many years ago I asked a developer which was a bigger deal: - Having an outage - Missing a deadline They answered: "oh having an outage is WAY worse". I then asked: "if that's the case, why do you push so hard to hit your deadlines with code you know and I both know is probably not ready?" They didn't really answer at the time but it eventually dawned on me what's happening: - odds of being yelled at if you miss a de…
> - odds of being yelled at if you miss a deadline: 100% > - odds of being yelled at due to an outage: unclear as it depends on the odds of an outage in general so let's say "less than 100%" To reframe this, the odds of a dev having to crunch to hit a deadline they're behind on is 100%, but the odds of any developer catching a support escalation or on-call page from an outage are usually way less, especially on large…
Just don't have sales and marketing ;)
Re: Asking developers to do QA is broken – why anyone should own QA
#104I've been a QA Automation Engineer for 15 years. This is the best thread ever on HN! I've read every single comment. I've found the best approach is for QAs to be embedded right into the product team - where the product team manager is the same person for both Dev and QA. QA and Dev all review each others code. QA reviews the Dev unit tests, and Devs review the QA integration tests (API and UI). QA may not be able to…
Thank you for your comment (useful and interesting information).
So it looks like your suggestion is to have 2 types of engineers in the team, where at least one of them is focused on QA + encourage bi-directional code reviews.
Could you tell a bit about the involvement of PMs in the QA process from your experience? What's working and what isn't?
Re: Asking developers to do QA is broken – why anyone should own QA
#105Tangentially, over my lifetime experience, I understand that QA Testing should be done on public (user) interfaces.
Intermediate data marshalling or transport, eg kafka delivering data from one application to another, is a dev ticket but has no QA component. Just as QA doesn't look at the individual data structures and method access modifiers, there are technical tasks and processes that QA has no stake in, nor can they do anything but complicate a delicate system with additional instrumentation/hooks. Generally, developers will want instrumentation, which is a public interface that QA can test, but is not mandated by QA as part of design.
Re: Asking developers to do QA is broken – why anyone should own QA
#106Many years ago I asked a developer which was a bigger deal: - Having an outage - Missing a deadline They answered: "oh having an outage is WAY worse". I then asked: "if that's the case, why do you push so hard to hit your deadlines with code you know and I both know is probably not ready?" They didn't really answer at the time but it eventually dawned on me what's happening: - odds of being yelled at if you miss a de…
> - odds of being yelled at if you miss a deadline: 100% > - odds of being yelled at due to an outage: unclear as it depends on the odds of an outage in general so let's say "less than 100%" To reframe this, the odds of a dev having to crunch to hit a deadline they're behind on is 100%, but the odds of any developer catching a support escalation or on-call page from an outage are usually way less, especially on large…
That (developers only being responsible for a fraction of bad rollouts they personally cause) reminds me of the water dynamic at a lot of apartments I've rented:
For "reasons" the water meters are per building rather than per unit, but the landlords are adamant that residents have to pay their fair share to ensure water isn't wasted. The scheme envisioned to meet those goals is that the total cost of water for a building is averaged out across all units (perhaps normalized by unit size). Looking at the net effect however, using $X of water only costs $X/N because my personal excess is split between the rest of the residents. Consequently, the entire building uses and pays for substantially more water than they would if the meters were more finely distributed.
Re: Asking developers to do QA is broken – why anyone should own QA
#107Asking developers to own QA is broken because developers are naturally biased towards the happy path. If you want to build a bar, you need someone to order -1 beers[1]. Handing off QA to an external team is broken because those people don't have the necessary experience with the product, nor can they quickly and easily engage with development to get to the heart of a problem (and a fix.) Having QA rely exclusively on…
I've also arrived at this approach and don't think it's that uncommon - IMO the article is presenting a false dichotomy. There are still gotchas to look out for in the team-embedded QA approach. In typical team sizes, you often end up with only one QA per team - you need to make sure they have cover (everyone needs a break), and they need support in their discipline (do something to share QA knowledge across teams).
What size team do you consider typical?
Most projects I've worked on have been games where the studio was split into teams of about 6-10 people with one dedicated in-house QA member for some but not all teams, a few in-house QA workers not assigned to a specific team, plus a much larger external QA team (from the parent company/ publisher). That has worked great.
I've also been on projects with only external QA. It works, but the lack of internal QA has been a frequent annoyance.
Re: Asking developers to do QA is broken – why anyone should own QA
#108Asking developers to own QA is broken because developers are naturally biased towards the happy path. If you want to build a bar, you need someone to order -1 beers[1]. Handing off QA to an external team is broken because those people don't have the necessary experience with the product, nor can they quickly and easily engage with development to get to the heart of a problem (and a fix.) Having QA rely exclusively on…
I've also arrived at this approach and don't think it's that uncommon - IMO the article is presenting a false dichotomy. There are still gotchas to look out for in the team-embedded QA approach. In typical team sizes, you often end up with only one QA per team - you need to make sure they have cover (everyone needs a break), and they need support in their discipline (do something to share QA knowledge across teams).
Agreed it's presenting a false dichotomy. The article ends with a pitch for Rainforest's "no-code" QA platform so it makes sense they want to present their product as the clear solution. I could take the article more seriously if it at least mentioned a more integrated approach.
Re: Asking developers to do QA is broken – why anyone should own QA
#109Earlier quoted context omitted.
I've also arrived at this approach and don't think it's that uncommon - IMO the article is presenting a false dichotomy. There are still gotchas to look out for in the team-embedded QA approach. In typical team sizes, you often end up with only one QA per team - you need to make sure they have cover (everyone needs a break), and they need support in their discipline (do something to share QA knowledge across teams).
What size team do you consider typical? Most projects I've worked on have been games where the studio was split into teams of about 6-10 people with one dedicated in-house QA member for some but not all teams, a few in-house QA workers not assigned to a specific team, plus a much larger external QA team (from the parent company/ publisher). That has worked great. I've also been on projects with only external QA. It w…
Yeah - in my experience 6-10 people per team is typical, often with one of them being dedicated QA. Having this, plus a separate central QA team is one way to address the pitfalls of embedded QA I was pointing out - they can cover for time away for team QA, or act as bench capacity.
Re: Asking developers to do QA is broken – why anyone should own QA
#110I think product owners should do (manual) QA. On their own, and when necessary for scalability reasons also via a small team under their direct control.
Yeah, this is the only thing that practically worked, in a complex product I used to work on. - If you ask the devs to do QA, you'll get no bugs other than the ones they already caught during testing and deployment. - If you have a mostly independent QA team, they will find somewhat silly/trivial bugs like the login page not working in an extreme edge case scenario. - However, when you ask your Product team to own QA…
> - If you have a mostly independent QA team, they will find somewhat silly/trivial bugs like the login page not working in an extreme edge case scenario.
Fair but you have to keep in mind that part of QA's job is to explore the silly and extreme edge cases too. One of the nastiest bugs I ever found started as a joke of a test. What happens if you enter a massive long text string (I think it was something like 600 kilobytes) into the password field and enter it?
I expected the UI to barf and a bit of mischief. What I got was the entire server backend crashing and loosing every user session with it when it restarted. The joke test revealed a uncovered a serious denial of service attack vector that was trivial to execute, trivial to automate, difficult to detect, and incredibly effective.
I agree it would be silly if QA insisted that trivial or edge cases be fixed; every bug has to be judged on the cost of fixing vs the risk of it happening in the field. But finding those silly edge case bugs is still very well worth the time to explore. If not because chasing the edge can reveal serious problems, but also because it also helps define where the limits of sanity really are instead of were it is imagined to be.