Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Maybe getting rid of your QA team was bad

davidkcaudill.medium.com

Post reply on HN
Print view View original

141–150 of 269 posts

Re: Maybe getting rid of your QA team was bad

#141

Post by w10-1 »

For QA to be respected and protected, it has to identify what it's responsible for.

Luckily, that's easy: the "fault model", all the ways things can break. That tends to be a lot more complex than the operating model, the domain model, or the business model.

Once all the potential issues and associated costs for all the fault models are enumerated, then QA can happily offer to any other organization the responsibility for each one, and see who steps up to take it on.

In many cases, it can be done more cheaply in design, engineering, or automation; it's usually easier to prevent a problem than capture, triage, debug, fix, and re-deploy.

Organizations commonly make the mistake of being oblivious to the fault models and failing to allocate responsibility. That's possible because most failures are rare, and the link from consequences back to cause is often unclear. The responsibility allocation devolves to blame, and blame to "who touched this last"? But catastrophic feedback is a terrible way to learn, and chronic irritants are among the best ways to lose customers and staff.

Re: Maybe getting rid of your QA team was bad

#142

Post by nlavezzo »

When we built FoundationDB, we had a maniacal focus on quality and testing. So much so that we built it in a language we invented, called Flow, that allowed us to deterministically simulate arbitrary sized FDB clusters and subject them to crazy conditions, then flag each system property violation and be able to perfectly reproduce the test run that triggered the violation.

We got to a point where the default was that all of our 10,000's of test runs each night would flash green if no new code was introduced. Tests that flashed red were almost always due to recent code additions, and therefore easily identified and fixed. It let our team develop knowing that any bugs they introduced would be quickly caught, and this translated to being able to confidently take on crazy new projects - like re-writing our transaction processing system post-launch and getting a 10x speed increase out of it.

In the end our focus on quality led to velocity - they weren't mutually exclusive at all. We don't think this is an isolated phenomenon, which led us to our newest project - but that's a story for another time.

Re: Maybe getting rid of your QA team was bad

#143

Post by bigbillheck »

hornban wrote: post #128
wrs wrote: post #123

Earlier quoted context omitted.

At Microsoft back in the day, we called those “bug bashes”, and my startup inherited the idea. We encouraged the whole company to take an afternoon off to participate, and gave out awards for highest impact bug, most interesting bug, etc.

This is a bit of an aside, but I have a question that I'd like to ask the wider community here. How can you do a proper bug-bash when also dealing with Scrum metrics that result in a race for new features without any regard for quality? I've tried to do this with my teams several times, but ultimately we're always coming down to the end of the sprint with too much to do to implement features, and so anybody that "tak…

Only assign points based on a (n-1)-day sprint instead of a n-day one.

Re: Maybe getting rid of your QA team was bad

#144

Post by bsder »

godelski wrote: post #32

The main problem with QA teams is the same problem with IT teams or even management. If they are doing their jobs well they appear to be doing nothing. This often creates a situation where people need to "justify" their jobs. Usually this happens due to an over reliance upon metrics (see Goodhart's Law) rather than understanding what the metrics are proxying and what the actual purpose of the job is. A bad QA team is…

> Shouldn't businesses be operating like the latter? Constantly training?

"But then they'll leave for somewhere else for more money."

Literally every company I have worked for. Meaningful training was always an uphill battle.

However, good training also requires someone good and broad in your technical ladder. They may not be the most up-to-date, but they need to be able to sniff out bullshit and call it out.

FAANG is no exception, either.

Re: Maybe getting rid of your QA team was bad

#145

Post by philk10 »

godelski wrote: post #125
philk10 wrote: post #38

Earlier quoted context omitted.

When I work with new devs I can often trip them up with basic tests of double-clicking, leading spaces, using the back button on Android. They then learn these and from then on these issues dont appear ( well, OK, it might take a couple of times of a ticket being rejected because of these but they do quickly learn all my tricks ) I don't get measured on bugs found so there's no pressure on me to find stupid bugs just…

> I don't get measured on bugs found so there's no pressure on me to find stupid bugs just to boost my figures. Sounds like the right incentive structure. If you don't mind, how are you judged? Do you feel like the system you're in is creating the appropriate incentives and actually being effective? Certainly this example is but I'd like to know more details from an expert so I can update my understanding.

The system is really effective, I wanted to work at a place where the cliche "everyone cares about quality" is actually true and I found it - devs test, designers test, I test, customer has the chance to test the latest build every 2 weeks so that we can check that our quality checks are aligning with theirs. It gets to be a game of 'can the devs get it past my checks' and 'can I find new ways to trip them up' which builds up confidence in each others skill levels.

Re: Maybe getting rid of your QA team was bad

#146

Post by ponector »

alkonaut wrote: post #105
Zelphyr wrote: post #56

I worked at two companies 15-20 years ago that invested in top-tier QA teams. They were worth their weight in gold. The products were world class because the QA team were fantastic at finding bugs we developers didn't think of looking for because we were too close to the problem. We are too used to looking at the happy path. One key attribute to both companies is that it was dictated from on high that the QA team had…

I love my really thorough QA’s. Yes it’s an antipattern to let me as a dev lean too much on them catching what I won’t. But where I dread even running the code for a minute, they enjoy it. They take pride in figuring out edge cases far beyond any spec. They are definitely worth their weight in gold. It lets developers have confidence when changing things in the same sense a good type system does. For some classes of…

People here are talking about skillful QA worth their weight in gold.

Unfortunately people in the industry who have actual power in planning budgets don't think so. An article is right. QA engineers now are viewed as janitors: no one respects then, better to outsource to cheap location.

Re: Maybe getting rid of your QA team was bad

#147

Post by taurath »

bbarn wrote: post #53
yabones wrote: post #23

Earlier quoted context omitted.

From what I've seen, the value in QA is product familiarity. Good QA'ers know more about how the product actually works than anybody else. More than PM's, more than sales, and more than most dev teams. They have a holistic knowledge of the entire user-facing system and can tell you exactly what to expect when any button gets pushed. Bad QA'ers are indeed a source of noise. But so are bad devs, sysadmins, T1/2 support…

This is a great model, until those people so familiar with the business needs end up.. doing business things instead. It's really hard to keep people like that in a QA role once the business recognizes their value. Kind of the same problem with QA automation people - once they become really good at test automation, they are effectively software developers, and want to go there.

I have never once heard of a problem that QA folks end up in project or product management too often, and almost always have the problem of not being able to escape the QA org despite many years. Most companies are extremely resistant to people moving tracks, especially from a “lower status” org like QA or CS. It’s the exception not the rule.

Re: Maybe getting rid of your QA team was bad

#148

Post by [deleted] »

tstrimple wrote: post #137
serial_dev wrote: post #111

Earlier quoted context omitted.

Even as a developer (mobile app developer) I feel like one has to be careful not to work on "cost center" things. Accessibility, observability, good logging, testing infrastructure improvements, CI/CD tweaks, stability, better linting and analyzer issues are all important, but you will be rewarded if you ship features fast. This year I spent too much time on the former because I felt like that's what the team and app…

The purpose behind all those things you were pursuing (apart from accessibility) should have been to increase the rate at which the team is able to ship features. If your work on these items over the course of a year haven't demonstrably improved delivery speed, then what value did they actually bring? If they have improved delivery speed and you can show evidence for that, why would you be nervous going into a revie…

[deleted]

Re: Maybe getting rid of your QA team was bad

#149

Post by bumby »

w10-1 wrote: post #141

For QA to be respected and protected, it has to identify what it's responsible for. Luckily, that's easy: the "fault model", all the ways things can break. That tends to be a lot more complex than the operating model, the domain model, or the business model. Once all the potential issues and associated costs for all the fault models are enumerated, then QA can happily offer to any other organization the responsibilit…

I agree with you, but have the hunch that many PMs don't.

>it's usually easier to prevent a problem than capture, triage, debug, fix, and re-deploy.

It really depends on the risk of the fault. To a PM under schedule pressure, the higher risk may be to break schedule in order to redesign to mitigate the fault. As you said, many failures are low probability, so PMs are used to rolling the dice and getting away with it. Often they've moved on before those failures rear their ugly heads.

An organization really needs the processes that establish guardrails against these biases. Establishing requirements to use the tools to define the fault model can go a long way, although I've seen people get away with turning a blind eye to those requirements as well. You also need to mate it with strong accountability.

Post reply on HN