Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Maybe getting rid of your QA team was bad

davidkcaudill.medium.com

Post reply on HN
Print view View original

221–230 of 269 posts

Re: Maybe getting rid of your QA team was bad

#221

Post by quadrifoliate »

robocat wrote: post #155
oaththrowaway wrote: post #20

I had a boss at Yahoo who gave our QA to another team because "Facebook doesn't use QA, we shouldn't either". I can't remember if it was Facebook or MS, but he was willing to buy all of us a book talking about how amazing it was. Long story short, it wasn't. It was like taking away a crutch. Of course we could have been more diligent about testing before having QA validate it, but it slowed development down so much t…

An article about Facebook's reason for no QA with some of the mitigations: https://blog.southparkcommons.com/move-fast-or-die/ A bit recent to have affected Yahoo - but it sells a good story. We would celebrate the first time someone broke something. Let anyone touch any part of the codebase and get in there to fix a bug or build a feature. Yes, this can cause bugs. Yes, that is an acceptable tradeoff. We had a share…

The difference is that bugs in the social network parts of Facebook (the ones where you see your friends and family's pictures and posts) are not directly making money for Facebook. The only real stuff that matters for the money is all the tracking.

I bet the people responsible for Facebook Ads Manager are a lot less enthusiastic about "move fast and break things", although I'd be interested to hear an opposing viewpoint from anyone here who's worked for that group.

Re: Maybe getting rid of your QA team was bad

#222

Post by godelski »

tstrimple wrote: post #194
ponector wrote: post #158

Earlier quoted context omitted.

It is not about DevOps. In every organization if you do your work good and have no fuckups - you will not be recognized and promoted. But if you are hero-firefighter - managers will love you and help with promotion. Because visibility is a key! Key to everything in the corporate life. If your work is not visible for managers - you are doing nothing.

Of course visibility is key. What other kind of option is there? > “We have no idea what you did this year and no one has any idea of the value you’re providing, but what the hell. Here’s a huge bonus!” That doesn’t seem a little insane to you?

> What other kind of option is there?

Placing value based on the work assigned? Obviously insanely easier said than done. But I think we should embrace the fuzziness a bit. Your manager should have a very good idea of this. If they don't, then they're the ones that should be let go because this is a significant aspect of their job. (may be learned through indirect methods)

> That doesn’t seem a little insane to you?

Honestly, a lot of business practices and economics sounds insane to me. Similarly a lot of alternative suggestions (especially the latter) since they tend to not address the underlying issues but be bandaids. The ones that don't sound insane are often boring and very reasonable, but I think we've established that that's considered undesirable when you're evaluated by visibility. I think this is probably a significant part of the negative feedback loop.

Re: Maybe getting rid of your QA team was bad

#223

Post by thaumasiotes »

mikestew wrote: post #134
Zelphyr wrote: post #56

I worked at two companies 15-20 years ago that invested in top-tier QA teams. They were worth their weight in gold. The products were world class because the QA team were fantastic at finding bugs we developers didn't think of looking for because we were too close to the problem. We are too used to looking at the happy path. One key attribute to both companies is that it was dictated from on high that the QA team had…

I can't count how many products I've seen with 100% code coverage that objectively, quantifiably doesn't work. That’s because code coverage doesn’t find the bugs that result from code you didn’t write, but should have. Code coverage is but one measure, and to treat it as the measure is folly. (But, yes, I have heard a test manager at a large software company we’ve all heard of declare that test team was done because…

It's more because code coverage isn't measured correctly. What you want is branch coverage.

If you have something like this:

    if condition1:
       do_something1()
    if condition2:
       do_something2()
    if condition3:
       do_something3()
There are 8 possible paths for the code to follow here, but you can cover 100% of lines of code in one test. If your code coverage measure tells you that testing the case where conditions 1, 2, and 3 are all true achieves 100% coverage, your measure is worthless. Covering this code cannot be done in less than eight tests.

Re: Maybe getting rid of your QA team was bad

#224

Post by pseudalopex »

godelski wrote: post #113
mattgreenrocks wrote: post #58

Earlier quoted context omitted.

> Shouldn't businesses be operating like the latter? Constantly training? There's a rampant cultural mind-virus that argues that 95%th percentile is somehow tons of work (rather than a lack of unforced mistakes), so everyone just writes it off. It's on full display at this very site. Just look on any post involving software quality, and read a bunch of comments suggesting widespread apathy from engineers. Obviously e…

Yeah it is weird that it is believed that there is a linear scale to work in and quality considering how well known pareto/power distributions are. These distributions are extremely prolific too. I mean we even codify that sentiment in the 80/20 rule or say that 20% of time is writing code and 80% is debugging it. What's interesting is this effect is scalable. Like you see this when comparing countries by population…

> To have an over-reliance on metrics and refusing to acknowledge that metrics are proxies and considering the nuances that they were explicitly trained to look for and is what meaningfully distinguishes them from less experienced people. I've been trying to coin the term Goodhart's Hell to describe this more general phenomena because I think it is a fairly apt and concise description.

McNamara fallacy or quantitative fallacy.[1]

[1] https://en.wikipedia.org/wiki/McNamara_fallacy

Re: Maybe getting rid of your QA team was bad

#226

Post by sharts »

The problem with QA is nobody outside of QA understands QA. They think QC is QA so they automate their QC "testing" and assume that is QA. It's not. Moreover they never listen to QA nor respect them as people.

QA are usually the best people to know what's what when the rubber meets the road. They know where the bodies are buried. They often understand and have a pulse on customers and usage better than product managers. They know the ins-and-outs of how various features interact and how a product as a whole works better than silo'd developers.

Historically they were effectively the only "product owners" that could certify a release before it went out. They would coordinate with the right people to ensure all technical and non-technical deliverables and dependencies were met before releases. T hey would be the best approximate and power users.

They often maintained test infrastructures on which deployments could be tested. In fact, they were the origins of automation or DevOps as a "thing" because they are the ones who saw all the friction points daily giving rise to CI/CD. Often times nobody listened because they were concerned with features.

QA has always been about investing resources within an organization to improve it. In effect, building things for within it to improve it. Now that we have gotten the message to some degree of optimizing some manual pain points -- we kick to the curb those who got us there without any regard to the value they provided -- and instead decide to push to prod and test on production to piss off customers even more.

If you ask yourself -- would you feel comfortable driving a car that was built and tested with automation alone? What about a space shuttle or an airplane? Or about medications?Or would you prefer that a human test drivers or test pilots put those products through their paces before signing off on them? -- it might drive it home better.

But then again software industry has devolved since the tech bro + VC culture pretty much ate it as they chased those sweet $$$

Re: Maybe getting rid of your QA team was bad

#227

Post by dclowd9901 »

amlozano wrote: post #21

This point is brought up in the article but I think it is at the real heart of the issue. QA is almost always seen as a 'cost center' by the business and upper management. I have a hypothesis that you never ought to work in a department that is seen as a 'cost center'. The bonuses, the recognition, and the respect always goes to the money makers. The cost center is the first place to get more work with less hands, ge…

I work on our frontend platform team and my perf work, CI design and process engineering definitely have a harder time getting recognition and promotions than folks who ship things that translate to dollars in the bank.

I don’t care though. I enjoy making things better and more robust. It makes my soul feel better. I’ll leave fucking things up to the cynics.

Re: Maybe getting rid of your QA team was bad

#228

Post by mynameisnoone »

It's not so much a Shakespearean "to have a QA team or not ...", but responsibility and accountability.

SWEs must produce unit testing because they know the code best. Dumping this responsibility onto QA is slow, evil, and wrong like quality control only vs. QA+QC.

QA teams must have the authority to ensure complicated code gets unit testing code coverage.

QA teams should provide partial and full-up integration tools and testing with the support of SWEs.

QA teams must have stop-the-assembly-line authority to ensure quality and testing requirements are met.

QA teams (or tools teams that support multiple QA teams) must make testing faster and more efficient.

There ought to be a suite of smoke tests such that untested code cannot be committed to the "production" branch, whatever that looks like, except in rare extreme emergencies.

All production-ready commits should be squashed, signed-off by another SWE, and have a test plan.

Test plans should be auto-generated, wherever possible.

Tests combined with test infrastructure should be able to be added to auto-bisect/blame breakage

Which tests must run and pass to accept a production proposed diff should be auto-minimized to those that touch particular area(s) and their precise-as-possible-but-still-correct dependencies.

Other areas that must not be neglected and shoveled onto SWEs: product management, UAT, UX, and operations.

Re: Maybe getting rid of your QA team was bad

#229

Post by alternatex »

thaumasiotes wrote: post #223
mikestew wrote: post #134

Earlier quoted context omitted.

I can't count how many products I've seen with 100% code coverage that objectively, quantifiably doesn't work. That’s because code coverage doesn’t find the bugs that result from code you didn’t write, but should have. Code coverage is but one measure, and to treat it as the measure is folly. (But, yes, I have heard a test manager at a large software company we’ve all heard of declare that test team was done because…

It's more because code coverage isn't measured correctly. What you want is branch coverage. If you have something like this: if condition1: do_something1() if condition2: do_something2() if condition3: do_something3() There are 8 possible paths for the code to follow here, but you can cover 100% of lines of code in one test. If your code coverage measure tells you that testing the case where conditions 1, 2, and 3 ar…

I think most code coverage analysis tools are smart enough to consider flow control statements. At least when it comes to C# ones.

Re: Maybe getting rid of your QA team was bad

#230

Post by fho »

alternatex wrote: post #229
thaumasiotes wrote: post #223

Earlier quoted context omitted.

It's more because code coverage isn't measured correctly. What you want is branch coverage. If you have something like this: if condition1: do_something1() if condition2: do_something2() if condition3: do_something3() There are 8 possible paths for the code to follow here, but you can cover 100% of lines of code in one test. If your code coverage measure tells you that testing the case where conditions 1, 2, and 3 ar…

I think most code coverage analysis tools are smart enough to consider flow control statements. At least when it comes to C# ones.

They do, but that's not the point GP made. An example how this could fail is if the first branch sets up something that branch two uses. Two tests are written that call branch 1&2 and not call them. 100% code coverage, even the non-happy path was tested.

But in reality, if those branches have any interaction, you would need to write eight test cases for every combination of branches being run and not being run.

Post reply on HN