Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Maybe getting rid of your QA team was bad

davidkcaudill.medium.com

Post reply on HN
Print view View original

241–250 of 269 posts

Re: Maybe getting rid of your QA team was bad

#241

Post by antupis »

sporedro wrote: post #185
amtamt wrote: post #54

> The most conscientious employees in your organization are the most bitter. They see the quality issues, they often address them, and they get no recognition for doing so. When they speak up about quality concerns, they get treated like mouthbreathers who want to slow down. They watch the “move fast and break things” crowd get rewarded time after time, while they run around angrily cleaning up their messes. To these…

While QA and testing is important, I’m not sure you can convince the people only concerned about profits… I think the “release it fast and patch it later” concept here to stay due to the internet being so accessible. Why bother spending tons of money and time when the users will just report the bugs and you can release updates over the internet they can download. Ever since physical copies of video games and software…

“release it fast and patch it later” You are testing just different things eg is this product viable in the market? QA has its place but you should not cram it into every corner of software development and also the same time, not everything should be "release fast and break things".

Re: Maybe getting rid of your QA team was bad

#242

Post by thaumasiotes »

fho wrote: post #230
alternatex wrote: post #229

Earlier quoted context omitted.

I think most code coverage analysis tools are smart enough to consider flow control statements. At least when it comes to C# ones.

They do, but that's not the point GP made. An example how this could fail is if the first branch sets up something that branch two uses. Two tests are written that call branch 1&2 and not call them. 100% code coverage, even the non-happy path was tested. But in reality, if those branches have any interaction, you would need to write eight test cases for every combination of branches being run and not being run.

Well, again, eight is a lower bound. If the software uses any values beyond the bare minimum (here) of three separate booleans, you're likely to need more.

Take this simplified code:

    if condition1:
       do_something()
You need one test for code coverage and two for branch coverage, but if there's a bug in the code as written you'll find that you actually need more. Assume that we want to run do_something() when, and only when, condition2 is true.

Now we have four cases:

- condition1 && condition2: do_something() runs, which is correct.

- condition1 && !condition2: do_something() runs, which is incorrect.

- !condition1 && !condition2: do_something() does not run, which is correct.

- !condition1 && condition2: do_something() does not run, which is incorrect.

If you happened to write tests for the first and third of those options, your tests will make it look like your code works, and your (official) coverage metrics will look comprehensive. But because your coverage metrics (in reality) are actually terrible, your code is secretly not working.

The big problem we've just run into is that the number of test cases we need depends on the number of bugs in our code. If the goal of writing tests is to prove that we don't have bugs, this is a terrible result; it means our goal is fundamentally impossible to achieve. We don't know whether we have bugs, so we don't know how many tests we need.

Re: Maybe getting rid of your QA team was bad

#243

Post by bolobo »

hornban wrote: post #128
wrs wrote: post #123

Earlier quoted context omitted.

At Microsoft back in the day, we called those “bug bashes”, and my startup inherited the idea. We encouraged the whole company to take an afternoon off to participate, and gave out awards for highest impact bug, most interesting bug, etc.

This is a bit of an aside, but I have a question that I'd like to ask the wider community here. How can you do a proper bug-bash when also dealing with Scrum metrics that result in a race for new features without any regard for quality? I've tried to do this with my teams several times, but ultimately we're always coming down to the end of the sprint with too much to do to implement features, and so anybody that "tak…

Why are you putting the blame on scrum if you don't even implement it? I did scum in a previous company and it worked fine. Nobody looked at the story points except the devs during planning. We had a honest discussion with the product owner every time and did find the time to do tech debt.

It wasn't perfect, but it worked well.

Granted, it required a very specific management, devs with the right mindset and constraints on the kind of projects that could be done (anything customer facing with a tight deadline was off for instance. We used that for the internal infra). So I don't see how you would build a plane at Boing with scrum for instance. Or anything that require very tight coupling af many teams (or hardware).

But for us (60 devs in a company of 200), Saas, it worked great.

Re: Maybe getting rid of your QA team was bad

#244

Post by xtracto »

mynameisnoone wrote: post #228

It's not so much a Shakespearean "to have a QA team or not ...", but responsibility and accountability. SWEs must produce unit testing because they know the code best. Dumping this responsibility onto QA is slow, evil, and wrong like quality control only vs. QA+QC. QA teams must have the authority to ensure complicated code gets unit testing code coverage. QA teams should provide partial and full-up integration tools…

I am a CTO and have always tried to have the QA team to be part of the Product Office (CPO) . It tends to keep us honest and ultimately it aligns the decision of churning out new features and building crap in the same team/person.

Now, don't get me wrong, my teams always do unit and integration testing, along with automation of those two. Devs are responsible for the quality of their work. But ultimately it os the product team, with input from their QA team the ones deciding if a new feature is ready for release as it is, or needs more polish

Re: Maybe getting rid of your QA team was bad

#245

Post by bolobo »

JohnFen wrote: post #175
Shaanie wrote: post #165

Earlier quoted context omitted.

That's not a problem with Scrum, it's a problem with your team. If you're doing a bug bash every sprint, then your velocity is already including the time spent on bug bashes. If it's not in every sprint, you can reduce the forecast for sprints where you do them to account for it (similar to what you do when someone is off etc). If you're competing within the team to complete as many story points as possible that's pr…

> Is someone using story points as a metric of anything other than forecasting? Very nearly every company I've worked at that uses Scrum uses story points, velocity, etc., as a means of measuring how good you or your team are. Forecasting is a secondary purpose.

Ahh the good old "Waterfall Scrum"

Re: Maybe getting rid of your QA team was bad

#246

Post by turok »

Zelphyr wrote: post #56

I worked at two companies 15-20 years ago that invested in top-tier QA teams. They were worth their weight in gold. The products were world class because the QA team were fantastic at finding bugs we developers didn't think of looking for because we were too close to the problem. We are too used to looking at the happy path. One key attribute to both companies is that it was dictated from on high that the QA team had…

We try to do this, QA owns deployment to production and has the final say if a feature needs a re-write. The almost adversarial incentives of dev and QA are why it works. Dev wants to close ticket, QA wants the feature to work, Product wants to tick a box, all collaborate so that the closed ticket delivers a working feature and not just one of those three things.

Re: Maybe getting rid of your QA team was bad

#247

Post by carbotaniuman »

tstrimple wrote: post #193
hj3939393d wrote: post #183

Earlier quoted context omitted.

> demonstrably improved delivery speed Thinking this can be reduced to a single metric is the blight of modern software (and business in general, I think). Mapping an individual change to improved delivery speed is in the vast majority of cases an impossible task and any decent developer knows this. It's management+ that wants simple easy metrics since they lack the deep understanding required to do their job well. S…

If you can’t prove your contributions are worthwhile, you’re not going to get recognition. People putting out fires get recognition because they are solving visible and urgent problems. If your work reduces the chances of those fires, it should be measurable otherwise what is the point? Do you honestly believe someone who has spent a year “improving processes” but cannot measure the impact of that year of work deserv…

It's wild that in this thread we're still falling victim to the McNamara fallacy.

https://en.wikipedia.org/wiki/McNamara_fallacy

Re: Maybe getting rid of your QA team was bad

#248

Post by MichaelZuo »

eitally wrote: post #108
supportengineer wrote: post #67

Earlier quoted context omitted.

I've seen QA/QE greatness and it was similar to how you describe. A different chain of command for deciding if releases are certified for production. Different incentive structures as well. Not to mention, at one recent employer, the QE team wrote an enormous amount of code to perform their tests - It was more LOC than the modules being tested/certified.

I had a team like that once. It was glorious. And ultimately, I'm convinced it led to overall faster development cycles because the baseline code quality & documentation was so much better than it would have been without such a great QA manager. The QA team, of course, was also technical -- mostly with SWE backgrounds -- and they were primarily colo'd in the same office as the dev team. I still remember the epiphany…

I've heard this enough times that I'm becoming convinced there's a lot of 'reinventing the wheel' busywork going on in the industry.

And not even superior wheels, lower quality, more fragile wheels by clumsier wheel designers.

Re: Maybe getting rid of your QA team was bad

#249

Post by avidiax »

ryan-duve wrote: post #240
jrockway wrote: post #154

Earlier quoted context omitted.

I've always been impressed by hardware QA test teams I've worked with. On Google Fiber, they had an elaborate lab with every possible piece of consumer electronics equipment in there, and would evaluate every release against a (controlled) unfriendly RF environment. ("In version 1.2.3.4, the download from this MacBook Pro while the microwave was running was 123.4Mbps, but in version 1.2.4.5, it's 96.8Mbps." We actual…

> Going deeper into the training aspect, something I find very useful are fuzz tests. Could you share some details of fuzz tests that you've found useful? I tend to work with backend systems and am trying to figure out whether they will still be useful in addition to unit and integration tests.

Fuzz tests are most useful if they are run continuously/concurrently with development. Doing it that way, a change or decision that causes a fuzz test failure hasn't been built upon yet. Imagine building an earthquake resistant house on a shaker table vs. putting the 100% completed house on the same shaker table.

Doing fuzz testing at the end leads to a lot of low priority but high cost bugs being filed (and many low-cost bugs as well).

The utility of it is quite clear for security bugs. It requires low effort to find lots of crashes or errors that might be exploitable. For development in general, it tends to identify small errors or faulty architectural or synchronization decisions very early, while they are still easy to repair.

Re: Maybe getting rid of your QA team was bad

#250

Post by bigEnotation »

>>> slowest part of software delivery is testing

In my experience the slowest part has been marking a feature as done. I loved working at places with QA. I could assign tickets to QA once the PR was up.

Now I gotta build in that I’ll be bumping PRs for review for approximately 30-50% of the time I’m working on a feature.

Post reply on HN