Maybe getting rid of your QA team was bad
61–70 of 269 posts
Re: Maybe getting rid of your QA team was bad
#62I worked at a company with a world-class QA team. They were amazing and I can't say enough nice things about them. They were comprehensive and professional and amazing human beings. They had great attention to detail and they catalogued a huge spreadsheet of manual things to test. Engineers loved working with them.
However -- the end result was that engineers got lazy. They were throwing code over to QA while barely testing code themselves. They were entirely reliant on manual QA, so every release bounced back and forth several times before release. Sometimes, we had feature branches being tested for months, creating HUGE merge conflicts.
Of course, management noticed this was inefficient, so they formed another team dedicated to automated QA. But their coverage was always tiny, and they didn't have resources to cover every release, so everyone wanted to continue using manual QA for CYA purposes.
When I started my own company, I hired some of my old engineering coworkers. I decided to not hire QA at all, which was controversial because we _loved_ our old QA team. However, the end result was that we were much faster.
1. It forced us to invest heavily on automation (parallelizing the bejesus out of everything, so it runs in 2. Engineers had a _lot_ of motivation to test things well themselves because there was no CYA culture. They couldn't throw things over a wall and wash their hands of any accountability.
We also didn't have a lack of end-to-end tests, as the author alludes to. Almost all of our tests were functional / integration tests, that run on top of a docker-compose set up that simulated production pretty well. After all, are unit tests where you mock every data source helpful at all? We invested a lot of time in making realistic fixtures.
Sure, we released some small bugs. But we never had huge, show stopping bugs because engineers acted as owners, carefully testing the worst-case scenarios themselves.
The only downside was that we were slower to catch subtle, not-caught-by-Sentry bugs, so things like UX transition weirdness. But that was mostly okay.
Now, there is still a use case for manual QA -- it's a question of risk tolerance. However, most applications don't fit that category.
Re: Maybe getting rid of your QA team was bad
#63This might be my personal experience, but I've never encountered a QA team that actually writes the tests for engineering. I have only had QA teams that wrote "test plans" and executed them manually, and in rarer cases, via automated browser / device tests. I consider these types of tests to be valuable, but less so than "unit tests" or "integration tests". With this model, I have found that the engineering team ends…
From what I've seen, the value in QA is product familiarity. Good QA'ers know more about how the product actually works than anybody else. More than PM's, more than sales, and more than most dev teams. They have a holistic knowledge of the entire user-facing system and can tell you exactly what to expect when any button gets pushed. Bad QA'ers are indeed a source of noise. But so are bad devs, sysadmins, T1/2 support…
Where I've seen QA teams most effective is providing more function than "just" QA. I've seen them used for 2nd tier support. I've seen them used to support sales engineers. I've also seen QA teams that take their manual test plans and automate their execution (think Selenium or UiPath) and have seen those automations included in dev pipelines.
Finally, the QA team are the masters and caretakers of your test environment(s), all the different types of accounts you need for testing, they should have the knowledge of all the different browsers and OSes your customers are using, and so forth.
That's a lot for the dev team to take on.
Re: Maybe getting rid of your QA team was bad
#64This might be my personal experience, but I've never encountered a QA team that actually writes the tests for engineering. I have only had QA teams that wrote "test plans" and executed them manually, and in rarer cases, via automated browser / device tests. I consider these types of tests to be valuable, but less so than "unit tests" or "integration tests". With this model, I have found that the engineering team ends…
That said, those test plans are gold. They form the definition of the product’s behavior better than any Google Doc, Integration Test, or rotating PM ever could.
Re: Maybe getting rid of your QA team was bad
#65Yes, part of the job was to write and run manual test suites, and to sign off as the DRI that a certain version had had all the automated and manual tests pass before release.
But their main value was in the completely vague mandate "get in there and try to break it." Having someone who knows the system and can really dig into the weak spots to find problems that devs will try to handwave away ("just one report of the problem? probably just some flaky browser extension") is so valuable.
In my current job, I have tried for 5+ years to get leadership to agree to a FT QA function. No dice. "Developers should test their own code." Yeah and humans should stop polluting the ocean and using fossil fuels, how's that going?
Re: Maybe getting rid of your QA team was bad
#66The main problem with QA teams is the same problem with IT teams or even management. If they are doing their jobs well they appear to be doing nothing. This often creates a situation where people need to "justify" their jobs. Usually this happens due to an over reliance upon metrics (see Goodhart's Law) rather than understanding what the metrics are proxying and what the actual purpose of the job is. A bad QA team is…
When I work with new devs I can often trip them up with basic tests of double-clicking, leading spaces, using the back button on Android. They then learn these and from then on these issues dont appear ( well, OK, it might take a couple of times of a ticket being rejected because of these but they do quickly learn all my tricks ) I don't get measured on bugs found so there's no pressure on me to find stupid bugs just…
Re: Maybe getting rid of your QA team was bad
#67I worked at two companies 15-20 years ago that invested in top-tier QA teams. They were worth their weight in gold. The products were world class because the QA team were fantastic at finding bugs we developers didn't think of looking for because we were too close to the problem. We are too used to looking at the happy path. One key attribute to both companies is that it was dictated from on high that the QA team had…
Not to mention, at one recent employer, the QE team wrote an enormous amount of code to perform their tests - It was more LOC than the modules being tested/certified.
Re: Maybe getting rid of your QA team was bad
#681. hire a contractor who just has no idea about anything. 2. hire someone and place them outside the engineering org (on the support team as a "support engineer" seems pretty popular) where they have little to no interaction with either engineering _or_ customers and expect them to work miracles.
Re: Maybe getting rid of your QA team was bad
#69This might be my personal experience, but I've never encountered a QA team that actually writes the tests for engineering. I have only had QA teams that wrote "test plans" and executed them manually, and in rarer cases, via automated browser / device tests. I consider these types of tests to be valuable, but less so than "unit tests" or "integration tests". With this model, I have found that the engineering team ends…
I might just be too old, but I remember when QA people didn't typically write tests, they manually tested your code and did all those weird things you were really hoping users wouldn't do. They found issues and bugs that would be hard to universally catch with tests.
Now we hoist QA on the user.
Working with younger devs I find that the very concept of QA is something that is increasingly foreign to them. It's astounding how often I've seen bugs get to prod and ask "how did it work when you play around with it locally?" only to get strange looks: it passed the type checker, why not ship it?
Programmer efficiency these days is measured in PRs/minute, so introducing bugs is not only not a problem, but great because it means you have another PR you can push in a few days once someone else notices it in prod! QA would have ruined this.
Re: Maybe getting rid of your QA team was bad
#70This might be my personal experience, but I've never encountered a QA team that actually writes the tests for engineering. I have only had QA teams that wrote "test plans" and executed them manually, and in rarer cases, via automated browser / device tests. I consider these types of tests to be valuable, but less so than "unit tests" or "integration tests". With this model, I have found that the engineering team ends…