Yahoo’s Engineers Move to Coding Without a QA Team

Incentives often cause really nasty political wars within orgs.

When a site gives a 500 because a database went down and the web app couldn't connect to it... is that a bug, a missed test case or should ops take a hit on down time? Furthermore, if you argue that the dev team should have reasonable failsafes in code to connect to a db, in a 10 year old organization, should the current Dev team pay for something that could have been in the code for years?

If you set up a system where devs hand off to QA then they hand off to ops to deploy. Furthermore every step is incentivised somewhat against each other. Even if all teams are equal, it ends up in my opinion, to the path to CYA and Waterfall. Everyone is more concerned with problems not being 'their' fault than shipping good code.

This is why specification should happen collaboratively with everyone with a stake (and that includes QA just as much as BA), you can fix them before they're developed against.

The thing that I think is missed in the "getting rid of QA" debate is the size of releases. When you're deploying continuously, the size of what's changing drops substantially. Adding QA as a gating step in the process between developers and production is inefficient and, worse yet, QA will be overwhelmed by the number of rounds of QA they're forced to perform. It's not uncommon for QA regression testing to take days so when there are multiple releases per day, you can see the problem with putting QA between development and production.

That's not to say that dedicated QA doesn't have a purpose. They can and should be testing the live code on a regular basis (weekly or daily, depending on how long the testing cycle takes). But automated tests are what's responsible for ensuring that nothing arrives in production in a completely-broken state and it's assumed that the benefits of continually shipping software will outweigh the downsides of occasionally having subtle bugs in production. And even with QA involved, bugs will make it through the process. At that point, the expedited process of pushing code to production becomes a huge win. Testing strategies often aim for the best MTBF, but when that comes at the cost of MTTF, it's not always a good thing. We've had bugs that were fixed in production less than 10 minutes after the bug is filed.

The other point that gets missed is that people assume that there's no manual QA happening and a developer's careless change just goes to production and wreaks havoc. This ignores the code review process, which is crucial to delivering quality software in a continuous deployment scenario. On my team, changes require 2 +1s before being merged into master, subjected to continuous integration again and eventually deployed to production. Moreover, if any engineer reviewing the code isn't sure they fully understand the change or otherwise wants to see the code running, a single command that runs in under 10 min will spin up an environment in AWS using the code in the pull request so that they can do any manual testing they need to feel comfortable adding their +1. When their done, a single command cleans up that environment.

The thing to keep in mind when designing a testing strategy is the context your software runs in. I would not advocate this testing strategy for code that runs in a vehicle where a bug could cause physical destruction. Likewise, I wouldn't use it for an application with access to highly-sensitive medical or financial information where a leak or data corruption could mean millions of dollars in losses/fines. But for the majority of internet software, the stakes just aren't that high and the gains from a streamlined development process will outweigh the losses from bugs that find their way into production.

Disclosure: I manage a team that deploys continually, usually upwards of 20 times per day. We're responsible for our own QA and have a significantly lower defect rate than other teams in the company with a more traditional QA strategy. However we still draw on QA resources when we feel like we're pushing something risky.

Well, I should have said that the idea is people would write well tested code before it gets released. I can also attest it didn't necessarily work out that ideally in practice.

On the other hand, I'll disagree that they combine dev/qa/support because they're cheap. This actually makes little sense to me because typically QA roles are paid less than traditional software developers. That being the case, it doesn't make sense to get rid of them and let your developers do that stuff if all you want to do is save money.

What I heard was that they used to have support staff but changed it because it wasn't working out. I wasn't there so I don't know if that actually happened, but I can see how it's both harder and slower to have someone who doesn't know the code base fix bugs on the spot.

Honestly I can see it both ways. I think there are a lot of benefits to having the developers write tests themselves. On the other hand, when a project gets big enough, I can see how it makes sense to have people only working on tools like automated test frameworks or build stuff.

> Traditional QA is pretty much dead

Where have you worked? Just inside the SV bubble? This is definitely not true.