Earlier quoted context omitted.
> AWS is pretty reliable for the most part so I am pretty surprise that the code quality is that bad. I'm not totally surprised because of two factors: very stable product definitions and lots and lots of users. A number of years back, I was talking with people at a famous and popular site with a broad audience. I asked them how much unit testing they did. They said that particular isolated pieces sometimes had tests…
For very, very popular services, a second of being live will exercise more code paths and edge cases than even the most dedicated testing team could ever dream of. We hear a hell of a lot about testing but the most fundamental piece of software quality nowadays is the release strategy: running on tee'd live production traffic, canarying, metrics and alerting, quick roll backs, etc.
Most of the code we care about is to handle anomalous situations. That AZ going down a week or two is a good example. It's when stuff like that happens that a bunch of code springs to life to keep things running. And indeed, things didn't exactly roll over just fine for us.