AWS outage shows internet users 'at mercy' of too few providers, experts say

If you were using AWS services that went down in us-east-1, how did you avoid an outage without failing over to anything outside that region?

Terms like reliability have specific definitions in computer systems:

  Term           | Definition                          | Measurement
  --------------   -----------------------------------   -------------------------------------------
  Availability   | Basically, system uptime            | A percentage over time
  Durability     | Basically, persistence of data      | A percentage over time
  Resiliency     | Basically, self-healing             | A probability within a time period (usually)
  Reliability    | Basically, operational probability  | A probability within a time period (usually)
  Fault tolerant | Basically, it cannot fail           | Binary (it has faults or it doesn't)

Unlike more mathy fields, reliability is more of a "quality" that is qualified by one or more measurements (like Mean Time Between Failure). You define your metric, you give an estimate of what that value should be, and if you come in under it, you're reliable.

AWS has always stretched the truth when it comes to these numbers, but they do come pretty close to them most of the time. If you can find a different provider who'll even offer a number, it is usually not as close, and there's usually no contract that has any teeth to enforce it. Or they'll give very vague claims that don't get into specifics.

At least, not for "cloud providers" (other than the hyperscalers). You can find a datacenter who'll give you a number, but that's for like, their power reliability. That's a very different thing than saying "there is X probability over Y time that a server I run for you will not go down". Partly because it's pretty freakin' hard to wrangle all the different things that can go wrong with so much certainty that you can put a number on it. So most people give things like reliability, durability, availability, etc numbers for specific components of a system.

AWS S3 offers 99.999999999% durability and 99.99% availability. Now, did AWS S3 go down completely during the outage? Not as far as I'm aware. Maybe the control plane did, or a management portal, or billing, or something? But I'll bet you the PUT, GET, DELETE operations kept on flowing within 99.99% availability. Some other components in AWS may have been failing like crazy (which may have no guarantees...), but that one component probably stayed up within its guaranteed amount.

Design your apps to run on AWS using the components with specific guarantees, and you can estimate how reliable your end product will be. As far as I know, nobody has a better track record for meeting the guarantees. Even considering events like this.

My first paid position as a software developer was for a small, dot-com startup in Windsor, ON Canada. We co-located in Detroit - which meant border crossings (though this was pre-911 so crossing as a Canadian citizen was easier) - and had just a couple of servers on a rack. We were software engineers and had people who knew what they were doing. So yeah we were technically proficient. But I'm not sure I'd call that company among "the most technically sophisticated of organizations". We were tiny. In fact, when I first started working there, we were working out of a house with a workforce of like 25 people max.

When that company went under during the dot-com crash, I started my first business shortly after. It was 2003, I was 21 years-old and this business allowed me to work from home and feed my family until I re-entered the job market in 2018. For 15 years, I was a one-person organization, and because my business operated "free" adult-entertainment websites, bandwidth was my most significant expense. For that reason, even when Cloud became a thing (which it wasn't in 2003), I never migrated because of the bandwidth costs alone. Cloudflare was a major game changer but even it didn't exist when I first started out. There were CDNs like Akamai but they were crazy expensive and out of my league. So at its peak, I had about 12 bare metal servers around the world (all rented from the same hosting company - original called Server Matrix it then became Softlayer and then was bought by IBM and went to shit and is now IBM Cloud). I admin'd those on top of writing and maintaining all of the code and running the business independently with occasional help from my wife.

I am obviously very technically competent. I'm a Principal Software Engineer today. But technically sophisticated? There wasn't much sophistication about it. I did bare metal servers because it was the only cost-effective way to run my business. It was attainable and it worked. And it worked in a way that Cloud couldn't when Cloud came on the scene - so I never went Cloud with that operation just due to cost alone.

We do, non-tech company's IT dept doesn't so much

There are multiple different providers, with nothing artificially limiting their use. Also idk what's so bad about Fortnite and Snapchat going down at once instead of it being staggered.