I thought the whole point of AWS that it was a distributed system with no one point of failure? In that case, how did it have an outage?
A lot of AWS is very old and it's very hard to go back and make everything nice and reliable without breaking things when you have a shitload of customers 24/7 using your services. So yes, in theory much of AWS's services are probably very reliable and distributed across AZs and regions, but in practice there's likely a whole bunch of debt where one thing gets fucked up and it cascades.
As bad as this article makes AWS sound, it's actually the reason you should go with AWS over say Azure or GCP; when AWS goes down, its owners actually feel the pain with you, Microsoft and Google run their own stuff elsewhere...
Source? I'd be surprised if Google does not dog food. Though, I guess when GCP had their recent global load balancer outage but neither Gmail or Google search went down maybe not.
Most of Google's stuff runs on Borg, which predates GCP (and is the fabric GCP runs on top of).
Check out the Amazon Flex Drivers subreddit[0], there are tons of people saying they were paid for the day of deliveries without delivering any packages and told to go home. [0] https://www.reddit.com/r/AmazonFlexDrivers/comments/rb3ggn/i...
It's official[0] Amazon is sending out emails to Flex drivers informing them Flex is down and they'll be paid for scheduled blocks without deliveries.
Apparently even some warehouses were ghost towns today without anyone sorting packages[1]
Here in Phoenix, I've had two packages supposed to be delivered today now delayed. Also, delivery on items just went from overnight delivery to 3+ day delivery as the quickest option.
Source? I'd be surprised if Google does not dog food. Though, I guess when GCP had their recent global load balancer outage but neither Gmail or Google search went down maybe not.
Most of Google's stuff runs on Borg, which predates GCP (and is the fabric GCP runs on top of).
Borg is the predecessor to Kubernetes (sort of) right? They could have switched to Kubernetes and run on Google Kubernetes Engine (GKE).
Check out the Amazon Flex Drivers subreddit[0], there are tons of people saying they were paid for the day of deliveries without delivering any packages and told to go home. [0] https://www.reddit.com/r/AmazonFlexDrivers/comments/rb3ggn/i...
Wouldn't surprise me. I mean, DSPs wouldn't be able to scan packages to pick them up without being able to access the flex app, so unless there's a procedure in place to allow packages to be picked up and manually marked as having been picked up... they wouldn't be able to do anything. Moreover, even if they could pick anything up, they wouldn't have any way to navigate, drop packages off, snap photos, and otherwise…
Failing over to another AWS region is actually pretty difficult for stateful services. Especially if you can't even access data in the primary region at all. Most teams probably don't have the bandwidth to solve this problem given the amount of outages you see in a year (1 or 2). Also, this would be a problem many teams would be solving, so most teams probably just wait and see what leaders have to say about it and well, nothing ends up getting done.
Because it's terrible and has always been terrible? After 2017 I'll never use US-east-1 again. Hell... I should have learned that particular lesson in 2011 but it took two catastrophic failures for me to figure it out. There are numerous threads here on HN covering the topic "why does US-east-1 suck so hard." https://news.ycombinator.com/item?id=13756082 is just one example.
Maybe it’s terrible because they host their own stuff there.
I thought the whole point of AWS that it was a distributed system with no one point of failure? In that case, how did it have an outage?
Region outages happen, which is why the guidance is to build in multiple regions, but even Amazon sometimes doesn't take their own advice. Sometimes the reason is good, sometimes it isn't.
Our EC2 instances in us-east-1 didn't actually go down. But IT engineering was completely disrupted because our SSH login mechanism relies on the API to show you the list of instances for you to select which one to start an IAM handshake with. Our support phone line was also down. Even SQS kept chugging along just fine. I'm actually glad we're not on Lambda because of this.
Check out the Amazon Flex Drivers subreddit[0], there are tons of people saying they were paid for the day of deliveries without delivering any packages and told to go home. [0] https://www.reddit.com/r/AmazonFlexDrivers/comments/rb3ggn/i...
That's actually pretty nice though; even though people are missing out on packages at least Amazon won't leave them out in the cold for today.
Whenever us-east-1 goes down you just get a really good feel for how many other companies also have pretty fragile setups. They apps I work on can deal with a few hours of downtime, so as long as I'm sure I can recover from getting totally leveled its ok. And I think it's that was for the majority of companies. Most don't want the extra effort and cost of failover.