Does Amazon host its core infrastructure in us-east-1?
why wouldn't they?
Amazon packages pile up after AWS outage spawns delivery havoc
11–20 of 219 posts
Re: Amazon packages pile up after AWS outage spawns delivery havoc
#12I've seen mention of a major cable tv operator/last-mile residential ISP that had their entire field-tech dispatch system go down (based on a web gui https app that people can use on their phones) because it's hosted in AWS.
Re: Amazon packages pile up after AWS outage spawns delivery havoc
#13I thought the whole point of AWS that it was a distributed system with no one point of failure? In that case, how did it have an outage?
I guess on a load balancer or DNS level? A request has to hit a single domain name / IP before it's load balanced to the distributed system right?
Re: Amazon packages pile up after AWS outage spawns delivery havoc
#14I thought the whole point of AWS that it was a distributed system with no one point of failure? In that case, how did it have an outage?
do they not teach people what a failure domain is anymore?
https://en.wikipedia.org/wiki/Failure_domain
Clouds aren't magic. They require a certain amount of operational confidence in order to understand that, yes, an entire region can fall out from under you at any time and it's your responsibility to detect and deploy into an unaffected region if possible.
edit: Generally, one entire region will not fail. However, core services like STS rely on us-east-1 so it's particularly susceptible to disruption.
Re: Amazon packages pile up after AWS outage spawns delivery havoc
#15I thought the whole point of AWS that it was a distributed system with no one point of failure? In that case, how did it have an outage?
Region outages happen, which is why the guidance is to build in multiple regions, but even Amazon sometimes doesn't take their own advice. Sometimes the reason is good, sometimes it isn't.
Re: Amazon packages pile up after AWS outage spawns delivery havoc
#16I thought the whole point of AWS that it was a distributed system with no one point of failure? In that case, how did it have an outage?
Region outages happen, which is why the guidance is to build in multiple regions, but even Amazon sometimes doesn't take their own advice. Sometimes the reason is good, sometimes it isn't.
I thought the guidance was to be multi-AZ, as an AZ is the failure boundary?
Re: Amazon packages pile up after AWS outage spawns delivery havoc
#17If this outage isn’t the catalyst to get the Amazon side of the house to finally move out of US-east, I don’t know what will be. Or at least be multi-region.
Although the cost to make all of Amazon commerce, logistics, and digital truly multi-region is probably an order of magnitude more than the impact of this outage.
Re: Amazon packages pile up after AWS outage spawns delivery havoc
#18Check out the Amazon Flex Drivers subreddit[0], there are tons of people saying they were paid for the day of deliveries without delivering any packages and told to go home.
[0] https://www.reddit.com/r/AmazonFlexDrivers/comments/rb3ggn/i...
Re: Amazon packages pile up after AWS outage spawns delivery havoc
#19Earlier quoted context omitted.
Region outages happen, which is why the guidance is to build in multiple regions, but even Amazon sometimes doesn't take their own advice. Sometimes the reason is good, sometimes it isn't.
I thought the guidance was to be multi-AZ, as an AZ is the failure boundary?
Multi-AZ deployment and multi-region failover is considered best practice.
Re: Amazon packages pile up after AWS outage spawns delivery havoc
#20Earlier quoted context omitted.
Region outages happen, which is why the guidance is to build in multiple regions, but even Amazon sometimes doesn't take their own advice. Sometimes the reason is good, sometimes it isn't.
I thought the guidance was to be multi-AZ, as an AZ is the failure boundary?
AZs are physically located near each other, usually within a small enough radius that they could be all impacted by the same natural disaster. In some cloud providers and regions, AZs are simply different parts of the same building (IIRC one of the Japan regions of Azure was essentially this, but don't quote me). And evidently, the share some infrastructure.
At a previous job where we needed to always be up, our disaster recovery plan assumed that the us-east-1 site had been hit by a meteor (not literally, but that's how we explained it to each other to put ourselves in the mindset.)