Live data from Hacker News

Amazon packages pile up after AWS outage spawns delivery havoc

detroitnews.com

121–130 of 219 posts

Re: Amazon packages pile up after AWS outage spawns delivery havoc

#121

Earlier quoted context omitted.

Us-East-1 took out commuter rail travel in Chicago for like a good 30 minutes, so not just companies

> Us-East-1 took out commuter rail travel in Chicago for like a good 30 minutes, so not just companies Was it Metra or the CTA out of curiosity?

In Chicago I believe “commuter rail” almost always refers to Metra.

Re: Amazon packages pile up after AWS outage spawns delivery havoc

#123
post #92

Something I'm not getting there. OK, one zone went down in AWS. But there supposed to be multiple ones? And if somebody knows how to use AWS so that one zone going down wouldn't knock over the whole thing, it should be Amazon? I thought the whole point of this multi-zone cloud setup would be so that even if one zone goes down, it could survive and perform as usual - maybe a little slower for a bit while backups kick…

This affected on region, us-east-1, which is the oldest and largest aws region, and also hosts many core aws services. Each region has several availability zones (AZs) that are basically whole datacenters. In theory AZs should be mostly isolated, but as we saw bugs happen. Reading between the lines of the status updates, it sounds like this either affected some core infra shared between AZs, or was from a change rolled out safely previously (maybe last night) and failed later (eg due to increased load in the morning).

At the end of the day, building redundant systems is expensive (and may introduce whole new bugs), so they probably did the math and figured the risk of a whole region outage was less than the cost of building redundancy in some systems that are hard to make redundant.

Re: Amazon packages pile up after AWS outage spawns delivery havoc

#124
This really tarnishes any trust you might have had in Amazon as a professionally run business.

(of course outages can happen and are to be expected especially at Amazon's scale, it's just the bad communication and the amateurish non-redundant setup of their own core services that is shocking)

Re: Amazon packages pile up after AWS outage spawns delivery havoc

#125
post #89
post #18

Check out the Amazon Flex Drivers subreddit[0], there are tons of people saying they were paid for the day of deliveries without delivering any packages and told to go home. [0] https://www.reddit.com/r/AmazonFlexDrivers/comments/rb3ggn/i...

That's actually pretty nice though; even though people are missing out on packages at least Amazon won't leave them out in the cold for today.

My inner cynic says Amazon paid Flex drivers because A) they still need drivers available during the christmas run-up and B) screwing drivers would put Amazon in an Ebenezer Scrooge-like PR disaster.

Re: Amazon packages pile up after AWS outage spawns delivery havoc

#126

As bad as this article makes AWS sound, it's actually the reason you should go with AWS over say Azure or GCP; when AWS goes down, its owners actually feel the pain with you, Microsoft and Google run their own stuff elsewhere...

No, not true at all. What do you think Google and Microsoft run their cloud services on?

AWS, of course

Re: Amazon packages pile up after AWS outage spawns delivery havoc

#127

Whenever us-east-1 goes down you just get a really good feel for how many other companies also have pretty fragile setups. They apps I work on can deal with a few hours of downtime, so as long as I'm sure I can recover from getting totally leveled its ok. And I think it's that was for the majority of companies. Most don't want the extra effort and cost of failover.

I'm still surprised how many people are still using us-east-1.

Re: Amazon packages pile up after AWS outage spawns delivery havoc

#128
post #89

Earlier quoted context omitted.

That's actually pretty nice though; even though people are missing out on packages at least Amazon won't leave them out in the cold for today.

My inner cynic says Amazon paid Flex drivers because A) they still need drivers available during the christmas run-up and B) screwing drivers would put Amazon in an Ebenezer Scrooge-like PR disaster.

Amazon in a PR disaster involving corporate greed and avarice? Oh no. Anyway. No really, I would not be surprised if Amazon has several playbooks upon playbooks involving commercials of how Amazon helped some warehouse worker get through college or some medical bill as their means of a public mea culpa.

Re: Amazon packages pile up after AWS outage spawns delivery havoc

#129

Whenever us-east-1 goes down you just get a really good feel for how many other companies also have pretty fragile setups. They apps I work on can deal with a few hours of downtime, so as long as I'm sure I can recover from getting totally leveled its ok. And I think it's that was for the majority of companies. Most don't want the extra effort and cost of failover.

I'm just surprised Amazon was not using something sectioned off from the rest of AWS.

[deleted]
Post reply on HN