Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Amazon packages pile up after AWS outage spawns delivery havoc

detroitnews.com

Post reply on HN
Print view View original

111–120 of 219 posts

Re: Amazon packages pile up after AWS outage spawns delivery havoc

#111

Post by mdavidn »

laluser wrote: post #86
EarlKing wrote: post #67

Earlier quoted context omitted.

Wouldn't surprise me. I mean, DSPs wouldn't be able to scan packages to pick them up without being able to access the flex app, so unless there's a procedure in place to allow packages to be picked up and manually marked as having been picked up... they wouldn't be able to do anything. Moreover, even if they could pick anything up, they wouldn't have any way to navigate, drop packages off, snap photos, and otherwise…

Failing over to another AWS region is actually pretty difficult for stateful services. Especially if you can't even access data in the primary region at all. Most teams probably don't have the bandwidth to solve this problem given the amount of outages you see in a year (1 or 2). Also, this would be a problem many teams would be solving, so most teams probably just wait and see what leaders have to say about it and w…

A one-day outage in December can be crippling for retail.

I don't doubt that many functions are difficult to failover, but a bare-bones minimum seems straightforward. For example, evidence of delivery is append-only and only needs to be globally consistent later, after a dispute.

Re: Amazon packages pile up after AWS outage spawns delivery havoc

#112

Post by anamexis »

throwntoday wrote: post #106
karmicthreat wrote: post #90

Whenever us-east-1 goes down you just get a really good feel for how many other companies also have pretty fragile setups. They apps I work on can deal with a few hours of downtime, so as long as I'm sure I can recover from getting totally leveled its ok. And I think it's that was for the majority of companies. Most don't want the extra effort and cost of failover.

I'm just surprised Amazon was not using something sectioned off from the rest of AWS.

Possibly stems from one of the tenets of the apocryphal Bezos decree that started AWS[1]

> All service interfaces, without exception, must be designed from the ground up to be externalizable. That is to say, the team must plan and design to be able to expose the interface to developers in the outside world. No exceptions.

1: https://nordicapis.com/the-bezos-api-mandate-amazons-manifes...

Re: Amazon packages pile up after AWS outage spawns delivery havoc

#113

Post by hinkley »

bombcar wrote: post #34
cbtacy wrote: post #21

Earlier quoted context omitted.

Because it's terrible and has always been terrible? After 2017 I'll never use US-east-1 again. Hell... I should have learned that particular lesson in 2011 but it took two catastrophic failures for me to figure it out. There are numerous threads here on HN covering the topic "why does US-east-1 suck so hard." https://news.ycombinator.com/item?id=13756082 is just one example.

Maybe it’s terrible because they host their own stuff there.

I wonder if there’s too much moral hazard for them to run a data center just for Amazon but run exactly the same as the other data centers? Or would Amazon scope creep it into its own snowflake.

I kind of doubt that they’re getting efficient use of hardware Amazon could be using but isn’t, since if someone allocates it it’s not available for Amazon anymore.

Re: Amazon packages pile up after AWS outage spawns delivery havoc

#114

Post by matteotom »

throwntoday wrote: post #106
karmicthreat wrote: post #90

Whenever us-east-1 goes down you just get a really good feel for how many other companies also have pretty fragile setups. They apps I work on can deal with a few hours of downtime, so as long as I'm sure I can recover from getting totally leveled its ok. And I think it's that was for the majority of companies. Most don't want the extra effort and cost of failover.

I'm just surprised Amazon was not using something sectioned off from the rest of AWS.

My understanding is there's a pretty strong culture of running amazon.com directly in AWS as much as possible. I don't think there's much reason to believe a sectioned off alternative would be much more reliable, it would just break at different times maybe.

Re: Amazon packages pile up after AWS outage spawns delivery havoc

#115

Post by hinkley »

cbtacy wrote: post #24
ABeeSea wrote: post #17

If this outage isn’t the catalyst to get the Amazon side of the house to finally move out of US-east, I don’t know what will be. Or at least be multi-region. Although the cost to make all of Amazon commerce, logistics, and digital truly multi-region is probably an order of magnitude more than the impact of this outage.

True. But let's be honest... this is neither the first such outage in US-east-1 nor the last. So I'd argue it's long past time for Amazon to pay the bill and go multi-region.

I wonder if their system would allow them to move availability zones between regions. They could create new stealth AZs, move their stuff into them, then use those to start building a new region only they use.

But I suspect there are third part integrations that benefit from being on the same AZ as Amazon APIs. I’ve been having little convos all day about how I think the inter region pricing creates a perverse incentive that’s exacerbating the us-east-1 situation.

Re: Amazon packages pile up after AWS outage spawns delivery havoc

#116

Post by tyingq »

authed wrote: post #62
slimsag wrote: post #18

Check out the Amazon Flex Drivers subreddit[0], there are tons of people saying they were paid for the day of deliveries without delivering any packages and told to go home. [0] https://www.reddit.com/r/AmazonFlexDrivers/comments/rb3ggn/i...

I read many comments and didn't see one that was saying that but you might be right as I was too lazy to read everything.

home finds them pretty quick for me.

Re: Amazon packages pile up after AWS outage spawns delivery havoc

#117

Post by andrewia »

moneywoes wrote: post #35

Will Amazon compensate all of their clients?

If there's an SLA or reputation to uphold, probably, and otherwise they probably won't. So big AWS customers with SLAs will probably get compensation. And Amazon dot com customers that complain might get some compensation.

Re: Amazon packages pile up after AWS outage spawns delivery havoc

#118

Post by blhack »

aschla wrote: post #37

Here in Chicago every item on Amazon is showing an earliest delivery date of Saturday the 11th. Checked an item today I had bought yesterday and delivered overnight, the item is now showing available delivery as Saturday. What a terrible time of the year for this to happen to them.

Exact same here in Phoenix.

Re: Amazon packages pile up after AWS outage spawns delivery havoc

#119

Post by aenis »

tw04 wrote: post #59
faridelnasire wrote: post #51

As bad as this article makes AWS sound, it's actually the reason you should go with AWS over say Azure or GCP; when AWS goes down, its owners actually feel the pain with you, Microsoft and Google run their own stuff elsewhere...

> Microsoft and Google run their own stuff elsewhere... That is simply not true. https://www.zdnet.com/article/microsoft-moves-closer-to-runn...

true for google though, with their borg-thingy being separate from gcp last time i asked them

Re: Amazon packages pile up after AWS outage spawns delivery havoc

#120

Post by tekno45 »

smsm42 wrote: post #92

Something I'm not getting there. OK, one zone went down in AWS. But there supposed to be multiple ones? And if somebody knows how to use AWS so that one zone going down wouldn't knock over the whole thing, it should be Amazon? I thought the whole point of this multi-zone cloud setup would be so that even if one zone goes down, it could survive and perform as usual - maybe a little slower for a bit while backups kick…

This zone hosts the infra to manage the rest of the zones. DNS management is also here and can easily cause a global failure.
Post reply on HN