Live data from Hacker News

Amazon packages pile up after AWS outage spawns delivery havoc

detroitnews.com

41–50 of 219 posts

Re: Amazon packages pile up after AWS outage spawns delivery havoc

#41
post #14
post #7

I thought the whole point of AWS that it was a distributed system with no one point of failure? In that case, how did it have an outage?

do they not teach people what a failure domain is anymore? https://en.wikipedia.org/wiki/Failure_domain Clouds aren't magic. They require a certain amount of operational confidence in order to understand that, yes, an entire region can fall out from under you at any time and it's your responsibility to detect and deploy into an unaffected region if possible. edit: Generally, one entire region will not fail. However,…

People get slagged on for not having inter-region redundancy. But unless your business model can accommodate that, you’re introducing another failure domain and alot of money for duplicative infrastructure, network fees, etc.

For many use cases, it’s acceptable to shrug and blame AWS for a failure. It’s harder when your high availability solution fails independently, which they almost always do more than US-East-1

Re: Amazon packages pile up after AWS outage spawns delivery havoc

#43
post #32

Earlier quoted context omitted.

When a residential house becomes a review farm.

when the house is becoming an amazon warehouse. This is crazy, i can't believe what the home owners thought of so many many boxes coming there.

There was a similar story on my local news as well. Someone's front yard filled with hundreds of packages.

Re: Amazon packages pile up after AWS outage spawns delivery havoc

#44
post #16

Earlier quoted context omitted.

Region outages happen, which is why the guidance is to build in multiple regions, but even Amazon sometimes doesn't take their own advice. Sometimes the reason is good, sometimes it isn't.

I thought the guidance was to be multi-AZ, as an AZ is the failure boundary?

AZs are physical boundaries, but the networking and software is interconnected. Regions are (mostly) isolated, though global services like IAM and CloudFront often have their main control plane in us-east-1

Re: Amazon packages pile up after AWS outage spawns delivery havoc

#45
post #16

Earlier quoted context omitted.

Region outages happen, which is why the guidance is to build in multiple regions, but even Amazon sometimes doesn't take their own advice. Sometimes the reason is good, sometimes it isn't.

I thought the guidance was to be multi-AZ, as an AZ is the failure boundary?

Everyone wants to be multi-AZ, multi-cell, but it's a multi-year project, especially for services that have been around for a while. My last team had been working on it for a couple of years when I left.

Re: Amazon packages pile up after AWS outage spawns delivery havoc

#46
post #24
post #17

If this outage isn’t the catalyst to get the Amazon side of the house to finally move out of US-east, I don’t know what will be. Or at least be multi-region. Although the cost to make all of Amazon commerce, logistics, and digital truly multi-region is probably an order of magnitude more than the impact of this outage.

True. But let's be honest... this is neither the first such outage in US-east-1 nor the last. So I'd argue it's long past time for Amazon to pay the bill and go multi-region.

Last big e-commerce outage I can think of was Prime day years ago when Sable browned out. But that was non-AWS infrastructure. There was also the infamous s3 fat-finger; but I can’t think of a holiday shopping and delivery day with a massive outage.

Re: Amazon packages pile up after AWS outage spawns delivery havoc

#48
post #28

I thought this story was about this house x-) "Neighbors in Tennessee city worry as Amazon packages pile up outside home" https://www.youtube.com/watch?v=qVQjEB2sxBw

I don't even understand what's going on with this one. Transcribing the blurb from the clip, from interviewing the residents of the house:

"[The people who live in the house] have a friend who has a contract with an Amazon warehouse in China. They say whenever that friend's contract expires, she will send the packages to their house for the family to sort and then send back to Amazon for the company to sell."

Re: Amazon packages pile up after AWS outage spawns delivery havoc

#49
post #16

Earlier quoted context omitted.

Region outages happen, which is why the guidance is to build in multiple regions, but even Amazon sometimes doesn't take their own advice. Sometimes the reason is good, sometimes it isn't.

I thought the guidance was to be multi-AZ, as an AZ is the failure boundary?

You can always do better because failures can always be bigger/wider. multi-instance < multi-AZ < multi-region < multiple vendors

Re: Amazon packages pile up after AWS outage spawns delivery havoc

#50
post #28

I thought this story was about this house x-) "Neighbors in Tennessee city worry as Amazon packages pile up outside home" https://www.youtube.com/watch?v=qVQjEB2sxBw

When a residential house becomes a review farm.

More like dropshipping gone wrong.
Post reply on HN