Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Amazon packages pile up after AWS outage spawns delivery havoc

detroitnews.com

Post reply on HN
Print view View original

131–140 of 219 posts

Re: Amazon packages pile up after AWS outage spawns delivery havoc

#131

Post by notatoad »

chrisseaton wrote: post #65
sneak wrote: post #16

Earlier quoted context omitted.

I thought the guidance was to be multi-AZ, as an AZ is the failure boundary?

AZs are in one physical location. If that physical location has a problem, all AZs within it will go down.

>An Availability Zone (AZ) is one or more discrete data centers with redundant power, networking, and connectivity in an AWS Region... AZs are physically separated by a meaningful distance, many kilometers, from any other AZ, although all are within 100 km (60 miles) of each other.

https://aws.amazon.com/about-aws/global-infrastructure/regio...

Re: Amazon packages pile up after AWS outage spawns delivery havoc

#132

Post by errcorrectcode »

karmicthreat wrote: post #90

Whenever us-east-1 goes down you just get a really good feel for how many other companies also have pretty fragile setups. They apps I work on can deal with a few hours of downtime, so as long as I'm sure I can recover from getting totally leveled its ok. And I think it's that was for the majority of companies. Most don't want the extra effort and cost of failover.

The frightening thing is how fragile it seems.

If you're going to concentrate risk on AWS, it better be essentially flawless, stable, and highly-redundant.

Re: Amazon packages pile up after AWS outage spawns delivery havoc

#133

Post by notatoad »

sneak wrote: post #16
andrewguenther wrote: post #15

Earlier quoted context omitted.

Region outages happen, which is why the guidance is to build in multiple regions, but even Amazon sometimes doesn't take their own advice. Sometimes the reason is good, sometimes it isn't.

I thought the guidance was to be multi-AZ, as an AZ is the failure boundary?

I think it'd be correct to call the AZ a failure boundary, not the failure boundary. the is hardly the first time a failure has exceeded an AZ.

Re: Amazon packages pile up after AWS outage spawns delivery havoc

#134

Post by professoretc »

karmicthreat wrote: post #90

Whenever us-east-1 goes down you just get a really good feel for how many other companies also have pretty fragile setups. They apps I work on can deal with a few hours of downtime, so as long as I'm sure I can recover from getting totally leveled its ok. And I think it's that was for the majority of companies. Most don't want the extra effort and cost of failover.

It's finals season, so all the students taking online exams on Canvas had a fun day.

Re: Amazon packages pile up after AWS outage spawns delivery havoc

#135

Post by phamilton »

Spooky23 wrote: post #41
riffic wrote: post #14

Earlier quoted context omitted.

do they not teach people what a failure domain is anymore? https://en.wikipedia.org/wiki/Failure_domain Clouds aren't magic. They require a certain amount of operational confidence in order to understand that, yes, an entire region can fall out from under you at any time and it's your responsibility to detect and deploy into an unaffected region if possible. edit: Generally, one entire region will not fail. However,…

People get slagged on for not having inter-region redundancy. But unless your business model can accommodate that, you’re introducing another failure domain and alot of money for duplicative infrastructure, network fees, etc. For many use cases, it’s acceptable to shrug and blame AWS for a failure. It’s harder when your high availability solution fails independently, which they almost always do more than US-East-1

If your site goes down while everyone else's goes down, it's weirdly forgivable.

And being the only one up doesn't win as many market cred points and you'd think.

Re: Amazon packages pile up after AWS outage spawns delivery havoc

#136

Post by tyingq »

smsm42 wrote: post #92

Something I'm not getting there. OK, one zone went down in AWS. But there supposed to be multiple ones? And if somebody knows how to use AWS so that one zone going down wouldn't knock over the whole thing, it should be Amazon? I thought the whole point of this multi-zone cloud setup would be so that even if one zone goes down, it could survive and perform as usual - maybe a little slower for a bit while backups kick…

There's a lot of pitfalls. You have to diligent about stuff like making sure you're using region-specific endpoints like sts.us-west-2.amazonaws.com instead of sts.amazonaws.com. And some services, like Route53 and Cloudfront have built-in dependencies on us-east-1. There's also internal dependencies that are sometimes not obvious until an outage occurs.

Though, as you say, none of that's a great excuse for Amazon itself. I know Alexa devices, Ring Devices, Prime Video, imdb.com, etc, also all had issues in the first hours of this outage.

Re: Amazon packages pile up after AWS outage spawns delivery havoc

#138

Post by Hamuko »

riffic wrote: post #9

correlation and causation, anyone? You'd think if it's anything mission critical, Amazon would be following its very own Well-Architected Framework. The Reliability pillar would speak to this.

I don't think the Well-Architected framework specifies that you should use multi-region availability, but it's certainly a mentioned option for mission critical applications. Usually the go-to doctorine for high availability in AWS documentation is multi-AZ, not multi-region.

https://docs.aws.amazon.com/wellarchitected/latest/reliabili...

Re: Amazon packages pile up after AWS outage spawns delivery havoc

#139

Post by techie128 »

laluser wrote: post #86
EarlKing wrote: post #67

Earlier quoted context omitted.

Wouldn't surprise me. I mean, DSPs wouldn't be able to scan packages to pick them up without being able to access the flex app, so unless there's a procedure in place to allow packages to be picked up and manually marked as having been picked up... they wouldn't be able to do anything. Moreover, even if they could pick anything up, they wouldn't have any way to navigate, drop packages off, snap photos, and otherwise…

Failing over to another AWS region is actually pretty difficult for stateful services. Especially if you can't even access data in the primary region at all. Most teams probably don't have the bandwidth to solve this problem given the amount of outages you see in a year (1 or 2). Also, this would be a problem many teams would be solving, so most teams probably just wait and see what leaders have to say about it and w…

Databases, stateful part of services, have matured to have multi-region support. This isn't new either. Nobody's saying that its easy to have multi-region redundancy for stateful services. Its just something you need to have to prevent nasty single region outages affect your service. This is an excellent example where it would have been better to have degraded performance (higher latency) instead of complete unavailability and interruption in business.

Re: Amazon packages pile up after AWS outage spawns delivery havoc

#140

Post by WoahNoun »

admax88qqq wrote: post #71
pyrophane wrote: post #68

Earlier quoted context omitted.

No, not true at all. What do you think Google and Microsoft run their cloud services on?

Google does not run the bulk of their services on GCP. Unlike AWS, GCP was not a productization of their existing infrastructure, but rather a separate cloud product developed fairly independently. I'm sure that's changing with time. YouTube also has its own infrastructure independent of GCP and the rest of Google

>GCP was not a productization of their existing infrastructure

Neither was AWS.

Post reply on HN