Live data from Hacker News

Amazon packages pile up after AWS outage spawns delivery havoc

detroitnews.com

201–210 of 219 posts

Re: Amazon packages pile up after AWS outage spawns delivery havoc

#201

Earlier quoted context omitted.

5 years ago you couldn't even start your car remotely, and many people lived in Alaska, this is such a first world problem lmao.

> 5 years ago you couldn't even start your car remotely Of course you could. There have been remote car starters for 20+ years. They don't rely on internet nonsense to work either.

Also block heaters are a thing

Re: Amazon packages pile up after AWS outage spawns delivery havoc

#202

Earlier quoted context omitted.

A lot of AWS is very old and it's very hard to go back and make everything nice and reliable without breaking things when you have a shitload of customers 24/7 using your services. So yes, in theory much of AWS's services are probably very reliable and distributed across AZs and regions, but in practice there's likely a whole bunch of debt where one thing gets fucked up and it cascades.

As an AWS employee, you’re under too much artificial deadline pressure to address technical debt like this… let alone engineer a service properly in the first place.

As a corporate employee, you’re under too much artificial deadline pressure to address technical debt like this… let alone engineer a service properly in the first place.

Re: Amazon packages pile up after AWS outage spawns delivery havoc

#203
post #92

Something I'm not getting there. OK, one zone went down in AWS. But there supposed to be multiple ones? And if somebody knows how to use AWS so that one zone going down wouldn't knock over the whole thing, it should be Amazon? I thought the whole point of this multi-zone cloud setup would be so that even if one zone goes down, it could survive and perform as usual - maybe a little slower for a bit while backups kick…

This affected on region, us-east-1, which is the oldest and largest aws region, and also hosts many core aws services. Each region has several availability zones (AZs) that are basically whole datacenters. In theory AZs should be mostly isolated, but as we saw bugs happen. Reading between the lines of the status updates, it sounds like this either affected some core infra shared between AZs, or was from a change roll…

The whole concept that there are "core AWS services" living in one single zone sounds like anti-thesis to everything AWS should be about. What's the use of having all this nice distributed setup - and paying for it! - if a single failure in a single zone takes everything down anyway? I mean sure, maybe they built it on a shoe-string budget years ago - but since then, they had years and billions, and still didn't bother to fix it?

Re: Amazon packages pile up after AWS outage spawns delivery havoc

#204
post #110

Weird I just ordered a package about 3-4 weeks ago and it didn’t show up so finally went to track it on amazon last week and got a message saying that they had lost the package and to request a refund. It kind of sucks because that was my mom’s Christmas gift now I have to figure out a plan B.

I have this happen constantly through Amazon. Now all the hassle is on you to get the refund. And they won't process the refund immediately, so you have to pay out of your own money to get a replacement. And then often, as I've found, the price of the item has now gone up too, so you have to pay extra.

p.s. Your username?! I can think this must be the only site you've managed to get that handle?

Re: Amazon packages pile up after AWS outage spawns delivery havoc

#205
post #20
post #16

Earlier quoted context omitted.

I thought the guidance was to be multi-AZ, as an AZ is the failure boundary?

AZs are physically located near each other, usually within a small enough radius that they could be all impacted by the same natural disaster. In some cloud providers and regions, AZs are simply different parts of the same building (IIRC one of the Japan regions of Azure was essentially this, but don't quote me). And evidently, the share some infrastructure. At a previous job where we needed to always be up, our disa…

A meteor did not hit us-east-1 yesterday.

Re: Amazon packages pile up after AWS outage spawns delivery havoc

#206

Whenever us-east-1 goes down you just get a really good feel for how many other companies also have pretty fragile setups. They apps I work on can deal with a few hours of downtime, so as long as I'm sure I can recover from getting totally leveled its ok. And I think it's that was for the majority of companies. Most don't want the extra effort and cost of failover.

The frightening thing is how fragile it seems. If you're going to concentrate risk on AWS, it better be essentially flawless, stable, and highly-redundant.

It is, most of the time. That's why these outages are such a big deal. Nobody would care if it went out multiple times a month.

Re: Amazon packages pile up after AWS outage spawns delivery havoc

#207
post #205
post #20

Earlier quoted context omitted.

AZs are physically located near each other, usually within a small enough radius that they could be all impacted by the same natural disaster. In some cloud providers and regions, AZs are simply different parts of the same building (IIRC one of the Japan regions of Azure was essentially this, but don't quote me). And evidently, the share some infrastructure. At a previous job where we needed to always be up, our disa…

A meteor did not hit us-east-1 yesterday.

You misunderstand- our planning was to prepare for a scenario of that extremity.

Re: Amazon packages pile up after AWS outage spawns delivery havoc

#208

Earlier quoted context omitted.

It's amazing to me that people think they're adding something to the conversation by posting the most banal response. "Yes, the dumb broad didn't think to walk outside and turn the car on! I shall right this wrong with my clever internet post! Behold my intelligence!"

To be fair if someone was using the “app didn’t work” excuse multiple times, suspicion could be raised. But for a one off occurrence? Why would you assume a car company knows what they’re doing over the person telling the story? It’s silly.

There is a certain personality type common among midwits who are smart enough to think of the most obvious workaround/solution/exception but not smart enough to realize that when people communicate, especially online, they usually choose brevity over exactness.

Like if I say "Humans have two feet" some midwit will come along with an article or anecdote about a person who was born without two feet.

And if I say the multi-hour outage of AWS made my friend minutes late because of her car warmer, what is going through the mind of a person who offers the solution "Did she try walking to the car and turning it on manually?" These are the kinds of people that if you met them in real life you'd quickly distance yourself from them.

Re: Amazon packages pile up after AWS outage spawns delivery havoc

#209

Whenever us-east-1 goes down you just get a really good feel for how many other companies also have pretty fragile setups. They apps I work on can deal with a few hours of downtime, so as long as I'm sure I can recover from getting totally leveled its ok. And I think it's that was for the majority of companies. Most don't want the extra effort and cost of failover.

I'm almost tempted to think that having an explicit policy of forcibly shutting down each region once per month for a few hours (at times that are not publicly announced in advance) would be a worthwhile value-add. A service that is unable to handle such a failure does not qualify as being ready for deployment. And I'm not just saying that to be a self-righteous pedant, I'm saying it because this kind of failure is s…

Most services and applications can occasionally have unplanned downtime. Many have also planed downtimes sometimes measured in half a day or more.

Services that truly need 100% uptime (and by that I don't mean what management say they WANT, but they are prepared to PAY) are a tiny minority, to the point that I imagine most software developers never work on one.

Even setups that i have seen to handle such cases, usually had some single point of failure somewhere.

And lest face it. Even if you do multi region on AWS, that won't protect you next time someone screws up BGP/switches/DNS or whatever and every region goes down for a while. In that case you better have failover to some other cloud vendor. Even planes or other safety critical equipment is not 100% failure proof.

Re: Amazon packages pile up after AWS outage spawns delivery havoc

#210
post #110

Weird I just ordered a package about 3-4 weeks ago and it didn’t show up so finally went to track it on amazon last week and got a message saying that they had lost the package and to request a refund. It kind of sucks because that was my mom’s Christmas gift now I have to figure out a plan B.

I have this happen constantly through Amazon. Now all the hassle is on you to get the refund. And they won't process the refund immediately, so you have to pay out of your own money to get a replacement. And then often, as I've found, the price of the item has now gone up too, so you have to pay extra. p.s. Your username?! I can think this must be the only site you've managed to get that handle?

It is the only site. Someone once had an even lower number and someone made mention to it and the person replied saying it was easy and he had just gotten it so I decided to try and see what was available. I had been a long term lurker and never posted but once I got this username I try and respectfully post when I can.
Post reply on HN