Earlier quoted context omitted.
5 years ago you couldn't even start your car remotely, and many people lived in Alaska, this is such a first world problem lmao.
> 5 years ago you couldn't even start your car remotely Of course you could. There have been remote car starters for 20+ years. They don't rely on internet nonsense to work either.
Amazon packages pile up after AWS outage spawns delivery havoc
201–210 of 219 posts
Re: Amazon packages pile up after AWS outage spawns delivery havoc
#202Earlier quoted context omitted.
A lot of AWS is very old and it's very hard to go back and make everything nice and reliable without breaking things when you have a shitload of customers 24/7 using your services. So yes, in theory much of AWS's services are probably very reliable and distributed across AZs and regions, but in practice there's likely a whole bunch of debt where one thing gets fucked up and it cascades.
As an AWS employee, you’re under too much artificial deadline pressure to address technical debt like this… let alone engineer a service properly in the first place.
Re: Amazon packages pile up after AWS outage spawns delivery havoc
#203Something I'm not getting there. OK, one zone went down in AWS. But there supposed to be multiple ones? And if somebody knows how to use AWS so that one zone going down wouldn't knock over the whole thing, it should be Amazon? I thought the whole point of this multi-zone cloud setup would be so that even if one zone goes down, it could survive and perform as usual - maybe a little slower for a bit while backups kick…
This affected on region, us-east-1, which is the oldest and largest aws region, and also hosts many core aws services. Each region has several availability zones (AZs) that are basically whole datacenters. In theory AZs should be mostly isolated, but as we saw bugs happen. Reading between the lines of the status updates, it sounds like this either affected some core infra shared between AZs, or was from a change roll…
Re: Amazon packages pile up after AWS outage spawns delivery havoc
#204Weird I just ordered a package about 3-4 weeks ago and it didn’t show up so finally went to track it on amazon last week and got a message saying that they had lost the package and to request a refund. It kind of sucks because that was my mom’s Christmas gift now I have to figure out a plan B.
p.s. Your username?! I can think this must be the only site you've managed to get that handle?
Re: Amazon packages pile up after AWS outage spawns delivery havoc
#205Earlier quoted context omitted.
I thought the guidance was to be multi-AZ, as an AZ is the failure boundary?
AZs are physically located near each other, usually within a small enough radius that they could be all impacted by the same natural disaster. In some cloud providers and regions, AZs are simply different parts of the same building (IIRC one of the Japan regions of Azure was essentially this, but don't quote me). And evidently, the share some infrastructure. At a previous job where we needed to always be up, our disa…
Re: Amazon packages pile up after AWS outage spawns delivery havoc
#206Whenever us-east-1 goes down you just get a really good feel for how many other companies also have pretty fragile setups. They apps I work on can deal with a few hours of downtime, so as long as I'm sure I can recover from getting totally leveled its ok. And I think it's that was for the majority of companies. Most don't want the extra effort and cost of failover.
The frightening thing is how fragile it seems. If you're going to concentrate risk on AWS, it better be essentially flawless, stable, and highly-redundant.
Re: Amazon packages pile up after AWS outage spawns delivery havoc
#207Earlier quoted context omitted.
AZs are physically located near each other, usually within a small enough radius that they could be all impacted by the same natural disaster. In some cloud providers and regions, AZs are simply different parts of the same building (IIRC one of the Japan regions of Azure was essentially this, but don't quote me). And evidently, the share some infrastructure. At a previous job where we needed to always be up, our disa…
A meteor did not hit us-east-1 yesterday.
Re: Amazon packages pile up after AWS outage spawns delivery havoc
#208Earlier quoted context omitted.
It's amazing to me that people think they're adding something to the conversation by posting the most banal response. "Yes, the dumb broad didn't think to walk outside and turn the car on! I shall right this wrong with my clever internet post! Behold my intelligence!"
To be fair if someone was using the “app didn’t work” excuse multiple times, suspicion could be raised. But for a one off occurrence? Why would you assume a car company knows what they’re doing over the person telling the story? It’s silly.
Like if I say "Humans have two feet" some midwit will come along with an article or anecdote about a person who was born without two feet.
And if I say the multi-hour outage of AWS made my friend minutes late because of her car warmer, what is going through the mind of a person who offers the solution "Did she try walking to the car and turning it on manually?" These are the kinds of people that if you met them in real life you'd quickly distance yourself from them.
Re: Amazon packages pile up after AWS outage spawns delivery havoc
#209Whenever us-east-1 goes down you just get a really good feel for how many other companies also have pretty fragile setups. They apps I work on can deal with a few hours of downtime, so as long as I'm sure I can recover from getting totally leveled its ok. And I think it's that was for the majority of companies. Most don't want the extra effort and cost of failover.
I'm almost tempted to think that having an explicit policy of forcibly shutting down each region once per month for a few hours (at times that are not publicly announced in advance) would be a worthwhile value-add. A service that is unable to handle such a failure does not qualify as being ready for deployment. And I'm not just saying that to be a self-righteous pedant, I'm saying it because this kind of failure is s…
Services that truly need 100% uptime (and by that I don't mean what management say they WANT, but they are prepared to PAY) are a tiny minority, to the point that I imagine most software developers never work on one.
Even setups that i have seen to handle such cases, usually had some single point of failure somewhere.
And lest face it. Even if you do multi region on AWS, that won't protect you next time someone screws up BGP/switches/DNS or whatever and every region goes down for a while. In that case you better have failover to some other cloud vendor. Even planes or other safety critical equipment is not 100% failure proof.
Re: Amazon packages pile up after AWS outage spawns delivery havoc
#210Weird I just ordered a package about 3-4 weeks ago and it didn’t show up so finally went to track it on amazon last week and got a message saying that they had lost the package and to request a refund. It kind of sucks because that was my mom’s Christmas gift now I have to figure out a plan B.
I have this happen constantly through Amazon. Now all the hassle is on you to get the refund. And they won't process the refund immediately, so you have to pay out of your own money to get a replacement. And then often, as I've found, the price of the item has now gone up too, so you have to pay extra. p.s. Your username?! I can think this must be the only site you've managed to get that handle?