Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Amazon packages pile up after AWS outage spawns delivery havoc

detroitnews.com

Post reply on HN
Print view View original

171–180 of 219 posts

Re: Amazon packages pile up after AWS outage spawns delivery havoc

#171

Post by X3AY7yZfpyWCmf »

pojzon wrote: post #148
techie128 wrote: post #139

Earlier quoted context omitted.

Databases, stateful part of services, have matured to have multi-region support. This isn't new either. Nobody's saying that its easy to have multi-region redundancy for stateful services. Its just something you need to have to prevent nasty single region outages affect your service. This is an excellent example where it would have been better to have degraded performance (higher latency) instead of complete unavaila…

Tell that to management of a medium company showing them the bill for something that has 0.01% chance of happening according to AWS.. Not every workload is of the micro size. MR up our databases would cost around 15mil per year for a company that makes 50mil..

I don't think it's correct to call Amazon a "medium company" and last time I checked they make more than 50mil

Re: Amazon packages pile up after AWS outage spawns delivery havoc

#172

Post by SpelingBeeChamp »

hef19898 wrote: post #147
mdavidn wrote: post #111

Earlier quoted context omitted.

A one-day outage in December can be crippling for retail. I don't doubt that many functions are difficult to failover, but a bare-bones minimum seems straightforward. For example, evidence of delivery is append-only and only needs to be globally consistent later, after a dispute.

I totally understand why Amazon halted everything. Sure, one could deliver shipments off-line and sort them out afterwards manually. And at a lower scale Amazon might have tried it (at least it would have been on the table when I worked there a couple of years ago). But what then? You had a complete loss of traceability of shipments and operations, once you regain it, a junk of shipments isn't there anymore where the…

Any idea if Amazon has insurance to cover this type of event?

Re: Amazon packages pile up after AWS outage spawns delivery havoc

#173

Post by hef19898 »

SpelingBeeChamp wrote: post #172
hef19898 wrote: post #147

Earlier quoted context omitted.

I totally understand why Amazon halted everything. Sure, one could deliver shipments off-line and sort them out afterwards manually. And at a lower scale Amazon might have tried it (at least it would have been on the table when I worked there a couple of years ago). But what then? You had a complete loss of traceability of shipments and operations, once you regain it, a junk of shipments isn't there anymore where the…

Any idea if Amazon has insurance to cover this type of event?

Can't speak for Amazon. Generally so I don't see how one could insure against it. I know that e.g. Allianz offers policies against IT outages. In that case so, what is the actual damage? Probably the delivery drivers paid without delivering and salaries, plus potential overtime to solve the backlog. Depending on the conditions a company the size of Amazon would get, maybe it's not worth it.

Re: Amazon packages pile up after AWS outage spawns delivery havoc

#174

Post by stdbrouw »

peterkelly wrote: post #143
karmicthreat wrote: post #90

Whenever us-east-1 goes down you just get a really good feel for how many other companies also have pretty fragile setups. They apps I work on can deal with a few hours of downtime, so as long as I'm sure I can recover from getting totally leveled its ok. And I think it's that was for the majority of companies. Most don't want the extra effort and cost of failover.

I'm almost tempted to think that having an explicit policy of forcibly shutting down each region once per month for a few hours (at times that are not publicly announced in advance) would be a worthwhile value-add. A service that is unable to handle such a failure does not qualify as being ready for deployment. And I'm not just saying that to be a self-righteous pedant, I'm saying it because this kind of failure is s…

Smallish non-tech companies typically have single points of failure that are much more severe than a server/service being down for a couple of hours. Accountant on holiday? Guess we're not writing any invoices this week. Two of our drivers are ill? Guess we'll have to let all of this stuff pile up here for a bit. By comparison, a public-facing service or intranet being down for a few hours hardly merits attention.

Re: Amazon packages pile up after AWS outage spawns delivery havoc

#175

Post by stdbrouw »

peterkelly wrote: post #150
kall wrote: post #144

Earlier quoted context omitted.

At the project level maybe. At the AWS level, that‘s ridiculous. A lot of services can live with one major outage a year, but not once a month. Completely failing when their region goes down is reasonable.

Region failures can be non-recoverable (e.g. natural disaster). One major outage per year doesn't necessarily mean just a few hours of downtime, it could mean having to redeploy your entire service somewhere else, which could take several days or more if you haven't prepared for it. How many of those services can live with that?

> How many of those services can live with that?

Apparently, many can! Look at how many companies choose not to pay ransom when hit with a ransomware attack, or prefer to negotiate for days instead of buckling straight away, even if it means operations are crippled for weeks. They don't typically go bankrupt afterwards, everyone coped, life moves on.

Re: Amazon packages pile up after AWS outage spawns delivery havoc

#176

Post by vineyardmike »

johncena33 wrote: post #94
faridelnasire wrote: post #51

As bad as this article makes AWS sound, it's actually the reason you should go with AWS over say Azure or GCP; when AWS goes down, its owners actually feel the pain with you, Microsoft and Google run their own stuff elsewhere...

HN is a very pro-Amazon place. Regardless of what Amazon does, at least one of the top 3 comments is always justifying Amazon's actions.

That’s not the impression I got from the last 24 hours, nor the last few years.

1. Scale is hard and downtime is hard, HNers either recognize the struggle or appreciate their lack of experience. When AWS fails many armchair architects come out to suggest solutions but many more techies just sympathize with the Amazonians.

2. Technically, Amazon has built something impressive. It might not be what you want, or what others have, but AWS is impressive in scale and scope and even reliability. Many people share credit where due.

3. One can criticize the treatment of warehouse and delivery workers that Amazon is known for, but this has little baring on the tech workers there nor AWS generally. So AWS stories tend to be free from the social critique the company as a whole receives.

Re: Amazon packages pile up after AWS outage spawns delivery havoc

#177

Post by bobsmooth »

ec109685 wrote: post #146
peterkelly wrote: post #143

Earlier quoted context omitted.

I'm almost tempted to think that having an explicit policy of forcibly shutting down each region once per month for a few hours (at times that are not publicly announced in advance) would be a worthwhile value-add. A service that is unable to handle such a failure does not qualify as being ready for deployment. And I'm not just saying that to be a self-righteous pedant, I'm saying it because this kind of failure is s…

Netflix has a chaos gorilla that does something like that: http://techblog.netflix.com/2011/07/netflix-simian-army.html And even more now: http://techblog.netflix.com/2013/12/active-active-for-multi-...

I wonder how the simian army has evolved in the past decade.

Re: Amazon packages pile up after AWS outage spawns delivery havoc

#178

Post by sciurus »

peterkelly wrote: post #143
karmicthreat wrote: post #90

Whenever us-east-1 goes down you just get a really good feel for how many other companies also have pretty fragile setups. They apps I work on can deal with a few hours of downtime, so as long as I'm sure I can recover from getting totally leveled its ok. And I think it's that was for the majority of companies. Most don't want the extra effort and cost of failover.

I'm almost tempted to think that having an explicit policy of forcibly shutting down each region once per month for a few hours (at times that are not publicly announced in advance) would be a worthwhile value-add. A service that is unable to handle such a failure does not qualify as being ready for deployment. And I'm not just saying that to be a self-righteous pedant, I'm saying it because this kind of failure is s…

Shutting down availability zones I could almost see, since it's not that much more difficult or expensive to architect to handle that gracefully.

But an entire region? I've never worked anywhere that decided being multi-region was a good tradeoff. At best we've replicated data to another region and had some of our management services there, so in the absolute worst case (which would need to be much worse than yesterday) we could rebuild our product's infrastructure there.

Do I agree with this approach? Not in all possible cases of course, but for my employers? Overall, yes. It mitigates the highest impact risks. Going further would have significant complexity and costs. Those companies success or failure haven't been impacted by their multi-region strategy AFAICT.

Re: Amazon packages pile up after AWS outage spawns delivery havoc

#179

Post by ImPostingOnHN »

X3AY7yZfpyWCmf wrote: post #171
pojzon wrote: post #148

Earlier quoted context omitted.

Tell that to management of a medium company showing them the bill for something that has 0.01% chance of happening according to AWS.. Not every workload is of the micro size. MR up our databases would cost around 15mil per year for a company that makes 50mil..

I don't think it's correct to call Amazon a "medium company" and last time I checked they make more than 50mil

I don't think the parent's "our company" is Amazon, because Amazon does indeed make more than $50m, but I can attest to cloud provider multi region replication being extremely expensive for us (also not Amazon), if only due to data transfer costs
Post reply on HN