Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Summary of the Amazon S3 Service Disruption

aws.amazon.com

Post reply on HN
Print view View original

171–180 of 535 posts

Re: Summary of the Amazon S3 Service Disruption

#171

Post by peu4000 »

erikbye wrote: post #130

What does everyone use S3 for? I'm genuinely curious. As my experiments with it have left me disappointed with its performance, I'm just not sure what I could use it for. Store massive amounts of data that is infrequently accessed? Well, unfortunately the upload speed I got to the standard rating one was so abysmal it would take too much time to move the data there; and then I suspect the inverse would be pretty bad…

Database backups, puppet modules, a couple low traffic static sites, assets for one of our projects.

Re: Summary of the Amazon S3 Service Disruption

#172

Post by all_usernames »

Overall, it's pretty amazing that the recovery was as fast as it was. Given the throughput of S3 API calls you can imagine the kind of capacity that's needed to do a full stop followed by a full start. Cold-starting a service when it has heavy traffic immediately pouring into it can be a nightmare.

It'd be very interesting to know what kind of tech they use at AWS to throttle or do circuit breaking to allow back-end services like the indexer to come up in a manageable way.

Re: Summary of the Amazon S3 Service Disruption

#173

Post by losteverything »

mcherm wrote: post #152
losteverything wrote: post #104

"I did." That was CEO Robert Allen's response when the AT&T network collapsed [1] on January 15, 1990 He was asked who made the mistake. I can't imagine any CEO now a days making a similar statement. [1] http://users.csc.calpoly.edu/~jdalbey/SWE/Papers/att_collaps...

Fascinating! Do you know of any source that documents the claim that Robert Allen made this statement?

I'll try. It was a dark day.

We all watched the news and I recall him saying that. The specific quote I don't remember but it was something like " you can consider that I did." I think he was asked what will happen to the person that caused it and who is that person.

Everyone knew right away this had to be human error. Right away. Switches simply had too much redundancy.

It was big then and not sure if I can locate a video.

Re: Summary of the Amazon S3 Service Disruption

#174

Post by HeyLaughingBoy »

pavel_lishin wrote: post #150
CobrastanJorji wrote: post #142

Earlier quoted context omitted.

It's one of the reasons that silly guarantees like "twelve 9s of reliability" are meaningless. There are humans here. "Accidental human mishap" is gonna happen sometimes, and when it happens it's probably gonna affect a lot of data. Heck, at around 7 or 8 nines you have to account for the possibility that your operations team will decide that all your data is a vicious pack of timberwolves and needs to be defeated.

As I understand it, those guarantees don't mean that the service will actually stay up for the given number of 9s; it's that you'll be reimbursed monetarily if and when they go down.

Kinda the same thing, though. I mean, from my perspective there's no substantive difference between me saying "this service will stay up 99.xx% of the time and me buying insurance to pay you for the 0.xx% of the time I might fail.

The alternative is that I use the insurance to pay my legal fees when you sue me for not meeting my uptime guarantees.

Re: Summary of the Amazon S3 Service Disruption

#175

Post by erikbye »

castis wrote: post #161
erikbye wrote: post #130

What does everyone use S3 for? I'm genuinely curious. As my experiments with it have left me disappointed with its performance, I'm just not sure what I could use it for. Store massive amounts of data that is infrequently accessed? Well, unfortunately the upload speed I got to the standard rating one was so abysmal it would take too much time to move the data there; and then I suspect the inverse would be pretty bad…

1 scenario: if you run a website that has a lot of static content (multiple GB of images, css, js, etc) and you dont want your http server to be responsible for serving that content then you give it all to s3 and let them serve it for you.

What about the performance of serving it? Sounds like I would need to cache it myself, anyway.

Re: Summary of the Amazon S3 Service Disruption

#176

Post by SilasX »

pavel_lishin wrote: post #150
CobrastanJorji wrote: post #142

Earlier quoted context omitted.

It's one of the reasons that silly guarantees like "twelve 9s of reliability" are meaningless. There are humans here. "Accidental human mishap" is gonna happen sometimes, and when it happens it's probably gonna affect a lot of data. Heck, at around 7 or 8 nines you have to account for the possibility that your operations team will decide that all your data is a vicious pack of timberwolves and needs to be defeated.

As I understand it, those guarantees don't mean that the service will actually stay up for the given number of 9s; it's that you'll be reimbursed monetarily if and when they go down.

I don't think it even means that; their policy says that the reimbursement only happens when your reliability dips all the way down to three 9s:

https://aws.amazon.com/s3/sla/

Re: Summary of the Amazon S3 Service Disruption

#177

Post by rkuykendall-com »

fixermark wrote: post #151
rkuykendall-com wrote: post #109

tl;dr: Engineer fat-fingered a command and shut everything down. Booting it back up took a long time. Then the backlog was huge, so getting back to normal took even longer. We made the command safer, and are gonna make stuff boot faster. Finally, we couldn’t report any of this on the service status dashboard, because we’re idiots, and the dashboard runs on AWS.

Everything except the "We're idiots" part I'd agree with. Self-hosting your diagnostic tools is an easy mistake to make, and I've seen both startups and large, multi-decade-experienced companies make it.

Of course. Meant it as more of an "I just spent 10 minutes searching for my car keys while holding them, because I'm an idiot." No disrespect to the engineers.

Re: Summary of the Amazon S3 Service Disruption

#178

Post by CWuestefeld »

St-Clock wrote: post #120

I am unpleasantly surprised that they do not mention why services that should be unrelated to S3 such as SES were impacted as well and what they are doing to reduce such dependencies. From a software development perspective, it makes sense to reuse S3 and rely on it internally if you need object storage, but from an ops perspective, it means that S3 is now a single point of failure and that SES's reliability will alw…

services that should be unrelated to S3 such as SES were impacted

I don't think this is particularly surprising. I'd already pretty much assumed that, e.g., a package of code for a Lambda function would be housed in an S3 bucket somewhere.

What's really surprising to me is how many of those buckets appear to live in US-EAST-1, and aren't able to keep functioning in a catastrophe by failing over to a different region.

Post reply on HN