Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Summary of the Amazon S3 Service Disruption

aws.amazon.com

Post reply on HN
Print view View original

301–310 of 535 posts

Re: Summary of the Amazon S3 Service Disruption

#301

Post by beat »

problems wrote: post #263
Bartweiss wrote: post #240

Earlier quoted context omitted.

Any time I see "we're going to train everyone better" or "we're going to fire the guy who did it", all I can read is "this will happen again". You can't actually solve the problem of user error with training, and it's good to see Amazon not playing that game.

This is true in some cases, but not when mitigations aren't practiced properly - it's not the fat fingered user who should be fired or retrained, but the designer or maintainer of the system that allowed it to become a serious issue. Look at the recent GitLab incident - one guy messed up and nuked a server. Okay, that happens sometimes, go to backups. Uh oh, all the backups are broken. Minor momentary problem just tu…

I don't buy the "If we just plan ENOUGH, disasters will never occur" argument. It's the universe is just too darn interesting for us to be able to plan enough to prevent it from being interesting.

Re: Summary of the Amazon S3 Service Disruption

#302

Post by aorloff »

yellow_postit wrote: post #39
Scramblejams wrote: post #13

Earlier quoted context omitted.

No, this is addressed: >We understand that the SHD provides important visibility to our customers during operational events and we have changed the SHD administration console to run across multiple AWS regions.

Which is fine until those regions go down. A status page, in my mind, should have a fallback on a completely different service provider.

Exactly. What they should do is have the status system be independent of AWS so it can report issues regardless of AWS service status.

Re: Summary of the Amazon S3 Service Disruption

#303

Post by make3 »

nemacol wrote: post #232
duskwuff wrote: post #162

Earlier quoted context omitted.

Quantum computing: giving humans the unprecedented ability to make every possible error at once

It will be fucked, not fucked, neither, and both... until we look. I feel bad for the poor bastard that has to look...

shrodinger's buttocks

Re: Summary of the Amazon S3 Service Disruption

#304

Post by gigatexal »

vocatus_gate wrote: post #258
gigatexal wrote: post #106

Earlier quoted context omitted.

Yeah as soon as I read this I felt bad for the employee. I remember writing an update statement without a where clause and having to restore the table from backup. But that was at a company not as advanced as Amazon. Fat fingering a key like that is just crazy (but comforting that even at Amazon it happens) and I'm sure they've fixed that from happening again.

I once brought down our entire production XenServer cluster group by issuing a "shutdown now" in the wrong SSH window. Needless to say it was a bad feeling watching Nagios go crazy and realizing what had just happened.

Oh crap! Yeah I bet you were pretty panicked. My update statement destroyed data that my team used all the time so I was worried I'd get fired. Luckily that wasn't the case.

Re: Summary of the Amazon S3 Service Disruption

#305

Post by perlgeek »

pinko wrote: post #67
savanaly wrote: post #52

Not as interesting an explanation as I was hoping for. Someone accidentally typed "delete 100 nodes" instead of "delete 10 nodes" or something. It sounds like the weakness in the process is that the tool they were using permitted destructive operations like that. The passage that stuck out to me: "in this instance, the tool used allowed too much capacity to be removed too quickly. We have modified this tool to remove…

I always wonder about unintended consequences of this sort of thing. Like someday there will be a worm about to rampage through their servers and someone says, "take them all offline now!" and the answer is, "we can't because of the throttle safeguard we put in place after incident XYZ, it will be about 17 hours..."

You can always build the safeguard to require approval from a peer (or superior) to an action that is normally considered dangerous, at least for overriding the throttle.

Re: Summary of the Amazon S3 Service Disruption

#306

Post by M_Grey »

nemacol wrote: post #232
duskwuff wrote: post #162

Earlier quoted context omitted.

Quantum computing: giving humans the unprecedented ability to make every possible error at once

It will be fucked, not fucked, neither, and both... until we look. I feel bad for the poor bastard that has to look...

Internship in the future just got a whole lot bleaker.

Re: Summary of the Amazon S3 Service Disruption

#307

Post by perlgeek »

westernmostcoy wrote: post #135

Take a moment to look at the construction of this report. There is no easily readable timeline. It is not discoverable from anywhere outside of social media or directly searching for it. As far as I know, customers were not emailed about this - I certainly wasn't. You're an important business, AWS. Burying outage retrospectives and live service health data is what I expect from a much smaller shop, not the leader in…

Also notably missing is the "we will automatically refund all affected customers" line that we'd expect from somebody who wants to provide excellent service.

A graphical illustration of the service dependencies they were talking about would have been nice as well.

Re: Summary of the Amazon S3 Service Disruption

#308

Post by abraves10001 »

jedberg wrote: post #300
frankc wrote: post #294

Earlier quoted context omitted.

I agree with that in general but having your monitoring system be dependent on the thing it monitors is a pretty big goof. It possible that the dependency was very non-obvious and many layers deep, which is more understandable, but still...its pretty fundamental.

The monitoring system was not dependent on the thing it was monitoring. The website that shows the public results of the monitoring, which is updates only by humans, depended on it.

An important distinction but doesn't negate the ideas that their users should be in the dark because of something this stupid.

Re: Summary of the Amazon S3 Service Disruption

#309

Post by hollander »

snewman wrote: post #131
conorh wrote: post #17

This part is also interesting: > While this is an operation that we have relied on to maintain our systems since the launch of S3, we have not completely restarted the index subsystem or the placement subsystem in our larger regions for many years. These sorts of things make me understand why the Netflix "Chaos Gorilla" style of operating is so important. As they say in this post: > We build our systems with the assu…

> Failure at every level has to be simulated pretty often to understand how to handle it, and it is a really difficult problem to solve well. Exactly. It seems likely that Amazon tests the restart operation, but it would be hard to test it at full us-east-1 scale. Running a full S3 test cluster at that scale would likely be a prohibitive expense. Perhaps the "index subsystem" and "placement subsystem" are small enoug…

Testing a full zone test is only possible when they have a new zone available, unused. I bet they do these test, and they now have a new scenario to test.

Re: Summary of the Amazon S3 Service Disruption

#310

Post by Dangeranger »

imsofuture wrote: post #214
Dangeranger wrote: post #23

So this is the second high profile outage in the last month caused by a simple command line mistake. > Unfortunately, one of the inputs to the command was entered incorrectly and a larger set of servers was removed than intended. If I would have guessed anyone could prevent mistakes like this from propagating it would be AWS. It points to just how easy it is to make these errors. I am sure that the SRE who made this…

Amazon has the wherewithal to not freaking publicly name their actually human employee, so I'd imagine their culture around outages is probably a lot more healthy.

Well to be fair, he named himself within his notes and did not object to the public nature of the disclosure. I agree with your sentiment though that names should not be included within postmortems in the general case.
Post reply on HN