Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Summary of the Amazon S3 Service Disruption

aws.amazon.com

Post reply on HN
Print view View original

371–380 of 535 posts

Re: Summary of the Amazon S3 Service Disruption

#371

Post by avsupport »

jph00 wrote: post #10

> From the beginning of this event until 11:37AM PST, we were unable to update the individual services’ status on the AWS Service Health Dashboard (SHD) because of a dependency the SHD administration console has on Amazon S3. Ensuring that your status dashboard doesn't depend on the thing it's monitoring is probably the first thing you should think about when designing your status system. This doesn't fill me with co…

Considering the size of AWS, the number of services one service relies on would be too many, even if a single service many layers deep depended on s3, and when s3 goes down, the service as a whole will be affected. Honestly S3, it like a blackhole, everybody stores everything in S3 these days. Its a horizontal component but used in a vertical manner. Weird but true.

Re: Summary of the Amazon S3 Service Disruption

#372

Post by Piskvorrr »

vocatus_gate wrote: post #276
TeMPOraL wrote: post #247

Earlier quoted context omitted.

There's a balance to be struck. I'd say number of hoops you have to jump through to do something should scale with the potential impact of an operation. That said, the only way to completely prevent mistakes is to make the tool unable to do anything at all. (Or to encode every possible meaning of the word "mistake" in your software. If you could do that, you would probably get a Nobel prize for it.)

In a program I wrote I make the user manually type "I AGREE" (case-sensitive) in a prompt before continuing, just to avoid situations where people just tap "y" a bunch of times.

Habituation is a powerful thing: a safety-critical program used in the 90s had a similar, hard-coded safety prompt (<10 uppercase ASCII characters). Within a few weeks, all elevated permission users had the combination committed to muscle memory and would bang it out without hesitation, just by reflex: "Warning: please confirm these potentially unsaf-" "IAGREE!"

Re: Summary of the Amazon S3 Service Disruption

#373

Post by refulgentis »

simplify wrote: post #327
refulgentis wrote: post #251

Earlier quoted context omitted.

How laudable is this, really? I've had the privilege of either working for myself, the company that acquired mine and let me run the dev, or at Google. From that perspective, and what I understand about ops, the rarity is not having the attitude mentioned in the parent.

Are you suggesting we take their behavior for granted? Positive behavior needs to be praised – it's part of how society influences its members.

No

Re: Summary of the Amazon S3 Service Disruption

#374

Post by Fiahil »

ajross wrote: post #8

> At 9:37AM PST, an authorized S3 team member using an established playbook executed a command which was intended to remove a small number of servers for one of the S3 subsystems that is used by the S3 billing process. Unfortunately, one of the inputs to the command was entered incorrectly and a larger set of servers was removed than intended. It remains amazing to me that even with all the layers of automation, the…

Someone forgot a --limit on their ansible playbook ? I did that too.

Re: Summary of the Amazon S3 Service Disruption

#375

Post by EternalData »

This caused panic and chaos for a bit among my team, which I imagine was replicated across the web.

Moments like these always remind me that a particularly clever or nefarious set of individuals could shut down essential parts of the Internet with a few surgical incisions.

Re: Summary of the Amazon S3 Service Disruption

#376

Post by Piskvorrr »

pwthornton wrote: post #202

This is the risk you run into by doing everything through the command line. This would be really hard to do through a good GUI.

"Next/Next/Yes/Ok/Kill all humans/Yes/Ok/Accept/Ok/Ok/Yes/Next/Confirm" Does that sequence look familiar? Well, that's the GUI equivalent.

Re: Summary of the Amazon S3 Service Disruption

#377

Post by logicallee »

ajross wrote: post #8

> At 9:37AM PST, an authorized S3 team member using an established playbook executed a command which was intended to remove a small number of servers for one of the S3 subsystems that is used by the S3 billing process. Unfortunately, one of the inputs to the command was entered incorrectly and a larger set of servers was removed than intended. It remains amazing to me that even with all the layers of automation, the…

serious question - why did no one ever accidentally launch and nuke a city, with thousands of nuclear warheads able to do so on short notice? like, AWS presumably puts a lot more redundancy in, and yet with all that effort comes up this far short. Why? It has a huge amount of brainpower all set up so that this never ever happens. Whatever works for the military, can't they adopt those actual best practices?

Re: Summary of the Amazon S3 Service Disruption

#378

Post by Piskvorrr »

thraway2016 wrote: post #344

Something doesn't pass the smell test. Over two hours to reboot the index hosts?

I don't think those would only have a mere 1 MB of data ;) Even considering RAM speeds, stuff scales in surprising ways when a third of the Internet relies on you. (Right, right, exaggerating - this was one AZ only; but you catch the drift, I assume)

Re: Summary of the Amazon S3 Service Disruption

#379

Post by chrisseaton »

Piskvorrr wrote: post #370
chrisseaton wrote: post #85

Earlier quoted context omitted.

Is it possible to build a UI that will not allow you to make a mistake? If the computer knows exactly what actions would be a mistake, why can't it just do the correct actions (those that aren't a mistake) automatically?

If the computer knows exactly what actions would be a mistake - how? The difference between correct and incorrect (not to mention legal and illegal) is usually inferred from a much wider context than what is accessible to a script. Mind you, in this specific case, Amazon even implies that such a command could have been correct under other circumstances. So, this means a) strong superhuman AI (good luck), b) deciding…

> If the computer knows exactly what actions would be a mistake - how?

I don't know. I was suggesting it wasn't realistic to do that, and therefore it wasn't realistic to implement a UI that prevents you making mistakes.

Post reply on HN