Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Summary of the Amazon S3 Service Disruption

aws.amazon.com

Post reply on HN
Print view View original

141–150 of 535 posts

Re: Summary of the Amazon S3 Service Disruption

#141

Post by fixermark »

1_2__3 wrote: post #113
pinko wrote: post #67

Earlier quoted context omitted.

I always wonder about unintended consequences of this sort of thing. Like someday there will be a worm about to rampage through their servers and someone says, "take them all offline now!" and the answer is, "we can't because of the throttle safeguard we put in place after incident XYZ, it will be about 17 hours..."

I'm remembering tools I've worked with where the cheeky dev required things like type the sentence "I know what I am doing and wish to proceed." in order to perform unsafe operations. I've always wondered why ops hasn't adopted some of the best practices that have been around for years to avoid fat finger errrors. Like why don't we have systems where to do something dangerous requires two separate people run the comm…

The most common explanation: "cute" interactions make it harder to script the command-line tools because you have to account for the extra layer of indirection or write a bit of screen-scrape logic to get the command prompt input.

I've always found that explanation a little threadbare.

Re: Summary of the Amazon S3 Service Disruption

#142

Post by CobrastanJorji »

ajross wrote: post #8

> At 9:37AM PST, an authorized S3 team member using an established playbook executed a command which was intended to remove a small number of servers for one of the S3 subsystems that is used by the S3 billing process. Unfortunately, one of the inputs to the command was entered incorrectly and a larger set of servers was removed than intended. It remains amazing to me that even with all the layers of automation, the…

It's one of the reasons that silly guarantees like "twelve 9s of reliability" are meaningless. There are humans here. "Accidental human mishap" is gonna happen sometimes, and when it happens it's probably gonna affect a lot of data. Heck, at around 7 or 8 nines you have to account for the possibility that your operations team will decide that all your data is a vicious pack of timberwolves and needs to be defeated.

Re: Summary of the Amazon S3 Service Disruption

#143

Post by fixermark »

ProAm wrote: post #88
cflewis wrote: post #68

Earlier quoted context omitted.

"It happens" is the only reasonable takeaway you can get from a postmortem like this. My worry is that people read it and go "I am aghast that such a command can be run!" without knowing that little commands like that are run numerous times a day without incident. The only thing I read in there and go "hmmm" is that it took quite that long for the S3 service to recover, and that the status page wasn't hosted on someo…

People need to realize when they go to the cloud it's not that 'it happens', it's that it will happen, and you have no ability to do anything about it. Fact of life and risk management.

... and it's a different risk from self-hosting, but self-hosting provides all sorts of similar issues (such as when you do this to yourself, the cost is now coming out of your pocket, not Amazon's, to employ software engineers to harden your scripts against making the same mistake twice).

Re: Summary of the Amazon S3 Service Disruption

#145

Post by DigitalBison »

snewman wrote: post #28
cnorthwood wrote: post #6

I'm surprised how transparent this is, I can find Amazon often a bit opaque when dealing with issues.

I've found them to be very opaque in most contexts, but for major outages (which have been rare), they do have a history of solid postmortems.

The public postmortem from the big DynamoDB outage in 2015(?) is a great example I think: https://aws.amazon.com/message/5467D2/

Re: Summary of the Amazon S3 Service Disruption

#147

Post by bm1362 »

savanaly wrote: post #52

Not as interesting an explanation as I was hoping for. Someone accidentally typed "delete 100 nodes" instead of "delete 10 nodes" or something. It sounds like the weakness in the process is that the tool they were using permitted destructive operations like that. The passage that stuck out to me: "in this instance, the tool used allowed too much capacity to be removed too quickly. We have modified this tool to remove…

I'm drawing conclusions based on my time at AWS, but I believe this is due to the service discovery mechanism most of AWS uses. It's a gossip protocol, with a daemon running on each service host. There are very valid reasons you would be manipulating the set of hosts currently being gossiped- to remove a host for maintenance for example. In this use case I think they wanted to take out an entire service, so ensuring a majority is still alive isn't necessarily a solution.

There are two distinct failures here, the tool being too liberal and then the failure mode not being well understood for this index subsystem.

If I remember correctly, all of our processes were organized through a weekly change management process that went through reviews of the exact commands to be run. Being oncall was a bit more liberal, you would typically execute commands on production as needed based on your experience and with others over your shoulder if you had any doubt. Interacting with the gossip protocol was a pretty common thing to do when you were triaging issues.

Unrelated, I was briefly a SME on the EBS billing system and probably interacted with the poor guy whom executed this command.

Re: Summary of the Amazon S3 Service Disruption

#148

Post by ckozlowski »

DaiPlusPlus wrote: post #119
omni wrote: post #24

Earlier quoted context omitted.

They just spent millions of dollars training that person to never make this mistake again, should definitely keep them around.

The wording of the article implies Amazon is shifting the blame entirely on the individual who typo'd: they indemnify themselves with "an authorized S3 team member using an established playbook..." ("don't blame us, our process is perfect!") There are process-fixes for this, such as requiring a two-person rule when at a production shell and modifying tooling to detect potentially unintentional commands (e.g. a SQL UP…

Not at all. The "an authorized S3 team member using an established playbook..." is to establish that:

1. They were authorized to be doing it. 2. They were following an established process, and not winging it.

The reason for that wording is to illustrate that operational changes are made in accordance with established CM.

Mistakes happen. The system didn't catch the error. That's why the mitigation is "fix the system", and not "fire the team member". =)

Re: Summary of the Amazon S3 Service Disruption

#149

Post by rmason »

ajross wrote: post #8

> At 9:37AM PST, an authorized S3 team member using an established playbook executed a command which was intended to remove a small number of servers for one of the S3 subsystems that is used by the S3 billing process. Unfortunately, one of the inputs to the command was entered incorrectly and a larger set of servers was removed than intended. It remains amazing to me that even with all the layers of automation, the…

So they're going to build a complex system to correct possible user command line errors. That new system itself will introduce possible errors. Wouldn't an administrative GUI have been much simpler to implement overall?

Re: Summary of the Amazon S3 Service Disruption

#150

Post by pavel_lishin »

CobrastanJorji wrote: post #142
ajross wrote: post #8

> At 9:37AM PST, an authorized S3 team member using an established playbook executed a command which was intended to remove a small number of servers for one of the S3 subsystems that is used by the S3 billing process. Unfortunately, one of the inputs to the command was entered incorrectly and a larger set of servers was removed than intended. It remains amazing to me that even with all the layers of automation, the…

It's one of the reasons that silly guarantees like "twelve 9s of reliability" are meaningless. There are humans here. "Accidental human mishap" is gonna happen sometimes, and when it happens it's probably gonna affect a lot of data. Heck, at around 7 or 8 nines you have to account for the possibility that your operations team will decide that all your data is a vicious pack of timberwolves and needs to be defeated.

As I understand it, those guarantees don't mean that the service will actually stay up for the given number of 9s; it's that you'll be reimbursed monetarily if and when they go down.
Post reply on HN