Summary of the Amazon S3 Service Disruption • Hacker News Board

Re: Summary of the Amazon S3 Service Disruption

#191

Post by andrewaylett » Thu, Mar 02, 2017, 6:41 PM UTC

CobrastanJorji wrote: post #142

ajross wrote: post #8

> At 9:37AM PST, an authorized S3 team member using an established playbook executed a command which was intended to remove a small number of servers for one of the S3 subsystems that is used by the S3 billing process. Unfortunately, one of the inputs to the command was entered incorrectly and a larger set of servers was removed than intended. It remains amazing to me that even with all the layers of automation, the…

It's one of the reasons that silly guarantees like "twelve 9s of reliability" are meaningless. There are humans here. "Accidental human mishap" is gonna happen sometimes, and when it happens it's probably gonna affect a lot of data. Heck, at around 7 or 8 nines you have to account for the possibility that your operations team will decide that all your data is a vicious pack of timberwolves and needs to be defeated.

Note that's durability not reliability. You might not be able to get at it with every request (I think 99.99% is the target) but it'll still be there if you try again later.

Re: Summary of the Amazon S3 Service Disruption

#192

Post by beamatronic » Thu, Mar 02, 2017, 6:41 PM UTC

M_Grey wrote: post #103

jerf wrote: post #73

Earlier quoted context omitted.

I've long said something like "To err is human. To fuck up a million times in a second you need a computer." I may have to upgrade that to take the mighty power of Cloud (TM) into account, though. Billions and trillions of fuck ups per second are now well within reach!

I can't wait until quantum computing lets us add a degree of simultaneity to fucking up. Fuck up in many ways... AT ONCE!

It will not be certain if you have fucked up or not until you actually go to check.

Re: Summary of the Amazon S3 Service Disruption

#193

Post by jonknee » Thu, Mar 02, 2017, 6:41 PM UTC

chrisseaton wrote: post #85

BurningFrog wrote: post #78

Earlier quoted context omitted.

Or the root cause is a UI that allows mistakes like these.

Is it possible to build a UI that will not allow you to make a mistake? If the computer knows exactly what actions would be a mistake, why can't it just do the correct actions (those that aren't a mistake) automatically?

It's not possible to be perfect, but you can certainly do better than taking down S3 because of a single command gone wrong.

One thing I have been doing for my own command line tools is making a preview feature for what a command will do and make the preview state be default. It's simple, but if the S3 engineer first saw a readout of the huge list of servers that were going to be taken offline instead of the small expected list we probably would not be talking about this. There's obviously a ton more you can do here (have the tool throw up "are you sure" messages for unusual inputs, etc).

Re: Summary of the Amazon S3 Service Disruption

#194

Post by eplanit » Thu, Mar 02, 2017, 6:42 PM UTC

So this week the poor soul at Amazon, along with the Price-Waterhouse guy, are the poster children of Human Error.

Re: Summary of the Amazon S3 Service Disruption

#195

Post by ckozlowski » Thu, Mar 02, 2017, 6:42 PM UTC

erikbye wrote: post #130

What does everyone use S3 for? I'm genuinely curious. As my experiments with it have left me disappointed with its performance, I'm just not sure what I could use it for. Store massive amounts of data that is infrequently accessed? Well, unfortunately the upload speed I got to the standard rating one was so abysmal it would take too much time to move the data there; and then I suspect the inverse would be pretty bad…

S3 uploads can be very dependent on what the network looks like between your systems and the endpoint. We introduced S3 Transfer Acceleration to help address this. This uses our edge network as your endpoint, then sending your upload across our backbone rather than traversing the commercial internet. It comes with a small fee, but the fee only gets applied if it improves on what the time to transfer would be otherwise.

https://aws.amazon.com/blogs/aws/aws-storage-update-amazon-s...

If you have lots of data that needs to be uploads (TB/PB worth), then I'd take a look at AWS Snowball. https://aws.amazon.com/snowball/

Also, if you're using the AWS CLI to upload, make sure multi-part upload is enabled.

Why use S3? It scales without any user interaction, is highly available (yes, I cringe saying that, but this is the first, and hopefully only time this has occurred! =D ) and extremely easy to access; it's as simple as an HTTP GET. Being able to address objects directly and not have to worry about managing file systems simplifies a lot.

Re: Summary of the Amazon S3 Service Disruption

#196

Post by nkrisc » Thu, Mar 02, 2017, 6:42 PM UTC

chrisseaton wrote: post #85

BurningFrog wrote: post #78

Earlier quoted context omitted.

Or the root cause is a UI that allows mistakes like these.

Is it possible to build a UI that will not allow you to make a mistake? If the computer knows exactly what actions would be a mistake, why can't it just do the correct actions (those that aren't a mistake) automatically?

No. But a good UI can help you see the mistake you're about to make.

Re: Summary of the Amazon S3 Service Disruption

#197

Post by guelo » Thu, Mar 02, 2017, 6:44 PM UTC

EvanAnderson wrote: post #123

ajross wrote: post #8

> At 9:37AM PST, an authorized S3 team member using an established playbook executed a command which was intended to remove a small number of servers for one of the S3 subsystems that is used by the S3 billing process. Unfortunately, one of the inputs to the command was entered incorrectly and a larger set of servers was removed than intended. It remains amazing to me that even with all the layers of automation, the…

Computers are devices built to amplify human error.

Bicycles for the bumbling mind.

Re: Summary of the Amazon S3 Service Disruption

#198

Post by avar » Thu, Mar 02, 2017, 6:44 PM UTC

rrmm wrote: post #122

avar wrote: post #92

Earlier quoted context omitted.

You don't have to know what would be a mistake. E.g. if the tool is used most of the time to operate on a small set of servers, you have some extra confirmation or command-line option for removing a large set. That's good UI design in tools with powerful destructive capabilities. You make the UI to do lots of things v.s. the few things you do routinely different enough that there's no mistaking them.

You can also have the program tell the user what's going to happen (if it can be computed beforehand), e.g. "This will affect 138 server(s)."

Yes, but be careful. UIs like that tend to accumulate "--yes" options, because you don't feel like being asked every time for 1 server. Then one day you screw up the wildcard and it's 1000 servers, but you used the --yes template.

Which is why I'm pointing out that to design UIs like these you should fall back on slightly different UIs depending on the severity of the operation.

Re: Summary of the Amazon S3 Service Disruption

#199

Post by ckozlowski » Thu, Mar 02, 2017, 6:44 PM UTC

erikbye wrote: post #175

castis wrote: post #161

Earlier quoted context omitted.

1 scenario: if you run a website that has a lot of static content (multiple GB of images, css, js, etc) and you dont want your http server to be responsible for serving that content then you give it all to s3 and let them serve it for you.

What about the performance of serving it? Sounds like I would need to cache it myself, anyway.

Performance out of S3 is generally really good. However, if you're looking to say, serve up a global website and your content is in a single S3 region, then you can leverage CloudFront CDN to serve up those objects. CloudFront integrates seamlessly with S3, and you don't pay transfer charges between CloudFront and S3.

Re: Summary of the Amazon S3 Service Disruption

#200

Post by user15672 » Thu, Mar 02, 2017, 6:44 PM UTC

prh8 wrote: post #80

For as much as people jumped all over Gitlab last month, this seems remarkably similar in terms of preparedness for accidental and unanticipated failure.

Beyond a typing mistake, it's not really very similar. The Gitlab incident was one avoidable problem after another, ending with a giant WTF when they found out that no-one had even tested the backups were working.

This is a case of someone slipping on the keyboard, removing more capacity than intended and the recovery process taking longer than expected. The process actually seems to be working (to a given value of working), but the amount of downtime was way above acceptable. They've already put more safeguards into the tooling to prevent the situation from happening again.

S3 is also orders of magnitude more complex than Gitlabs infrastructure, so while the amount of time the outage lasted for is not acceptable, it does show that they at least have working processes for critical situations that allow them to get back in service within a day, which is pretty impressive.