Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Summary of the Amazon S3 Service Disruption

aws.amazon.com

Post reply on HN
Print view View original

501–510 of 535 posts

Re: Summary of the Amazon S3 Service Disruption

#501

Post by jerf »

cjbprime wrote: post #423
jerf wrote: post #73

Earlier quoted context omitted.

I've long said something like "To err is human. To fuck up a million times in a second you need a computer." I may have to upgrade that to take the mighty power of Cloud (TM) into account, though. Billions and trillions of fuck ups per second are now well within reach!

> I've long said something like "To err is human. To fuck up a million times in a second you need a computer." This quote (paraphrased) actually dates all the way back to 1969: > To err is human; to really foul things up requires a computer. -- http://quoteinvestigator.com/2010/12/07/foul-computer/

Yes, it is. I believe I added the concept of "fuckups per second", but my memory being what it is and the general creativity of the internet being what it is, I would not be surprised that it either wasn't original or I wasn't the first.

Re: Summary of the Amazon S3 Service Disruption

#502

Post by twistedpair »

Atheros wrote: post #480
twistedpair wrote: post #419

Earlier quoted context omitted.

> Failure at every level has to be simulated pretty often to understand how to handle it Keep in mind, S3 "fails" all the time. We regularly make millions of S3 requests at my work. Usually we get 1:240K failure rate (mostly GETs), returning 500 errors. However, if you're really hammering an S3 node in the hash ring (e.g. Spark job), we see failures in the 1/10K range, including SocketExceptions, where the routed IP…

If you retry the get does it succeed usually?

Yes, retry has always worked (*us-west-2).

Sometimes it's a 404 for an object written 1 sec prior, other times it's an S3 node that died mid request. Retry gets you to a different node.

Re: Summary of the Amazon S3 Service Disruption

#503

Post by pfortuny »

I guess it is time to define commands whose inputs have great distance in, say the Damerau-Levenshtein metric.

For numerical inputs, one might use both the digits and the textual expression. This would make them quite cumbersone but much less prone to errors. Or devise some shorthand for them...

156 (on fi six). 35. (zer th fi). 170 (on se zer). 28 (two eig) evens have three letters odds have two.

This is just my 2 cents.

Re: Summary of the Amazon S3 Service Disruption

#504

Post by bsaul »

Wonder if every numbers for critical command lines shouldn't be spelled out as well. If you think about how checks works, you're supposed to write the number as well as the words for the number. -nbs two_hundreds instead of twenty is much less likely to happen..

just like rm -rf / should really be rm -rf `root`

Re: Summary of the Amazon S3 Service Disruption

#505

Post by hueving »

jordigh wrote: post #238
hueving wrote: post #69

Earlier quoted context omitted.

The term "playbook" just originates from sports as a documentation of what you are going to do for for a specific task (i.e. a "play" in sports).

I'm pretty sure the sports meaning comes from the theathre meaning. Ansible has other bits of theatre metaphor in it, such as "roles" and "scripts".

What is the theatre meaning? I'm not familiar with that term.

Re: Summary of the Amazon S3 Service Disruption

#506

Post by Bartweiss »

thr0waway1239 wrote: post #464
mml wrote: post #450

Earlier quoted context omitted.

apparently this is (or was) a job in japan. companies would hire what amounts to an actor to get screamed at by the angry customer, and pretend to get fired on the spot. rinse, repeat whenever such appeasement is required.

Imagine if the customer saw the same actor getting fired in different companies! Is the customer going to catch on? More likely, they will think "Yeah, no wonder there was a problem. This same incompetent dude wormed his way into this company too" :-)

There's a movie sketch in here somewhere. A guy has the worst day of his life, every single thing goes wrong, and at every single company the same person is "responsible" for the issue.

Re: Summary of the Amazon S3 Service Disruption

#507

Post by officelineback »

erikbye wrote: post #183
peu4000 wrote: post #171

Earlier quoted context omitted.

Database backups, puppet modules, a couple low traffic static sites, assets for one of our projects.

Database backups: with the upload speeds I've seen, completing a backup of a database with hundreds of GB would take a really long time. I don't want that extra load and keeping that connection open forever.

S3 endpoints in your VPC?

Re: Summary of the Amazon S3 Service Disruption

#508

Post by Bartweiss »

cjs wrote: post #457
hibikir wrote: post #430

Earlier quoted context omitted.

Training about testing backups is still a bad idea: Why make someone do a job that purely verification? Those jobs eventually stop getting done, and it's hard to keep people doing them. Instead, you make a machine verify the backups simply by using the backups all the time. For example, at work I feed part of our data pipeline with backups: Those processes have no access to the live data. If the backups break, those…

I agree. I have the same policy when setting up servers: don't have a "primary" and a "backup" server, make both servers production servers and have the code that uses them alternate between them, pick a random one, whatever. (I don't always get to implement this policy, of course.)

This makes some sense, but I don't think it negates testing backups? For duplicate live data, yeah, you can't just use both. But most businesses have at least some things backed up to cold storage, and that still needs to be popped in a tape deck (or whatever's relevant) and verified.

Re: Summary of the Amazon S3 Service Disruption

#509

Post by officelineback »

tifa2up wrote: post #156

Do you know if Amazon is giving any refunds/credits for the service outbreak?

I don't think it's automatic. I just helped my former boss with his decision to go for a refund (he asked me for help drafting a request, but I reminded him that like 99.99% of their S3 storage is backups that are IA-Standard, so it may not be worth it).

Re: Summary of the Amazon S3 Service Disruption

#510

Post by beat »

sah2ed wrote: post #490
beat wrote: post #301

Earlier quoted context omitted.

I don't buy the "If we just plan ENOUGH, disasters will never occur" argument. It's the universe is just too darn interesting for us to be able to plan enough to prevent it from being interesting.

"[T]he universe is just too darn interesting for us to be able to plan enough to prevent it from being interesting." -- Beat That's a great line. How should I attribute it?

If you're actually quoting me somewhere, either use "Dave Stagner", or "Some asshole on the internet". Same diff.
Post reply on HN