Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Summary of the Amazon S3 Service Disruption

aws.amazon.com

Post reply on HN
Print view View original

461–470 of 535 posts

Re: Summary of the Amazon S3 Service Disruption

#461

Post by scott_karana »

mmanfrin wrote: post #299

There are quite a few comments here ignoring the clarity that hindsight is giving them. Apparently the devops engineers commenting here have never fucked up.

On the contrary: I feel like Amazon is taking some flak because everyone here has messed up before, and are surprised that engineers (seemingly lacking failure experience) were able to do what they did.

I wouldn't task a junior sysadmin a server deletion, would you? Nor could I ever consider someone without a fuckup a senior ;)

Re: Summary of the Amazon S3 Service Disruption

#462

Post by greglindahl »

saganus wrote: post #459
greglindahl wrote: post #456

Earlier quoted context omitted.

A well-designed confirmation doesn't give you the same prompt for deleting some random test server as it does for deleting a production server. That helps with the "autopilot mode" issue.

I agree that it should help reduce the amount of mistakes. But I still believe auto-pilot mode is a real thing (and a danger!) . My point is that I'm not sure if it's even possible to design one that actually cuts errors to 0. And if that's indeed the case, even if it's close to 0, it's still non-zero, thus at the scale Amazon operates at, it's very probable that it will happen at least one time. Maybe sometime in th…

I totally agree that it's a real issue, a danger, and that it's impossible to cut errors to zero.

I've also built complex systems that have been run in production for years with relatively few typo-related problems. The way I do it is with the design patterns like the one I just mentioned, which is also what TeMPOraL was talking about (and I guess you missed it.)

If you have the same kind of confirmation whenever you delete a thing, whether it's an important thing or not, you're designing a system which encourages bad auto-pilot habits.

You'll also note that Amazon's description of the way that they plan on changing their system is intended to fire extra confirmation only when it looks like the operator is about to make a massive mistake. That follows the design pattern I'm suggesting.

Re: Summary of the Amazon S3 Service Disruption

#463

Post by thewhitetulip »

gregtaleck wrote: post #187
tdicola wrote: post #166

Earlier quoted context omitted.

To be fair, the real problem isn't that someone screwed up a playbook or command. The real problem is that a tiny mistake in a command can cause an entire service to be disrupted for hours. That's the problem that needs to be fixed.

"While removal of capacity is a key operational practice, in this instance, the tool used allowed too much capacity to be removed too quickly. We have modified this tool to remove capacity more slowly and added safeguards to prevent capacity from being removed when it will take any subsystem below its minimum required capacity level. This will prevent an incorrect input from triggering a similar event in the future."

I wonder as to why this wasn't thought of while creating the system? Of course, I don't have experience at that scale, so just wondering.

Re: Summary of the Amazon S3 Service Disruption

#464

Post by thr0waway1239 »

mml wrote: post #450
patio11 wrote: post #432

Earlier quoted context omitted.

One reason to like a facet of Japanese management culture: if a customer wants someone to rake over the coals you offer management, not employees. Internal repercussions notwithstanding, externally the company is a united front. It cannot cause mistakes by luck, accident, or happenstance, because the world includes luck, accidents, and happenstance, so any user-visible error is ipso facto a failure of management.

apparently this is (or was) a job in japan. companies would hire what amounts to an actor to get screamed at by the angry customer, and pretend to get fired on the spot. rinse, repeat whenever such appeasement is required.

Imagine if the customer saw the same actor getting fired in different companies! Is the customer going to catch on? More likely, they will think "Yeah, no wonder there was a problem. This same incompetent dude wormed his way into this company too" :-)

Re: Summary of the Amazon S3 Service Disruption

#465

Post by thr0waway1239 »

fixermark wrote: post #143
ProAm wrote: post #88

Earlier quoted context omitted.

People need to realize when they go to the cloud it's not that 'it happens', it's that it will happen, and you have no ability to do anything about it. Fact of life and risk management.

... and it's a different risk from self-hosting, but self-hosting provides all sorts of similar issues (such as when you do this to yourself, the cost is now coming out of your pocket, not Amazon's, to employ software engineers to harden your scripts against making the same mistake twice).

Not to mention, Amazon is catching the long tail of cloud failure like Google is catching the long tail of search keywords. They can now say with a somewhat straight face - "You know all those scripts you run to keep everything up? We have figured out many, many more possible ways for them to fail than you probably ever will, and we have added more layers of safeguards than you can even imagine."

Re: Summary of the Amazon S3 Service Disruption

#466

Post by gioele »

mml wrote: post #450
patio11 wrote: post #432

Earlier quoted context omitted.

One reason to like a facet of Japanese management culture: if a customer wants someone to rake over the coals you offer management, not employees. Internal repercussions notwithstanding, externally the company is a united front. It cannot cause mistakes by luck, accident, or happenstance, because the world includes luck, accidents, and happenstance, so any user-visible error is ipso facto a failure of management.

apparently this is (or was) a job in japan. companies would hire what amounts to an actor to get screamed at by the angry customer, and pretend to get fired on the spot. rinse, repeat whenever such appeasement is required.

Inspired by the Daniel Pennac's novels? https://fr.wikipedia.org/wiki/Saga_Malauss%C3%A8ne

Re: Summary of the Amazon S3 Service Disruption

#467

Post by XorNot »

dbenhur wrote: post #458
evilDagmar wrote: post #439

Earlier quoted context omitted.

No. You don't make a daily task of testing backups. That would be wrong for precisely the reasons you cite. It's a waste of effort and time, and ignores what the point of testing them is for: ensuring that the procedure still works . One would only actually test the backups about twice a year just to be damn sure they are still resulting in restorable data. The rest of the year it's only worth keeping an automated pr…

> One would only actually test the backups about twice a year just to be damn sure they are still resulting in restorable data. Nope. Nope. Nope. You test every backup by automatically restoring from it in a sandbox and verifying its integrity and functionality in the restored state. Backups are worthless unless verified for their intended use of recovering a functioning system.

You're imagining an automated test system. But Gitlabs problem was the automated system was not communicating failures properly.

And constant "this succeeded" messages don't scale well.

Re: Summary of the Amazon S3 Service Disruption

#468

Post by groby_b »

geodel wrote: post #254
liquidise wrote: post #227

Earlier quoted context omitted.

This is the major basis of the CMM Levels [1]. At higher levels of maturity and necessity, systems and processes are designed to increasingly prevent errors from reaching a production environment. Amazon is taking the right approach here. The fact that a system as complex and important as S3 can be taken down is a failure of the system, not the person who took it down accidentally. 1. https://en.wikipedia.org/wiki/Ca…

A lot of IT vendors I have worked with, they all were CMM/CMMi level 5. But the crappiness in their work development/process/deployment etc make me wonder if all their efforts go in attaining those certifications as oppose to doing something better.

CMM level 5 ==> You have a well-documented, repeatable, and still horrible process that declares all errors statistically uncommon by "augmenting" the root cause with random factors. Insta-certification.

(I lied about the "insta" part)

Re: Summary of the Amazon S3 Service Disruption

#469

Post by WillyOnWheels »

spullara wrote: post #334

Twitter once had 2 hours of downtime because an operations engineer accidentally asked a tool to restart all memcached servers instead of a certain server. The tool was then changed to make sure that you couldn't restart more than a few servers without additional confirmation. Sounds very similar to this situation. Something to think about when you are building your tools to be more error proof.

Outage was almost a day long.

Re: Summary of the Amazon S3 Service Disruption

#470

Post by [deleted] »

majewsky wrote: post #387
cobookman wrote: post #285

Earlier quoted context omitted.

11 9s of durability is about the likelyhood of AWS loosing your data. It doesn't cover the likelyhood of you being able to access your data that's called availability. For example on GCS (Google's S3)...A storage class specifies how many locations the data is made available. All storage classes share the same durability (chance of google loosing your data) of 99.999999999%, but have different availability (chance of…

> chance of google not loosing your data of 99.999999999% git commit -m 'typo'

[deleted]
Post reply on HN