Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Summary of the Amazon S3 Service Disruption

aws.amazon.com

Post reply on HN
Print view View original

411–420 of 535 posts

Re: Summary of the Amazon S3 Service Disruption

#411

Post by richardwhiuk »

i336_ wrote: post #388

> (...) [W]e have not completely restarted the index subsystem or the placement subsystem in our larger regions for many years. S3 has experienced massive growth over the last several years and the process of restarting these services and running the necessary safety checks to validate the integrity of the metadata took longer than expected. All those tweets saying "turn it off and back on again"? "We accidentally tu…

I don't think you don't understand the architecture of the system, if you are describing the indexer as a SPOF.

The system is a collection of shards. If you replicate it to create a second shard, then you'll just have a large a system, which is still a single point of failure.

The index, by necessity, has to be able to answer the question 'this object exists' or 'this object doesn't exit' - so it needs to have consensus.

Re: Summary of the Amazon S3 Service Disruption

#413

Post by jsmeaton »

jph00 wrote: post #10

> From the beginning of this event until 11:37AM PST, we were unable to update the individual services’ status on the AWS Service Health Dashboard (SHD) because of a dependency the SHD administration console has on Amazon S3. Ensuring that your status dashboard doesn't depend on the thing it's monitoring is probably the first thing you should think about when designing your status system. This doesn't fill me with co…

How many companies do you think running on AWS host their status page on S3?

Re: Summary of the Amazon S3 Service Disruption

#414

Post by [deleted] »

ckozlowski wrote: post #199
erikbye wrote: post #175

Earlier quoted context omitted.

What about the performance of serving it? Sounds like I would need to cache it myself, anyway.

Performance out of S3 is generally really good. However, if you're looking to say, serve up a global website and your content is in a single S3 region, then you can leverage CloudFront CDN to serve up those objects. CloudFront integrates seamlessly with S3, and you don't pay transfer charges between CloudFront and S3.

[deleted]

Re: Summary of the Amazon S3 Service Disruption

#415

Post by i336_ »

richardwhiuk wrote: post #411
i336_ wrote: post #388

> (...) [W]e have not completely restarted the index subsystem or the placement subsystem in our larger regions for many years. S3 has experienced massive growth over the last several years and the process of restarting these services and running the necessary safety checks to validate the integrity of the metadata took longer than expected. All those tweets saying "turn it off and back on again"? "We accidentally tu…

I don't think you don't understand the architecture of the system, if you are describing the indexer as a SPOF. The system is a collection of shards. If you replicate it to create a second shard, then you'll just have a large a system, which is still a single point of failure. The index, by necessity, has to be able to answer the question 'this object exists' or 'this object doesn't exit' - so it needs to have consen…

Hrm.

My speculative presumption was going off the sole datapoint of "we have not completely restarted the index subsystem or the placement subsystem in our larger regions for many years". I'm not quite sure how to interpret "restart" in this context, mostly due to lack of exposure or experience.

The report also says "Unfortunately, one of the inputs to the command was entered incorrectly and a larger set of servers was removed than intended. The servers that were inadvertently removed supported two other S3 subsystems." So you're right, it looks like multiple servers were supporting these systems, which does make sense (especially considering the load they would have seen). Okay.

I guess I didn't quite think through the load requirements and thought these were single machines - which is certainly ludicrous thinking :) - and that's where I got the SPOF reasoning from.

You're very right though, these consensus systems must be built as bottlenecks in order to see everything.

And there aren't really any alternatives: "build extra indexers and placement systems!" just gives you "but what if _all_ of them get taken offline?" and "it can't leave the datacenter, it sees 100GB/s of throughput" (number taken out of thin air).

Good points.

Re: Summary of the Amazon S3 Service Disruption

#416

Post by colanderman »

TeMPOraL wrote: post #252
colanderman wrote: post #219

Earlier quoted context omitted.

It's not really a UI issue. The tool as a whole should incorporate a model of S3. Any action you take through the UI should first be applied to this model, and then the resulting impact analyzed. If the impact is "service goes down", then don't apply the action without raising red flags. Where I work we use PCS for high availability, and it bugs the heck out of me that a fat-fingered command can bring down a service.…

Interesting. In practice, it would likely be very hard to make a model of your infrastructure to test against, but I can imagine a tool that would run each query against a set of heuristics, and if any flags pop up, it would make you jump through some hoops to confirm. Such a tool should NEVER have an option to silently confirm, and the only way to adjust a heuristic if it becomes invalid should be formally getting s…

For many systems, the rule is simply "X of Y servers must be up". Something like that isn't too hard to enforce.

Re: Summary of the Amazon S3 Service Disruption

#417

Post by askmike »

TeMPOraL wrote: post #402
Piskvorrr wrote: post #372

Earlier quoted context omitted.

Habituation is a powerful thing: a safety-critical program used in the 90s had a similar, hard-coded safety prompt (<10 uppercase ASCII characters). Within a few weeks, all elevated permission users had the combination committed to muscle memory and would bang it out without hesitation, just by reflex: "Warning: please confirm these potentially unsaf-" "IAGREE!"

It's indeed a real problem. Hell, I myself am habituated to logins and passwords for frequently used dialog boxes, and so just two days ago I tried to log in on my work's JIRA account using test credentials for an app we're developing... For securing very dangerous commands, I'd recommend asking the user to retype a phrase composed of random words, or maybe a random 8-character hexadecimal number - something that's d…

I think that even if someone can't memorize the exact characters, they'll memorize the task of having to type over the characters. Better would be to never ask for confirmation except in the worst of worst cases.

Re: Summary of the Amazon S3 Service Disruption

#418

Post by justin_oaks »

jacquesm wrote: post #398
Bartweiss wrote: post #240

Earlier quoted context omitted.

Any time I see "we're going to train everyone better" or "we're going to fire the guy who did it", all I can read is "this will happen again". You can't actually solve the problem of user error with training, and it's good to see Amazon not playing that game.

What bothered me about running TrueTech is that customers would sometimes demand repercussions against employees for making mistakes. Enter Frans Plugge. Whenever a customer would get into that mode we'd fire Frans. This was easy, simply because he didn't exist in the first place (his name was pulled from a skit by two Dutch comedians, bonus points if you know who and which skit). This usually then caused the custome…

It is unreasonable for me to think that company owners should have the spine to say, "We take the decision to fire someone very seriously. We'll take your comments under consideration, but we retain sole discretion over such decisions."

It irks me that businesses fire people because of pressure from clients or social media. But having never been the boss, I may be missing something.

Re: Summary of the Amazon S3 Service Disruption

#419

Post by twistedpair »

snewman wrote: post #131
conorh wrote: post #17

This part is also interesting: > While this is an operation that we have relied on to maintain our systems since the launch of S3, we have not completely restarted the index subsystem or the placement subsystem in our larger regions for many years. These sorts of things make me understand why the Netflix "Chaos Gorilla" style of operating is so important. As they say in this post: > We build our systems with the assu…

> Failure at every level has to be simulated pretty often to understand how to handle it, and it is a really difficult problem to solve well. Exactly. It seems likely that Amazon tests the restart operation, but it would be hard to test it at full us-east-1 scale. Running a full S3 test cluster at that scale would likely be a prohibitive expense. Perhaps the "index subsystem" and "placement subsystem" are small enoug…

> Failure at every level has to be simulated pretty often to understand how to handle it

Keep in mind, S3 "fails" all the time. We regularly make millions of S3 requests at my work. Usually we get 1:240K failure rate (mostly GETs), returning 500 errors. However, if you're really hammering an S3 node in the hash ring (e.g. Spark job), we see failures in the 1/10K range, including SocketExceptions, where the routed IP is dead.

You need to always expect such services to die in your code, setting the proper timeouts, backoffs, retries, queues, and dead letter queues.

Re: Summary of the Amazon S3 Service Disruption

#420

Post by spydum »

conorh wrote: post #17

This part is also interesting: > While this is an operation that we have relied on to maintain our systems since the launch of S3, we have not completely restarted the index subsystem or the placement subsystem in our larger regions for many years. These sorts of things make me understand why the Netflix "Chaos Gorilla" style of operating is so important. As they say in this post: > We build our systems with the assu…

Totally agree. Would also point out that if you have systems up for many years, they like haven't been updated in the same... shouldn't people find that alarming?
Post reply on HN