Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Summary of the Amazon Kinesis Event in the Northern Virginia (US-East-1) Region

aws.amazon.com

Post reply on HN
Print view View original

81–90 of 153 posts

Re: Summary of the Amazon Kinesis Event in the Northern Virginia (US-East-1) Region

#81

Post by [deleted] »

freehunter wrote: post #44
qz2 wrote: post #16

Earlier quoted context omitted.

Correct. I, as many people have, discovered this when something broke in one of the golden regions. In my case cloudfront and ACM. Realistically you can’t trust one provider at all if you have high availability requirements. The justification is apparently that the cloud is taking all this responsibility away from people but from personal experience running two cages of kit at two datacenters the TCO was lower and th…

>you can’t trust one provider at all The hard part with multi-cloud is, you're just increasing your risk of being impacted by someone's failure. Sure if you're all-in on AWS and AWS goes down, you're all-out. But if you're on [AWS, GCP] and GCP goes down, you're down anyway. Even though AWS is up, you're down because Google went down. And if you're on [AWS, GCP, Azure] and Azure goes down, it doesn't matter than AWS…

[deleted]

Re: Summary of the Amazon Kinesis Event in the Northern Virginia (US-East-1) Region

#82

Post by [deleted] »

hedora wrote: post #57
qz2 wrote: post #50

Earlier quoted context omitted.

I disagree. It’s about mitigating the risk of a single provider’s failure. Single providers go down all the time. We’ve seen it from all three major cloud vendors.

How do you test failover from provider-wide outages? I’ve never heard of an untested failover mechanism that worked. Most places are afraid to invoke such a thing, even during a major outage.

[deleted]

Re: Summary of the Amazon Kinesis Event in the Northern Virginia (US-East-1) Region

#83

Post by steelframe »

> Cellularization is an approach we use to isolate the effects of failure within a service, and to keep the components of the service (in this case, the shard-map cache) operating within a previously tested and operated range. This had been under way for the front-end fleet in Kinesis, but unfortunately the work is significant and had not yet been completed.

Translation: The eng team knew that they had accumulated tech debt by cutting a corner here in order to meet one of Amazon's typical and insane "just get the feature out the door" timelines. Eng warned management about it, and management decided to take the risk and lean on on-call to pull heroics to just fix any issues as they come up. Most of the time yanking a team out of bed in the middle of the night works, so that's the modus operandi at Amazon. This time, the actual problem was more fundamental and wasn't effectively addressable with middle-of-the-night heroics.

Management rolled the "just page everyone and hope they can fix it" dice yet again, as they usually do, and this time they got snake eyes.

I guarantee you that the "cellularization" of the front-end fleet wasn't actually under way, but the teams were instead completely consumed with whatever the next typical and insane "just get the feature out the door" thing was at AWS. The eng team was never going to get around to cellularizing the front-end fleet because they were given no time or incentive to do so by management. During/after this incident, I wouldn't be surprised if management didn't yell at the eng team, "Wait, you KNEW this was a problem, and you're not done yet?!?" Without recognizing that THEY are the ones actually culpable for failing to prioritize payments on tech debt vs. "new shiny" feature work, which is typical of Amazon product development culture.

I've worked with enough former AWS engineers to know what goes on there, and there's a really good reason why anybody who CAN move on from AWS will happily walk away from their 3rd- and 4th-year stock vest schedules (when the majority of your promised amount of your sign-on RSUs actually starts to vest) to flee to a company that fosters a healthy product development and engineering culture.

(Not to mention that, this time, a whole bunch of peoples' Thanksgiving plans were preempted with the demand to get a full investation and post-mortem written up, including the public post, ASAP. Was that really necessary? Couldn't it have waited until next Wednesday or something?)

Re: Summary of the Amazon Kinesis Event in the Northern Virginia (US-East-1) Region

#84

Post by lend000 »

Even today I had a few minutes of intermittent networking outages around 9:30am EST (which started on the day of the incident), and compared to other regions, I frequently get timeouts when calling S3 from us-east-1 (although that has been happening since forever).

Re: Summary of the Amazon Kinesis Event in the Northern Virginia (US-East-1) Region

#85

Post by arduinomancer »

hintymad wrote: post #78

A tangential question, why would AWS even use the term "microservice"? A service is a service, right? I'm not sure what the term "microservice" signifies here.

It’s because service can be confused with “AWS Service” which is not the same as a microservice (a component of a full service)

Re: Summary of the Amazon Kinesis Event in the Northern Virginia (US-East-1) Region

#86

Post by joneholland »

hintymad wrote: post #80
joneholland wrote: post #27

Running out of file handles and other IO limits is embarrassing and happens at every company, but I’m surprised that AWS was not monitoring this. I’m also surprised at the general architecture of Kinesis. What appears to be their own hand rolled gossip protocol (that is clearly terrible compared to raft or paxos, a thread per cluster member? Everyone talking to everyone? An hour to reach consensus?) and the front end…

AWS frontend services are usually implemented in Java. If Kinesis' frontend does too, then it's surprising that the threads created by a frontend service would exceed the OS limit. This tells three possibilities: 1. Kinesis did not impose a max thread count in their app, which is a gross omission; 2. Or there was a resource leak in their code. 3. Each of their frontend instances stored all the placement information o…

My understanding is that every front end server has at least one connection (on a dedicated thread) to every other front end server.

Assuming they have say, 5000 front end instances, thats 5000 file descriptors being used just for this, before you are even talking about whatever threads the application needs.

It’s not surprising that they bumped into ulimits, though as part of OS provisioning, you typically have those tuned for workload.

More concerning is the 5000 x 5000 amount of open tcp sessions across their network to support this architecture. This has to be a lot of fun on any stateful firewall it might cross.

Re: Summary of the Amazon Kinesis Event in the Northern Virginia (US-East-1) Region

#87

Post by WookieRushing »

throwaway189262 wrote: post #35
joneholland wrote: post #27

Running out of file handles and other IO limits is embarrassing and happens at every company, but I’m surprised that AWS was not monitoring this. I’m also surprised at the general architecture of Kinesis. What appears to be their own hand rolled gossip protocol (that is clearly terrible compared to raft or paxos, a thread per cluster member? Everyone talking to everyone? An hour to reach consensus?) and the front end…

I can't remember which db, but somebody a while back claimed that one of Amazon's "infinitely scalable" dbs was tons of abstraction on top a massive heap of MySQL instances. I don't trust anything outside core services on AWS. Regardless of whether the rumor I heard is true, it's clear they appreciate quantity over quality.

This is really common and works pretty awesomely. MySQL is extremely battle tested and there’s lots of experts out there for it.

FB built a similar system to maintain their graph: https://blog.yugabyte.com/facebooks-user-db-is-it-sql-or-nos...

It’s a ton of tiny DBs that look like one massive eventually consistent DB

Re: Summary of the Amazon Kinesis Event in the Northern Virginia (US-East-1) Region

#88

Post by why-el »

londons_explore wrote: post #79

One requirement on my "production ready" checklist is that any catastrophic system failure can be resolved by starting a completely new instance of the service, and it be ready to serve traffic inside 10 minutes. That should be tested at least quarterly (but preferably automatically with every build). If Amazon did that, this outage would have been reduced to 10 mins, rather than the 12+ hours that some super slow ro…

The same OS limits would apply to new instances, unless they knew the root cause and forced new instances to be configured with larger descriptor limits, which is....well, hindsight is 20/20, no?

Re: Summary of the Amazon Kinesis Event in the Northern Virginia (US-East-1) Region

#89

Post by loriverkutya »

lytigas wrote: post #3

> During the early part of this event, we were unable to update the Service Health Dashboard because the tool we use to post these updates itself uses Cognito, which was impacted by this event. Poetry. Then, to be fair: > We have a back-up means of updating the Service Health Dashboard that has minimal service dependencies. While this worked as expected, we encountered several delays during the earlier part of the ev…

I can confirm we got the Personal Health Dashboard notifications.

Re: Summary of the Amazon Kinesis Event in the Northern Virginia (US-East-1) Region

#90

Post by why-el »

karmakaze wrote: post #53

Seems to me that the root problem could also be fixed by not using presumably blocking application threads talking to each of the other servers. Any async or poll mechanism wouldn't require N^2 threads across the pool.

I wonder if the new wonders coming out from linux (io_uring...) would have made this a better design, but that work in the kernel is still in active development.
Post reply on HN