Summary of the Amazon Kinesis Event in the Northern Virginia (US-East-1) Region • Hacker News Board

Re: Summary of the Amazon Kinesis Event in the Northern Virginia (US-East-1) Region

#41

Post by qz2 » Sat, Nov 28, 2020, 3:46 PM UTC

lttlrck wrote: post #36

qz2 wrote: post #16

Earlier quoted context omitted.

Correct. I, as many people have, discovered this when something broke in one of the golden regions. In my case cloudfront and ACM. Realistically you can’t trust one provider at all if you have high availability requirements. The justification is apparently that the cloud is taking all this responsibility away from people but from personal experience running two cages of kit at two datacenters the TCO was lower and th…

Harry-Potter-esque pricing? Is that a reference to the difficulty of calculating the cost of visiting all the rides at universal? That's my best guess...

It's more a stab at the inconsistency of rules around magic.

"Well this pricing rule only works on a Tuesday lunch time if you're standing on one leg with a sausage under each arm and a traffic cone on your head"

And there are a million of those to navigate.

Re: Summary of the Amazon Kinesis Event in the Northern Virginia (US-East-1) Region

#42

Post by morsma » Sat, Nov 28, 2020, 3:59 PM UTC

joneholland wrote: post #40

pentlander wrote: post #32

Earlier quoted context omitted.

The hand rolled gossip protocol (DFDD) is not used for consensus, it's just used for cluster membership and health information. It's used by pretty much every foundational AWS service. There's a separate internal service for consensus that uses paxos. The thread per frontend member definitely sounds like a problematic early design choice. It wouldn't be the first time I heard of an AWS issue due to "too many threads"…

Are they still using Coral and Codigo as the RPC stack?

Not Codigo, but Coral, yes.

Re: Summary of the Amazon Kinesis Event in the Northern Virginia (US-East-1) Region

#43

Post by pentlander » Sat, Nov 28, 2020, 4:04 PM UTC

joneholland wrote: post #40

pentlander wrote: post #32

Earlier quoted context omitted.

The hand rolled gossip protocol (DFDD) is not used for consensus, it's just used for cluster membership and health information. It's used by pretty much every foundational AWS service. There's a separate internal service for consensus that uses paxos. The thread per frontend member definitely sounds like a problematic early design choice. It wouldn't be the first time I heard of an AWS issue due to "too many threads"…

Are they still using Coral and Codigo as the RPC stack?

Yeah Amazon still runs on Coral, there were some recent (release a few years ago) advances on it under the hood and ergonomically. I think the "replacement" for it is Smithy[0] though it will likely just replace the XML schema and codegen and not the protocol. Honestly at this point I think it would be in Amazon's best interest to heavily invest in Java Project Loom rather than trying to convert to async.

[0] https://awslabs.github.io/smithy/

Re: Summary of the Amazon Kinesis Event in the Northern Virginia (US-East-1) Region

#44

Post by freehunter » Sat, Nov 28, 2020, 4:12 PM UTC

qz2 wrote: post #16

EdwardDiego wrote: post #10

Earlier quoted context omitted.

Indeed. We're paying (and designing our systems to work on multiple AZs) to reduce the risk of outages, but then their back-end services are reliant on services in a sole region?

Correct. I, as many people have, discovered this when something broke in one of the golden regions. In my case cloudfront and ACM. Realistically you can’t trust one provider at all if you have high availability requirements. The justification is apparently that the cloud is taking all this responsibility away from people but from personal experience running two cages of kit at two datacenters the TCO was lower and th…

>you can’t trust one provider at all

The hard part with multi-cloud is, you're just increasing your risk of being impacted by someone's failure. Sure if you're all-in on AWS and AWS goes down, you're all-out. But if you're on [AWS, GCP] and GCP goes down, you're down anyway. Even though AWS is up, you're down because Google went down. And if you're on [AWS, GCP, Azure] and Azure goes down, it doesn't matter than AWS and GCP are up... you're down because Azure is down. The only way around that is architecting your business to run with only one of those vendors, which means you're paying 3x more than you need to 99.99999% of the time.

The probability that one of [AWS, Azure, GCP] is down is way higher than the probability that just one of them is down. And the probability that your two cages in your datacenter is down is way higher than the probability that any one of the hyperscalers is down.

Re: Summary of the Amazon Kinesis Event in the Northern Virginia (US-East-1) Region

#45

Post by solatic » Sat, Nov 28, 2020, 4:28 PM UTC

joneholland wrote: post #27

Running out of file handles and other IO limits is embarrassing and happens at every company, but I’m surprised that AWS was not monitoring this. I’m also surprised at the general architecture of Kinesis. What appears to be their own hand rolled gossip protocol (that is clearly terrible compared to raft or paxos, a thread per cluster member? Everyone talking to everyone? An hour to reach consensus?) and the front end…

> The problem with growing as fast as Amazon has is that their talent bar couldn’t keep up. I can’t imagine this design being okay 10 years ago when I was there.

I see where you're coming from with this, but you really have to wonder. It sounds more like the original architects made implicit assumptions regarding scale that, likely due to original architects and engineers moving on, were not re-evaluated by the current engineers on Kinesis as Kinesis grew. While it may take an hour now for the front-end cache to sync, I find it highly unlikely that it needed that much time when Kinesis first launched.

The process failure here is organizational, one where Amazon failed to audit its current systems in a complete and current manner such that sufficient attention and resources could be paid to a re-architecture of a critical service before it caused the service to fail. Even now, vertically scaling the front-end cache fleet is just a band-aid - eventually, that won't be possible anymore. Sadly, the postmortem doesn't seem to identify the organizational failure that was the true root cause of the outage.

Re: Summary of the Amazon Kinesis Event in the Northern Virginia (US-East-1) Region

#46

Post by karmakaze » Sat, Nov 28, 2020, 4:40 PM UTC

joneholland wrote: post #27

Running out of file handles and other IO limits is embarrassing and happens at every company, but I’m surprised that AWS was not monitoring this. I’m also surprised at the general architecture of Kinesis. What appears to be their own hand rolled gossip protocol (that is clearly terrible compared to raft or paxos, a thread per cluster member? Everyone talking to everyone? An hour to reach consensus?) and the front end…

Kinesis was the worst AWS tech I've ever used. Splitting a stream into shards doesn't increase throughput if you still need to run the same types/number of consumers on every shard. The suggested workaround at the time was to use larger batches and add latency to the processing pipeline.

Re: Summary of the Amazon Kinesis Event in the Northern Virginia (US-East-1) Region

#47

Post by edoceo » Sat, Nov 28, 2020, 4:40 PM UTC

solatic wrote: post #45

joneholland wrote: post #27

Running out of file handles and other IO limits is embarrassing and happens at every company, but I’m surprised that AWS was not monitoring this. I’m also surprised at the general architecture of Kinesis. What appears to be their own hand rolled gossip protocol (that is clearly terrible compared to raft or paxos, a thread per cluster member? Everyone talking to everyone? An hour to reach consensus?) and the front end…

> The problem with growing as fast as Amazon has is that their talent bar couldn’t keep up. I can’t imagine this design being okay 10 years ago when I was there. I see where you're coming from with this, but you really have to wonder. It sounds more like the original architects made implicit assumptions regarding scale that, likely due to original architects and engineers moving on, were not re-evaluated by the curre…

Oof. My little company is refactoring some five year old architecture design choices. Ugly. Process isn't visible outside the refactor and the work is tedious. Can't imagine what a service refactor is like at A. I bet it sucks

Re: Summary of the Amazon Kinesis Event in the Northern Virginia (US-East-1) Region

#48

Post by imajoredinecon » Sat, Nov 28, 2020, 4:44 PM UTC

marcinzm wrote: post #29

joneholland wrote: post #27

Running out of file handles and other IO limits is embarrassing and happens at every company, but I’m surprised that AWS was not monitoring this. I’m also surprised at the general architecture of Kinesis. What appears to be their own hand rolled gossip protocol (that is clearly terrible compared to raft or paxos, a thread per cluster member? Everyone talking to everyone? An hour to reach consensus?) and the front end…

I don't think it's about growing fast so much as, from those I talked to, Amazon now has a fairly bad reputation in the tech community. You only go to work there if you don't have a better option (Google, Facebook, etc) or have some specialty skill they're willing to pay for. Pay is below other FAANG companies and the work culture isn't great (toxic even some would say). edit: They also had the most disorganized and…

Interesting re interview experience

I interviewed as a new grad SWE and the process was totally straightforward, and way lower friction (albeit much less human interaction, which made it feel even more impersonal) than almost everywhere else I applied: initial online screen, online programming task, and then a video call with an engineer where you explained your answer to the programming task.

Re: Summary of the Amazon Kinesis Event in the Northern Virginia (US-East-1) Region

#49

Post by rubiquity » Sat, Nov 28, 2020, 4:50 PM UTC

throwaway189262 wrote: post #35

joneholland wrote: post #27

Running out of file handles and other IO limits is embarrassing and happens at every company, but I’m surprised that AWS was not monitoring this. I’m also surprised at the general architecture of Kinesis. What appears to be their own hand rolled gossip protocol (that is clearly terrible compared to raft or paxos, a thread per cluster member? Everyone talking to everyone? An hour to reach consensus?) and the front end…

I can't remember which db, but somebody a while back claimed that one of Amazon's "infinitely scalable" dbs was tons of abstraction on top a massive heap of MySQL instances. I don't trust anything outside core services on AWS. Regardless of whether the rumor I heard is true, it's clear they appreciate quantity over quality.

Disclosure: I work at AWS, possibly near the system you're describing. Opinions are my own.

If we're talking about the same thing then I think casting stones just because it is based on MySQL is severely misguided. MySQL has decades of optimizations and this particular system at Amazon has solved scaling problems and brought reliability to countless services without ever being the direct cause of an outage (to the best of my knowledge).

Indeed, MySQL is not without its flaws but many of these are related to its quirks in transactions and replication which this system completely solves. The cherry on top is that you have a rock solid database with a familiar querying language and a massive knowledge base to get help from when needed. Oh, and did I mention this system supports multiple storage engines besides just MySQL/InnoDB?

I for one wish we would open source this system though there are a ton of hurdles both technical and not. I think it would do wonders for the greater tech community by providing a much better option as your needs grow beyond a single node system. It has certainly served Amazon well in that role and I've heard Facebook and YouTube have similar systems based on MySQL.

To further address your comment about Amazon/AWS lacking quality: this system is the epitome of our values of pragmatism and focusing our efforts on innovating where we can make the biggest impact. Hand rolling your own storage engines is fun and all but countless others have already spent decades doing so for marginal gains.

Re: Summary of the Amazon Kinesis Event in the Northern Virginia (US-East-1) Region

#50

Post by qz2 » Sat, Nov 28, 2020, 4:56 PM UTC

freehunter wrote: post #44

qz2 wrote: post #16

Earlier quoted context omitted.

Correct. I, as many people have, discovered this when something broke in one of the golden regions. In my case cloudfront and ACM. Realistically you can’t trust one provider at all if you have high availability requirements. The justification is apparently that the cloud is taking all this responsibility away from people but from personal experience running two cages of kit at two datacenters the TCO was lower and th…

>you can’t trust one provider at all The hard part with multi-cloud is, you're just increasing your risk of being impacted by someone's failure. Sure if you're all-in on AWS and AWS goes down, you're all-out. But if you're on [AWS, GCP] and GCP goes down, you're down anyway. Even though AWS is up, you're down because Google went down. And if you're on [AWS, GCP, Azure] and Azure goes down, it doesn't matter than AWS…

I disagree. It’s about mitigating the risk of a single provider’s failure. Single providers go down all the time. We’ve seen it from all three major cloud vendors.