From the summary I don't understand why front end servers need to talk to each other ("continuous processing of messages from other Kinesis front-end servers"). It sounds like this is part of building the shard map or the cache. Well in the end an unfortunate design decision. #hugops for the team handling this. Cascading failures are the worst.
Summary of the Amazon Kinesis Event in the Northern Virginia (US-East-1) Region
31–40 of 153 posts
Re: Summary of the Amazon Kinesis Event in the Northern Virginia (US-East-1) Region
#32Running out of file handles and other IO limits is embarrassing and happens at every company, but I’m surprised that AWS was not monitoring this. I’m also surprised at the general architecture of Kinesis. What appears to be their own hand rolled gossip protocol (that is clearly terrible compared to raft or paxos, a thread per cluster member? Everyone talking to everyone? An hour to reach consensus?) and the front end…
The hand rolled gossip protocol (DFDD) is not used for consensus, it's just used for cluster membership and health information. It's used by pretty much every foundational AWS service. There's a separate internal service for consensus that uses paxos.
The thread per frontend member definitely sounds like a problematic early design choice. It wouldn't be the first time I heard of an AWS issue due to "too many threads". Unlike gRPC, the internal RPC framework defaults to a thread per request rather than an async model. The async way was pretty painful and error prone.
Re: Summary of the Amazon Kinesis Event in the Northern Virginia (US-East-1) Region
#33> During the early part of this event, we were unable to update the Service Health Dashboard because the tool we use to post these updates itself uses Cognito, which was impacted by this event. Poetry. Then, to be fair: > We have a back-up means of updating the Service Health Dashboard that has minimal service dependencies. While this worked as expected, we encountered several delays during the earlier part of the ev…
The PHD is always updated first, long before the global status page is updated. Every single one of my clients that use AWS got updates on the PHD literally hours before the status page was even showing any issues, which is typical. It’s the entire point of the PHD.
Through reading Reddit and HN during this event I learned that most people apparently aren’t even aware of the existence of the PHD and rely solely on the global status page, despite the fact that there is a giant “View my PHD” button at the very top of the global status page, and additionally there is a notification icon on the header of every AWS console page that lights up and links you directly to the PHD whenever there is an issue.
The PHD is always where you should look first. It is, by design, updated long before the global status page is.
Re: Summary of the Amazon Kinesis Event in the Northern Virginia (US-East-1) Region
#34> During the early part of this event, we were unable to update the Service Health Dashboard because the tool we use to post these updates itself uses Cognito, which was impacted by this event. Poetry. Then, to be fair: > We have a back-up means of updating the Service Health Dashboard that has minimal service dependencies. While this worked as expected, we encountered several delays during the earlier part of the ev…
Yes, we had some messages coming through in our PHD.
Re: Summary of the Amazon Kinesis Event in the Northern Virginia (US-East-1) Region
#35Running out of file handles and other IO limits is embarrassing and happens at every company, but I’m surprised that AWS was not monitoring this. I’m also surprised at the general architecture of Kinesis. What appears to be their own hand rolled gossip protocol (that is clearly terrible compared to raft or paxos, a thread per cluster member? Everyone talking to everyone? An hour to reach consensus?) and the front end…
I can't remember which db, but somebody a while back claimed that one of Amazon's "infinitely scalable" dbs was tons of abstraction on top a massive heap of MySQL instances.
I don't trust anything outside core services on AWS. Regardless of whether the rumor I heard is true, it's clear they appreciate quantity over quality.
Re: Summary of the Amazon Kinesis Event in the Northern Virginia (US-East-1) Region
#36Earlier quoted context omitted.
Indeed. We're paying (and designing our systems to work on multiple AZs) to reduce the risk of outages, but then their back-end services are reliant on services in a sole region?
Correct. I, as many people have, discovered this when something broke in one of the golden regions. In my case cloudfront and ACM. Realistically you can’t trust one provider at all if you have high availability requirements. The justification is apparently that the cloud is taking all this responsibility away from people but from personal experience running two cages of kit at two datacenters the TCO was lower and th…
Harry-Potter-esque pricing?
Is that a reference to the difficulty of calculating the cost of visiting all the rides at universal? That's my best guess...
Re: Summary of the Amazon Kinesis Event in the Northern Virginia (US-East-1) Region
#37Running out of file handles and other IO limits is embarrassing and happens at every company, but I’m surprised that AWS was not monitoring this. I’m also surprised at the general architecture of Kinesis. What appears to be their own hand rolled gossip protocol (that is clearly terrible compared to raft or paxos, a thread per cluster member? Everyone talking to everyone? An hour to reach consensus?) and the front end…
I can't remember which db, but somebody a while back claimed that one of Amazon's "infinitely scalable" dbs was tons of abstraction on top a massive heap of MySQL instances. I don't trust anything outside core services on AWS. Regardless of whether the rumor I heard is true, it's clear they appreciate quantity over quality.
Sounds like Dynamo
Re: Summary of the Amazon Kinesis Event in the Northern Virginia (US-East-1) Region
#38Running out of file handles and other IO limits is embarrassing and happens at every company, but I’m surprised that AWS was not monitoring this. I’m also surprised at the general architecture of Kinesis. What appears to be their own hand rolled gossip protocol (that is clearly terrible compared to raft or paxos, a thread per cluster member? Everyone talking to everyone? An hour to reach consensus?) and the front end…
> What appears to be their own hand rolled gossip protocol (that is clearly terrible compared to raft or paxos) Raft and Paxos are not gossip protocols - they are consensus protocols.
Fair. What I meant to say is “hand rolling a way to have consensus on a shared piece of data” by implementing it with a naive gossip system.
Re: Summary of the Amazon Kinesis Event in the Northern Virginia (US-East-1) Region
#39Running out of file handles and other IO limits is embarrassing and happens at every company, but I’m surprised that AWS was not monitoring this. I’m also surprised at the general architecture of Kinesis. What appears to be their own hand rolled gossip protocol (that is clearly terrible compared to raft or paxos, a thread per cluster member? Everyone talking to everyone? An hour to reach consensus?) and the front end…
I can't remember which db, but somebody a while back claimed that one of Amazon's "infinitely scalable" dbs was tons of abstraction on top a massive heap of MySQL instances. I don't trust anything outside core services on AWS. Regardless of whether the rumor I heard is true, it's clear they appreciate quantity over quality.
What’s the problem with this? Anything that’s scalable is “just an abstraction” on top of a heap of shards/processes/nodes/data centers/whatever.
Re: Summary of the Amazon Kinesis Event in the Northern Virginia (US-East-1) Region
#40Running out of file handles and other IO limits is embarrassing and happens at every company, but I’m surprised that AWS was not monitoring this. I’m also surprised at the general architecture of Kinesis. What appears to be their own hand rolled gossip protocol (that is clearly terrible compared to raft or paxos, a thread per cluster member? Everyone talking to everyone? An hour to reach consensus?) and the front end…
The hand rolled gossip protocol (DFDD) is not used for consensus, it's just used for cluster membership and health information. It's used by pretty much every foundational AWS service. There's a separate internal service for consensus that uses paxos. The thread per frontend member definitely sounds like a problematic early design choice. It wouldn't be the first time I heard of an AWS issue due to "too many threads"…
Are they still using Coral and Codigo as the RPC stack?