Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Summary of the Amazon Kinesis Event in the Northern Virginia (US-East-1) Region

aws.amazon.com

Post reply on HN
Print view View original

101–110 of 153 posts

Re: Summary of the Amazon Kinesis Event in the Northern Virginia (US-East-1) Region

#101

Post by [deleted] »

pentlander wrote: post #43
joneholland wrote: post #40

Earlier quoted context omitted.

Are they still using Coral and Codigo as the RPC stack?

Yeah Amazon still runs on Coral, there were some recent (release a few years ago) advances on it under the hood and ergonomically. I think the "replacement" for it is Smithy[0] though it will likely just replace the XML schema and codegen and not the protocol. Honestly at this point I think it would be in Amazon's best interest to heavily invest in Java Project Loom rather than trying to convert to async. [0] https:/…

[deleted]

Re: Summary of the Amazon Kinesis Event in the Northern Virginia (US-East-1) Region

#103

Post by [deleted] »

joneholland wrote: post #40
pentlander wrote: post #32

Earlier quoted context omitted.

The hand rolled gossip protocol (DFDD) is not used for consensus, it's just used for cluster membership and health information. It's used by pretty much every foundational AWS service. There's a separate internal service for consensus that uses paxos. The thread per frontend member definitely sounds like a problematic early design choice. It wouldn't be the first time I heard of an AWS issue due to "too many threads"…

Are they still using Coral and Codigo as the RPC stack?

[deleted]

Re: Summary of the Amazon Kinesis Event in the Northern Virginia (US-East-1) Region

#104

Post by cowsandmilk »

ignoramous wrote: post #60
pentlander wrote: post #32

Earlier quoted context omitted.

The hand rolled gossip protocol (DFDD) is not used for consensus, it's just used for cluster membership and health information. It's used by pretty much every foundational AWS service. There's a separate internal service for consensus that uses paxos. The thread per frontend member definitely sounds like a problematic early design choice. It wouldn't be the first time I heard of an AWS issue due to "too many threads"…

Are you sure Kinesis uses DFDD [0]? [0] Seems like a relic of years gone by https://patents.justia.com/patent/9838240

That patent is from when Kinesis Data Streams were originally announced to the public. Any reason not to think it uses it. Seems like it would have been a logical choice in the initial architecture and change is slow.

Re: Summary of the Amazon Kinesis Event in the Northern Virginia (US-East-1) Region

#105

Post by justicezyx »

crgwbr wrote: post #94
ryanworl wrote: post #76

Earlier quoted context omitted.

128bit might be accurate, I meant more along the lines of they are non-contiguous and don't seem to be correlated with the amount of records actually being written to a stream.

I always assumed they were sequential per-region rather than per-stream, but that’s just a guess.

I recall it was sequential per-shard, but no ordering among the shards of the same stream. But I literally haven't touch Kinesis after 2013.

Re: Summary of the Amazon Kinesis Event in the Northern Virginia (US-East-1) Region

#106

Post by ActorNightly »

marcinzm wrote: post #29
joneholland wrote: post #27

Running out of file handles and other IO limits is embarrassing and happens at every company, but I’m surprised that AWS was not monitoring this. I’m also surprised at the general architecture of Kinesis. What appears to be their own hand rolled gossip protocol (that is clearly terrible compared to raft or paxos, a thread per cluster member? Everyone talking to everyone? An hour to reach consensus?) and the front end…

I don't think it's about growing fast so much as, from those I talked to, Amazon now has a fairly bad reputation in the tech community. You only go to work there if you don't have a better option (Google, Facebook, etc) or have some specialty skill they're willing to pay for. Pay is below other FAANG companies and the work culture isn't great (toxic even some would say). edit: They also had the most disorganized and…

Just as a general reminder to anyone reading this: forum comments are incredibly biased and hardly ever represent reality accurately.

Re: Summary of the Amazon Kinesis Event in the Northern Virginia (US-East-1) Region

#107

Post by ayberk »

edoceo wrote: post #47
solatic wrote: post #45

Earlier quoted context omitted.

> The problem with growing as fast as Amazon has is that their talent bar couldn’t keep up. I can’t imagine this design being okay 10 years ago when I was there. I see where you're coming from with this, but you really have to wonder. It sounds more like the original architects made implicit assumptions regarding scale that, likely due to original architects and engineers moving on, were not re-evaluated by the curre…

Oof. My little company is refactoring some five year old architecture design choices. Ugly. Process isn't visible outside the refactor and the work is tedious. Can't imagine what a service refactor is like at A. I bet it sucks

Disclaimer: I work for AWS, but I have no ties to Kinesis. Opinions are my own.

AWS has more than enough learnings to avoid these "events". The problem is the whole culture is focused on delivering new stuff instead of preventing problems and improving existing systems.

Some folks made these decisions with best intentions. I'm pretty sure they all got promoted and then moved on. Now, people who inherited these systems have no incentive to fix these, because at most you'll receive a pat in the back. I'm also pretty sure that those teams talked about the shortcomings of the current architecture. TODOs must be present somewhere in the backlog.

I don't think this is an AWS specific problem, but we have to start treating people who prevent problems like the heroes they are. Everyone congratulates when you put out a fire. No one gives a damn if you prevent the fire in the first place.

Re: Summary of the Amazon Kinesis Event in the Northern Virginia (US-East-1) Region

#108

Post by robbiemitchell »

newscom59 wrote: post #33
lytigas wrote: post #3

> During the early part of this event, we were unable to update the Service Health Dashboard because the tool we use to post these updates itself uses Cognito, which was impacted by this event. Poetry. Then, to be fair: > We have a back-up means of updating the Service Health Dashboard that has minimal service dependencies. While this worked as expected, we encountered several delays during the earlier part of the ev…

The PHD is always updated first, long before the global status page is updated. Every single one of my clients that use AWS got updates on the PHD literally hours before the status page was even showing any issues, which is typical. It’s the entire point of the PHD. Through reading Reddit and HN during this event I learned that most people apparently aren’t even aware of the existence of the PHD and rely solely on th…

> despite the fact that there is a giant “View my PHD” button at the very top of the global status page

If you don’t know what the PHD is, a big button pointing to it won’t do anything. People ignore big boxes of irrelevant stuff all the time.

AWS user of ~8 years and I’ve never heard of the PHD nor this sequencing of updating it first.

Re: Summary of the Amazon Kinesis Event in the Northern Virginia (US-East-1) Region

#109

Post by joshuamorton »

ignoramous wrote: post #61
talawahtech wrote: post #28

Earlier quoted context omitted.

Another relevant point made in the video is that they restrict cells to a maximum size which then makes it easier to test behavior at that size. This would have also helped avoid this specific issue since the number of threads would have been tied to the number of instances in a cell. I definitely recommend checking out the video. Even if you have seen it before, rewatching it in the context of this post-mortem reall…

> Another relevant point made in the video is that they restrict cells to a maximum size which then makes it easier to test behavior at that size. Googlers would be quick to point out that Borg does this natively across all their services: https://news.ycombinator.com/item?id=19393926

As another googler, I'd argue that Borg's concept of cells aren't like what Amazon is calling "cells" here. Borg cells are, as far as I can tell, akin to an AWS Zone. There are similar concepts within Google that match the concept of "an application unit that is in multiple compute units but is isolated from other similar application units, and can be used for a singular customer or workload". There are multiple terms for this concept, which I'd be happy to share within Google.

Re: Summary of the Amazon Kinesis Event in the Northern Virginia (US-East-1) Region

#110

Post by marcinzm »

ActorNightly wrote: post #106
marcinzm wrote: post #29

Earlier quoted context omitted.

I don't think it's about growing fast so much as, from those I talked to, Amazon now has a fairly bad reputation in the tech community. You only go to work there if you don't have a better option (Google, Facebook, etc) or have some specialty skill they're willing to pay for. Pay is below other FAANG companies and the work culture isn't great (toxic even some would say). edit: They also had the most disorganized and…

Just as a general reminder to anyone reading this: forum comments are incredibly biased and hardly ever represent reality accurately.

Yup, as I said, just my personal observations based on those I talked to. Your mileage will vary.
Post reply on HN