Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Summary of the Amazon Kinesis Event in the Northern Virginia (US-East-1) Region

aws.amazon.com

Post reply on HN
Print view View original

111–120 of 153 posts

Re: Summary of the Amazon Kinesis Event in the Northern Virginia (US-East-1) Region

#111

Post by foota »

joneholland wrote: post #86
hintymad wrote: post #80

Earlier quoted context omitted.

AWS frontend services are usually implemented in Java. If Kinesis' frontend does too, then it's surprising that the threads created by a frontend service would exceed the OS limit. This tells three possibilities: 1. Kinesis did not impose a max thread count in their app, which is a gross omission; 2. Or there was a resource leak in their code. 3. Each of their frontend instances stored all the placement information o…

My understanding is that every front end server has at least one connection (on a dedicated thread) to every other front end server. Assuming they have say, 5000 front end instances, thats 5000 file descriptors being used just for this, before you are even talking about whatever threads the application needs. It’s not surprising that they bumped into ulimits, though as part of OS provisioning, you typically have thos…

The tcp connections are probably not an issue, working in cloud it's never something I've seen worried about, so maybe the architecture doesn't have that limitation?

Re: Summary of the Amazon Kinesis Event in the Northern Virginia (US-East-1) Region

#112

Post by Twirrim »

pentlander wrote: post #77
Twirrim wrote: post #69

Earlier quoted context omitted.

> Can't imagine what a service refactor is like at A. I bet it sucks It's not all that hard. AWS heavily focuses on Service Oriented Architecture approaches, with specific knowledge/responsibility domains for each. It's a proven scalable pattern. The APIs will often be fairly straight-forward behind the front end. With clearly lines of responsibility between components, you'll almost never have to worry about what ot…

As another ex-Glacier dev, I disagree with it not being "all that hard" but I agree with everything else. Now I'm curious who this is :)

The English guy, if that's enough of a clue. It has been about 4 1/2 years now since I left Glacier, so there's every chance our paths never overlapped.

Re: Summary of the Amazon Kinesis Event in the Northern Virginia (US-East-1) Region

#113

Post by roman_sf »

Twirrim wrote: post #69
edoceo wrote: post #47

Earlier quoted context omitted.

Oof. My little company is refactoring some five year old architecture design choices. Ugly. Process isn't visible outside the refactor and the work is tedious. Can't imagine what a service refactor is like at A. I bet it sucks

> Can't imagine what a service refactor is like at A. I bet it sucks It's not all that hard. AWS heavily focuses on Service Oriented Architecture approaches, with specific knowledge/responsibility domains for each. It's a proven scalable pattern. The APIs will often be fairly straight-forward behind the front end. With clearly lines of responsibility between components, you'll almost never have to worry about what ot…

>> A design proposal gets created and refined over a few weeks of presentations to the team and direct leadership.

Dat soundz like a bank and not a cloud provider.

Re: Summary of the Amazon Kinesis Event in the Northern Virginia (US-East-1) Region

#114

Post by toomuchtodo »

ayberk wrote: post #107
edoceo wrote: post #47

Earlier quoted context omitted.

Oof. My little company is refactoring some five year old architecture design choices. Ugly. Process isn't visible outside the refactor and the work is tedious. Can't imagine what a service refactor is like at A. I bet it sucks

Disclaimer: I work for AWS, but I have no ties to Kinesis. Opinions are my own. AWS has more than enough learnings to avoid these "events". The problem is the whole culture is focused on delivering new stuff instead of preventing problems and improving existing systems. Some folks made these decisions with best intentions. I'm pretty sure they all got promoted and then moved on. Now, people who inherited these system…

> I don't think this is an AWS specific problem, but we have to start treating people who prevent problems like the heroes they are. Everyone congratulates when you put out a fire.

This is sort of comforting to hear that Google’s same problems have reached Amazon, in that no tech behemoth is immune to prioritizing promotion and glitz over the maintenance grind.

> No one gives a damn if you prevent the fire in the first place.

Amen.

Re: Summary of the Amazon Kinesis Event in the Northern Virginia (US-East-1) Region

#115

Post by ThrowawayR2 »

ayberk wrote: post #107
edoceo wrote: post #47

Earlier quoted context omitted.

Oof. My little company is refactoring some five year old architecture design choices. Ugly. Process isn't visible outside the refactor and the work is tedious. Can't imagine what a service refactor is like at A. I bet it sucks

Disclaimer: I work for AWS, but I have no ties to Kinesis. Opinions are my own. AWS has more than enough learnings to avoid these "events". The problem is the whole culture is focused on delivering new stuff instead of preventing problems and improving existing systems. Some folks made these decisions with best intentions. I'm pretty sure they all got promoted and then moved on. Now, people who inherited these system…

> "The problem is the whole culture is focused on delivering new stuff instead of preventing problems and improving existing systems."

The same disease exists at other FAANGs and large tech companies. Nobody ever gets a promo for maintenance work and being on a sustained engineering team is seen as a career dead-end.

Re: Summary of the Amazon Kinesis Event in the Northern Virginia (US-East-1) Region

#116

Post by roman_sf »

otterley wrote: post #96
EdwardDiego wrote: post #10

Earlier quoted context omitted.

Indeed. We're paying (and designing our systems to work on multiple AZs) to reduce the risk of outages, but then their back-end services are reliant on services in a sole region?

(Disclaimer: I work for AWS but opinions are my own. I also do not work with the Kinesis team.) Nearly all AWS services are regional in scope, and for many (if not most) services, they are scaled at a cellular level within a region. Accounts are assigned to specific cells within that region. There are very, very few services that are global in scope, and it is strongly discouraged to create cross-regional dependencie…

What do you mean by cross-regional dependencies? Isn't running in multi-region setup is by itself adding dependency?

Speaking about multi-region services. What do you think about Google now offering all three major building pieces as multi-regional?

They have muti-regional buckets, LB with single anycast IP, document db (firebase). Pubsub can route automatically to nearest region. Nothing like this is available in amazon, well only DIY building blocks.

Re: Summary of the Amazon Kinesis Event in the Northern Virginia (US-East-1) Region

#117

Post by mooreds »

toomuchtodo wrote: post #114
ayberk wrote: post #107

Earlier quoted context omitted.

Disclaimer: I work for AWS, but I have no ties to Kinesis. Opinions are my own. AWS has more than enough learnings to avoid these "events". The problem is the whole culture is focused on delivering new stuff instead of preventing problems and improving existing systems. Some folks made these decisions with best intentions. I'm pretty sure they all got promoted and then moved on. Now, people who inherited these system…

> I don't think this is an AWS specific problem, but we have to start treating people who prevent problems like the heroes they are. Everyone congratulates when you put out a fire. This is sort of comforting to hear that Google’s same problems have reached Amazon, in that no tech behemoth is immune to prioritizing promotion and glitz over the maintenance grind. > No one gives a damn if you prevent the fire in the fir…

> This is sort of comforting

I would personally put emphasis on the "sort of" clause.

If bigcos with tons of resources can't align incentives to build solid software instead of deliver new features, who among us has a chance?

Re: Summary of the Amazon Kinesis Event in the Northern Virginia (US-East-1) Region

#118

Post by roman_sf »

freeone3000 wrote: post #4

The failure to update the Service Health Dashboard was due to reliance on internal services to update. This also happened in March 2017[0]. Perhaps a general, instead of piecemeal, approach to removing dependencies on running services from the dashboard would be valuable here? 0: https://aws.amazon.com/message/41926/

But this time it was a _different_ dependency. They just want to make sure all dependencies are ruled out before migrating to GCP :)

Re: Summary of the Amazon Kinesis Event in the Northern Virginia (US-East-1) Region

#119

Post by mooreds »

Twirrim wrote: post #69
edoceo wrote: post #47

Earlier quoted context omitted.

Oof. My little company is refactoring some five year old architecture design choices. Ugly. Process isn't visible outside the refactor and the work is tedious. Can't imagine what a service refactor is like at A. I bet it sucks

> Can't imagine what a service refactor is like at A. I bet it sucks It's not all that hard. AWS heavily focuses on Service Oriented Architecture approaches, with specific knowledge/responsibility domains for each. It's a proven scalable pattern. The APIs will often be fairly straight-forward behind the front end. With clearly lines of responsibility between components, you'll almost never have to worry about what ot…

> "respect what has gone before"

Ah, Chesterton’s Fence strikes again!

https://en.wikipedia.org/wiki/G._K._Chesterton#Chesterton's_...

Re: Summary of the Amazon Kinesis Event in the Northern Virginia (US-East-1) Region

#120

Post by roman_sf »

steelframe wrote: post #83

> Cellularization is an approach we use to isolate the effects of failure within a service, and to keep the components of the service (in this case, the shard-map cache) operating within a previously tested and operated range. This had been under way for the front-end fleet in Kinesis, but unfortunately the work is significant and had not yet been completed. Translation: The eng team knew that they had accumulated te…

Hahaha, well this time Jessy got paged so yeah... the summary got priority over turkey.
Post reply on HN