Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Summary of the Amazon Kinesis Event in the Northern Virginia (US-East-1) Region

aws.amazon.com

Post reply on HN
Print view View original

121–130 of 153 posts

Re: Summary of the Amazon Kinesis Event in the Northern Virginia (US-East-1) Region

#121

Post by [deleted] »

mooreds wrote: post #117
toomuchtodo wrote: post #114

Earlier quoted context omitted.

> I don't think this is an AWS specific problem, but we have to start treating people who prevent problems like the heroes they are. Everyone congratulates when you put out a fire. This is sort of comforting to hear that Google’s same problems have reached Amazon, in that no tech behemoth is immune to prioritizing promotion and glitz over the maintenance grind. > No one gives a damn if you prevent the fire in the fir…

> This is sort of comforting I would personally put emphasis on the "sort of" clause. If bigcos with tons of resources can't align incentives to build solid software instead of deliver new features, who among us has a chance?

[deleted]

Re: Summary of the Amazon Kinesis Event in the Northern Virginia (US-East-1) Region

#122

Post by joneholland »

foota wrote: post #111
joneholland wrote: post #86

Earlier quoted context omitted.

My understanding is that every front end server has at least one connection (on a dedicated thread) to every other front end server. Assuming they have say, 5000 front end instances, thats 5000 file descriptors being used just for this, before you are even talking about whatever threads the application needs. It’s not surprising that they bumped into ulimits, though as part of OS provisioning, you typically have thos…

The tcp connections are probably not an issue, working in cloud it's never something I've seen worried about, so maybe the architecture doesn't have that limitation?

Any time you traverse a firewall or a NAT you’ll run into issues if you have a massive amount of open sessions and you are dealing with a stateful FW.

Just because you haven’t encountered it doesn’t mean it’s not there, it’s probably just properly tuned and balanced for the load.

Re: Summary of the Amazon Kinesis Event in the Northern Virginia (US-East-1) Region

#123

Post by zxcvbn4038 »

They didn’t really discuss their remediation plans but maybe having one fleet of servers for everything isn’t the best setup. I’d love to know which OS setting they ran into. In their defense this is exactly the sort of change that never shows up in testing because the dev and qa environments are always smaller then production.

I’m wondering how many people Amazon fired over this incident - that seems to be their goto answer to everything.

Re: Summary of the Amazon Kinesis Event in the Northern Virginia (US-East-1) Region

#124

Post by schoolornot »

ayberk wrote: post #107
edoceo wrote: post #47

Earlier quoted context omitted.

Oof. My little company is refactoring some five year old architecture design choices. Ugly. Process isn't visible outside the refactor and the work is tedious. Can't imagine what a service refactor is like at A. I bet it sucks

Disclaimer: I work for AWS, but I have no ties to Kinesis. Opinions are my own. AWS has more than enough learnings to avoid these "events". The problem is the whole culture is focused on delivering new stuff instead of preventing problems and improving existing systems. Some folks made these decisions with best intentions. I'm pretty sure they all got promoted and then moved on. Now, people who inherited these system…

Another reason to have technical PMs & managers... because even if they don't have meticulous understandings of underlying systems, they can make cases for additional headcount/funding and recognize efforts that will affect tomorrow's bottom lines/issues.

Re: Summary of the Amazon Kinesis Event in the Northern Virginia (US-East-1) Region

#125

Post by reducesuffering »

nixass wrote: post #100
marcinzm wrote: post #29

Earlier quoted context omitted.

I don't think it's about growing fast so much as, from those I talked to, Amazon now has a fairly bad reputation in the tech community. You only go to work there if you don't have a better option (Google, Facebook, etc) or have some specialty skill they're willing to pay for. Pay is below other FAANG companies and the work culture isn't great (toxic even some would say). edit: They also had the most disorganized and…

Very anecdotal

Here's some anecdata that matches up with GP[0]. Note how Amazon is substantially worse-rated than most other companies.

[0]https://www.teamblind.com/company/Amazon/

Disclaimer: Am Amazon employee

Re: Summary of the Amazon Kinesis Event in the Northern Virginia (US-East-1) Region

#126

Post by reducesuffering »

steelframe wrote: post #83

> Cellularization is an approach we use to isolate the effects of failure within a service, and to keep the components of the service (in this case, the shard-map cache) operating within a previously tested and operated range. This had been under way for the front-end fleet in Kinesis, but unfortunately the work is significant and had not yet been completed. Translation: The eng team knew that they had accumulated te…

Ow, this was traumatizing to relive.

Yes, this is exactly how product development works at many (if not most) places within Amazon for engineers. It can be this toxic.

Disclaimer: Amazon engineer

Re: Summary of the Amazon Kinesis Event in the Northern Virginia (US-East-1) Region

#127

Post by pentlander »

Twirrim wrote: post #112
pentlander wrote: post #77

Earlier quoted context omitted.

As another ex-Glacier dev, I disagree with it not being "all that hard" but I agree with everything else. Now I'm curious who this is :)

The English guy, if that's enough of a clue. It has been about 4 1/2 years now since I left Glacier, so there's every chance our paths never overlapped.

Is it PG? If so, I was an intern at the time and you were gone once I joined full time.

Re: Summary of the Amazon Kinesis Event in the Northern Virginia (US-East-1) Region

#128

Post by otterley »

roman_sf wrote: post #116
otterley wrote: post #96

Earlier quoted context omitted.

(Disclaimer: I work for AWS but opinions are my own. I also do not work with the Kinesis team.) Nearly all AWS services are regional in scope, and for many (if not most) services, they are scaled at a cellular level within a region. Accounts are assigned to specific cells within that region. There are very, very few services that are global in scope, and it is strongly discouraged to create cross-regional dependencie…

What do you mean by cross-regional dependencies? Isn't running in multi-region setup is by itself adding dependency? Speaking about multi-region services. What do you think about Google now offering all three major building pieces as multi-regional? They have muti-regional buckets, LB with single anycast IP, document db (firebase). Pubsub can route automatically to nearest region. Nothing like this is available in am…

If your workload can run in region B even if there is a serious failure of a service in region A, in which your workload normally runs, then no, you have not created a cross-regional dependency.

When I talk about cross regional dependency, I talk about an architectural decision that can lead to a cascading failure in region B, which is healthy by all accounts, when there is a failure in region A.

AWS has services that allow for regional replication and failover. DynamoDB, RDS, and S3 all offer cross region replication. And Global Accelerator provides an anycast IP that can front regional services and fail over in the event of an incident.

Re: Summary of the Amazon Kinesis Event in the Northern Virginia (US-East-1) Region

#129

Post by Twirrim »

pentlander wrote: post #127
Twirrim wrote: post #112

Earlier quoted context omitted.

The English guy, if that's enough of a clue. It has been about 4 1/2 years now since I left Glacier, so there's every chance our paths never overlapped.

Is it PG? If so, I was an intern at the time and you were gone once I joined full time.

That's me :)
Post reply on HN