Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How Amazon uses chaos engineering to handle 80k requests per second

community.aws

Post reply on HN
Print view View original

111–119 of 119 posts

Re: How Amazon uses chaos engineering to handle 80k requests per second

#111

Post by NBJack »

dastbe wrote: post #103
NBJack wrote: post #77

Earlier quoted context omitted.

Back in The Day, rumor had it the detail page hosting/rendering would easily max out a single machine after only a handful of queries per second. I regrettably can't verify the truth of that, or if it still holds, but there is a LOT going on with a given /dp . "How many of those requests translate to actual purchases?" is the next question.

i do recall the simplestack version of dpx had some obscene issues with garbage generation, followed by some obscene issues with throughput due to aggressive locking (to avoid the obscene amount of garbage generated).

I am not surprised. That platform was a neat concept, but wow, the nesting and resource consumption was atrocious. There was potential there, yet the implementation in Java made IMHO some fundamentaly flawed assertions about small objects. I unfortunately know of a great director who had to find new opportunities outside of the company over it.

Re: How Amazon uses chaos engineering to handle 80k requests per second

#112

Post by hughesjj »

Weryj wrote: post #109

Can confirm, working there was Chaos, not sure that's what they meant by chaos engineering though.

It's where they delete entire teams and remove funding for critical infrastructure and see how well the system self-repairs

It's just in this case the system is the company itself instead of just a suite of software services

Re: How Amazon uses chaos engineering to handle 80k requests per second

#113

Post by hughesjj »

dmoy wrote: post #63
Tomis02 wrote: post #61

Earlier quoted context omitted.

> Not all req/s are made the same. > Amazon search is made of 100s of services, and Amazon's search page loads 20 products per page, that means 80k search req/s translates to 1.6 MM product API req/s for example. That's their problem, no? Nobody's forcing them to have an architecture where a request propagates to hundreds of services.

> That's their problem, no? Nobody's forcing them to have an architecture where a request propagates to hundreds of services. Well, Jeff Bezos circa 2002 or so did: https://news.ycombinator.com/item?id=3102800 It's hard to remove that ethos now

And thank God he did, amazon would have direct without the SOA enforcement.

Scale issues are emergent issues. It's a phase transition.

Re: How Amazon uses chaos engineering to handle 80k requests per second

#114

Post by hughesjj »

tacozilla wrote: post #40
baby_souffle wrote: post #16

Earlier quoted context omitted.

> I would love to know what software stack, hardware, and uplink connections in total they utilize to accomplish a real-world 80k request per second throughput. How many instances do you guys think Amazon runs for its primary e-commerce front-end stack? In total, and per region? Assuming they have a multi-region rollout. > But I never see anyone talk about real-world numbers. Just synthetic poopoo. The number probabl…

While I can understand the cynicism, the real answer is a lot closer to something much more boring, which is most people just don't care about the actual numbers, and if they were to release them, while interesting to a small few, generally no one would actually care. There's also a common misnomer that Amazon.com is somehow just this one giant app running on a set of servers, which isn't remotely how it's actually d…

Also their "stack" includes various degrees of AWS (maws, naws, and I'm sure a bunch of snowflake situations innumerable here, I mean do you count corpinfra? Controls?)

Not to mention all the data center logistics

Re: How Amazon uses chaos engineering to handle 80k requests per second

#115

Post by hughesjj »

ilyt wrote: post #60
andrewmcwatters wrote: post #13

I would love to know what software stack, hardware, and uplink connections in total they utilize to accomplish a real-world 80k request per second throughput. How many instances do you guys think Amazon runs for its primary e-commerce front-end stack? In total, and per region? Assuming they have a multi-region rollout. If it's the real-deal, and not like people saying "Bun.js can serve 65k req/s+ ( cough cough to loc…

Well, to get actual impact you'd need infrastructure-wide tracing and that's hard. Like, you could hit a cache and serve 95% of page from it. Or hit some long path that will burn half a second on 20 servers in the backend to serve some big query But yeah title is useless

I mean, coralmetrics+pmet has been doing that infrastructure wide tracing for decades (albeit being slowly replaced in spots now).

Back in 2014 they were still sharing a few years old (at that time) detail-level service call graph that had so many nodes and lines it looked like string art.

Re: How Amazon uses chaos engineering to handle 80k requests per second

#116

Post by hughesjj »

tree666 wrote: post #67

Your article shows you put in a lot of work, and was written well. But... A few thoughts... Service Owners usually tend to design and build their systems in a way which allows them to understand and know how the system will behave under the worst conditions. It is not difficult to test these conditions either. A bash script utilizing curl will suffice. If you have to use 'Chaos Engineering' to experiment your way int…

> Service Owners usually tend to design and build their systems in a way which allows them to understand and know how the system will behave under the worst conditions.

With you so far

> A bash script utilizing curl will suffice.

Lol hell no. Yes, AWS/amazon does require a "GameDay" before launching a service which will execute an mcm (managed change management) that's basically a runbook of (way more in depth and comprehensive way to test your service than a single bash script with curl), but chaos engineering is a great additive, additional verification mechanism that really helps with service outages.

How are you going to test a thundering herd with a bash script executing curl?

How many machines are you running this simple bash+curl script on anyway? Using a single node to generate requests isn't going to do much in testing a service's reliability.

Also, they literally did invent a wheel

https://aws.amazon.com/blogs/opensource/the-wheel/

Re: How Amazon uses chaos engineering to handle 80k requests per second

#117

Post by hughesjj »

bluedevilzn wrote: post #100

When I was at Amazon, I thought this was an insane scale. Then, I joined Google to learn how they serve couple orders of magnitudes higher.

this is low acale for any given public api at amazon. You should see how much kinesis/ddb/iam/sqs/cloudwatch handles (in total per region)

Re: How Amazon uses chaos engineering to handle 80k requests per second

#118

Post by baja_blast »

moribvndvs wrote: post #99
offmycloud wrote: post #95

Amazon product search results are the chaos.

1000 sponsored links that are exactly not what you searched and filtered for.

And a hundred different identical products at vastly different prices from brand new sellers like `z-qq-yadonk-8771` that some how has 4.7k reviews at 4.5 stars.

Re: How Amazon uses chaos engineering to handle 80k requests per second

#119

Post by scrum-treats »

AtlasBarfed wrote: post #91

Amazon's ultimate chaos engineering is a working culture that turns over its entire staff every two years and stays afloat.

Agreed. The only thing chaotic is the turnover and the level of managerial incompetence.
Post reply on HN