Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How Amazon uses chaos engineering to handle 80k requests per second

community.aws

Post reply on HN
Print view View original

101–110 of 119 posts

Re: How Amazon uses chaos engineering to handle 80k requests per second

#101

Post by slipshady »

NBJack wrote: post #77
bdamm wrote: post #46

Earlier quoted context omitted.

Can anyone calculate the dollars-per-request revenue and profit? Would be interesting to see how much it costs Amazon to make money and make some connection to the request rate.

Back in The Day, rumor had it the detail page hosting/rendering would easily max out a single machine after only a handful of queries per second. I regrettably can't verify the truth of that, or if it still holds, but there is a LOT going on with a given /dp . "How many of those requests translate to actual purchases?" is the next question.

My professor at Cornell whose lab Werner Vogel (Amazon's current CTO) worked loved telling this story. Maybe there's some veracity to it.

Re: How Amazon uses chaos engineering to handle 80k requests per second

#102

Post by slipshady »

bluedevilzn wrote: post #100

When I was at Amazon, I thought this was an insane scale. Then, I joined Google to learn how they serve couple orders of magnitudes higher.

DynamoDB served upwards of 100MM tps during Prime Day '23 - https://aws.amazon.com/blogs/aws/prime-day-2023-powered-by-a...

Re: How Amazon uses chaos engineering to handle 80k requests per second

#103

Post by dastbe »

NBJack wrote: post #77
bdamm wrote: post #46

Earlier quoted context omitted.

Can anyone calculate the dollars-per-request revenue and profit? Would be interesting to see how much it costs Amazon to make money and make some connection to the request rate.

Back in The Day, rumor had it the detail page hosting/rendering would easily max out a single machine after only a handful of queries per second. I regrettably can't verify the truth of that, or if it still holds, but there is a LOT going on with a given /dp . "How many of those requests translate to actual purchases?" is the next question.

i do recall the simplestack version of dpx had some obscene issues with garbage generation, followed by some obscene issues with throughput due to aggressive locking (to avoid the obscene amount of garbage generated).

Re: How Amazon uses chaos engineering to handle 80k requests per second

#106

Post by yathaid »

Tomis02 wrote: post #61
yazaddaruvala wrote: post #44

Earlier quoted context omitted.

> If it's the real-deal, and not like people saying "Bun.js can serve 65k req/s+ (cough cough to localhost,)" that's impressive. Not all req/s are made the same. Amazon search is made of 100s of services, and Amazon's search page loads 20 products per page, that means 80k search req/s translates to 1.6 MM product API req/s for example. FWIW a search request at Amazon hits roughly 100 unique search clusters (think of…

> Not all req/s are made the same. > Amazon search is made of 100s of services, and Amazon's search page loads 20 products per page, that means 80k search req/s translates to 1.6 MM product API req/s for example. That's their problem, no? Nobody's forcing them to have an architecture where a request propagates to hundreds of services.

My favorite kind of HN answer. 100% snark, 100% certain, 100% wrong.

So, instead of that, what would you have? A single, binary that makes multiple DB calls? Hmmm, let's see, some people have tried that and written extensively about the problems they faced doing that. Wait, one of them is actually a small e-commerce firm named after a larger river. Wonder what the issues were ...

Re: How Amazon uses chaos engineering to handle 80k requests per second

#107

Post by yathaid »

bananapub wrote: post #45

why would they need to do anything - including blogging - to handle 80kqps, that's ~one machine nowadays

I am sorry, do you think building a reverse index for a billion products fits in one machine? Is that seriously the amount of thought you put into this comment?

Re: How Amazon uses chaos engineering to handle 80k requests per second

#108

Post by yathaid »

tree666 wrote: post #67

Your article shows you put in a lot of work, and was written well. But... A few thoughts... Service Owners usually tend to design and build their systems in a way which allows them to understand and know how the system will behave under the worst conditions. It is not difficult to test these conditions either. A bash script utilizing curl will suffice. If you have to use 'Chaos Engineering' to experiment your way int…

Is your bash script open or closed? [1]

How does a bash script simulate network failures, connectivity drops or a gray-failure in one of your dependencies? [2]

A lot of thought has been put into this domain. Dismissing that without understanding any of the complexities is just showing your ignorance.

[1] - https://brooker.co.za/blog/2023/05/10/open-closed.html

[2] - https://docs.aws.amazon.com/fis/latest/userguide/fis-actions...

Re: How Amazon uses chaos engineering to handle 80k requests per second

#110

Post by NovemberWhiskey »

wincy wrote: post #96
SteveNuts wrote: post #29

Earlier quoted context omitted.

> But that’s almost barely a problem compared to the gamed reviews Which pales in comparison to the problem of counterfeit goods, IMO. I can at least somewhat comb through the reviews to look for outliers of well written reviews. Getting something that's obviously a fake (has happened to me multiple times) is completely unacceptable. Newegg has this issue too, I got a knock-off Intel CPU there once, I was furious.

What the heck, what does a knock off Intel CPU even look like? Who is making them and more importantly, how?

It's more like: old being sold as new, engineering samples sold as production parts, relabelling lower spec parts as higher spec on the packages etc.
Post reply on HN