Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How Amazon uses chaos engineering to handle 80k requests per second

community.aws

Post reply on HN
Print view View original

21–30 of 119 posts

Re: How Amazon uses chaos engineering to handle 80k requests per second

#21

Post by andrewmcwatters »

baby_souffle wrote: post #16
andrewmcwatters wrote: post #13

I would love to know what software stack, hardware, and uplink connections in total they utilize to accomplish a real-world 80k request per second throughput. How many instances do you guys think Amazon runs for its primary e-commerce front-end stack? In total, and per region? Assuming they have a multi-region rollout. If it's the real-deal, and not like people saying "Bun.js can serve 65k req/s+ ( cough cough to loc…

> I would love to know what software stack, hardware, and uplink connections in total they utilize to accomplish a real-world 80k request per second throughput. How many instances do you guys think Amazon runs for its primary e-commerce front-end stack? In total, and per region? Assuming they have a multi-region rollout. > But I never see anyone talk about real-world numbers. Just synthetic poopoo. The number probabl…

I'm not sure it's the biggest deal in the world, plus real-world market presence data is regularly detailed in shareholder reports for various companies.

You could take low-end instance specifications and standard industry stacks and extrapolate forward how many instances they might need to maintain at a maximum, but those numbers are going to be off.

Are they running 400 low-end front-end instances across the globe? Probably, (plus the 40 or so other services they claim to need, multiplied by region count at a minimum) and that would actually be well below realistic and reasonable for a company like Amazon. You can take a bunch of regional instances that handle roughly 200 req/s and make that work.

Re: How Amazon uses chaos engineering to handle 80k requests per second

#22

Post by waynesonfire »

RockRobotRock wrote: post #18
latchkey wrote: post #2

I feel like Amazon search is one of the worst products I've ever used. It is a clusterf/ck of paid advertisements and obviously gamed results. I don't care how many requests/sec you get. If the results are horrible, what does it matter?

Working as intended.

source: https://www.wsj.com/articles/amazon-changed-search-algorithm...

Re: How Amazon uses chaos engineering to handle 80k requests per second

#23

Post by yazaddaruvala »

latchkey wrote: post #2

I feel like Amazon search is one of the worst products I've ever used. It is a clusterf/ck of paid advertisements and obviously gamed results. I don't care how many requests/sec you get. If the results are horrible, what does it matter?

Disclaimer: Used to work at Amazon.

Funny story, internally Amazon Search doesn't consider the ads products to be part of the "search results". It is tracked and accounted to Ads.

The way Ads are handled on Amazon is really poorly done. The Ads teams claim to make a lot of money (and based on the internal accounting tricks they do), and as such have been pushing Amazon's leadership to go more into Ads, even tho every person I've met that worked at Amazon also hated the prevalence of Ads.

Literally, Directors and VPs at Amazon are afraid to step on the toes of Ads' leadership team because of how well they have told the story about "Ads is excessively profitable".

Meanwhile, all of us in the thread can easily say, even if it is short term profitable, it most certainly is not long term profitable for Amazon.

Both from internally and externally it has been very disappointing to watch actually.

Re: How Amazon uses chaos engineering to handle 80k requests per second

#26

Post by adamckay »

andrewmcwatters wrote: post #8
truetraveller wrote: post #3

Surprised that 80K/second is called "massive" for Amazon.com's main search feature.

It dawned on me that in web software, people talk about req/s from two entirely different perspectives and it's borderline fraud: req/s from localhost to localhost, and req/s from the Internet to any user. The latter is actually interesting. People saying you can get 10k req/s from Node.js is stupid. You're not actually getting that on say, a single low-end instance over the Internet, which is what most developers ar…

> People saying you can get 10k req/s from Node.js is stupid.

No, it's not, for exactly the reason you state:

> You're not actually getting that on say, a single low-end instance over the Internet

Some languages are, of course, more efficient, but it doesn't matter - you can get very good performance out of any language/runtime - it's all about your architecture and infrastructure.

Re: How Amazon uses chaos engineering to handle 80k requests per second

#27

Post by acchow »

yazaddaruvala wrote: post #23
latchkey wrote: post #2

I feel like Amazon search is one of the worst products I've ever used. It is a clusterf/ck of paid advertisements and obviously gamed results. I don't care how many requests/sec you get. If the results are horrible, what does it matter?

Disclaimer: Used to work at Amazon. Funny story, internally Amazon Search doesn't consider the ads products to be part of the "search results". It is tracked and accounted to Ads. The way Ads are handled on Amazon is really poorly done. The Ads teams claim to make a lot of money (and based on the internal accounting tricks they do), and as such have been pushing Amazon's leadership to go more into Ads, even tho every…

Sure the ads aren’t great. But that’s almost barely a problem compared to the gamed reviews

Re: How Amazon uses chaos engineering to handle 80k requests per second

#28

Post by digging »

beebmam wrote: post #12
latchkey wrote: post #2

I feel like Amazon search is one of the worst products I've ever used. It is a clusterf/ck of paid advertisements and obviously gamed results. I don't care how many requests/sec you get. If the results are horrible, what does it matter?

I'm sick of it being impossible to identify cheaply made products from high quality, durable products on Amazon. The rating system is flat out broken and there's an entire industry built around gaming those ratings. I'm at the point that I rarely ever buy products on Amazon anymore. It's a total disgrace. On an ethical level, I wish I had the ability to say "I only want to be presented with results that weren't made…

These days it should be assumed that the quality of anything bought on Amazon is dogshit. Which, even though I canceled Prime years ago for ethical reasons, is honestly the biggest reason I won't even click an Amazon link.

Re: How Amazon uses chaos engineering to handle 80k requests per second

#29

Post by SteveNuts »

acchow wrote: post #27
yazaddaruvala wrote: post #23

Earlier quoted context omitted.

Disclaimer: Used to work at Amazon. Funny story, internally Amazon Search doesn't consider the ads products to be part of the "search results". It is tracked and accounted to Ads. The way Ads are handled on Amazon is really poorly done. The Ads teams claim to make a lot of money (and based on the internal accounting tricks they do), and as such have been pushing Amazon's leadership to go more into Ads, even tho every…

Sure the ads aren’t great. But that’s almost barely a problem compared to the gamed reviews

> But that’s almost barely a problem compared to the gamed reviews

Which pales in comparison to the problem of counterfeit goods, IMO.

I can at least somewhat comb through the reviews to look for outliers of well written reviews. Getting something that's obviously a fake (has happened to me multiple times) is completely unacceptable.

Newegg has this issue too, I got a knock-off Intel CPU there once, I was furious.

Re: How Amazon uses chaos engineering to handle 80k requests per second

#30

Post by malfist »

acchow wrote: post #27
yazaddaruvala wrote: post #23

Earlier quoted context omitted.

Disclaimer: Used to work at Amazon. Funny story, internally Amazon Search doesn't consider the ads products to be part of the "search results". It is tracked and accounted to Ads. The way Ads are handled on Amazon is really poorly done. The Ads teams claim to make a lot of money (and based on the internal accounting tricks they do), and as such have been pushing Amazon's leadership to go more into Ads, even tho every…

Sure the ads aren’t great. But that’s almost barely a problem compared to the gamed reviews

You can complain about two things you know. Just because search is bad doesn't mean reviews can't be bad too. Or
Post reply on HN