Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How Amazon uses chaos engineering to handle 80k requests per second

community.aws

Post reply on HN
Print view View original

51–60 of 119 posts

Re: How Amazon uses chaos engineering to handle 80k requests per second

#51

Post by ndriscoll »

andrewmcwatters wrote: post #8
truetraveller wrote: post #3

Surprised that 80K/second is called "massive" for Amazon.com's main search feature.

It dawned on me that in web software, people talk about req/s from two entirely different perspectives and it's borderline fraud: req/s from localhost to localhost, and req/s from the Internet to any user. The latter is actually interesting. People saying you can get 10k req/s from Node.js is stupid. You're not actually getting that on say, a single low-end instance over the Internet, which is what most developers ar…

Where are you saying the difference would exist? I haven't seen local network tests be worse than localhost (usually it's better since the client uses a lot of CPU itself). Why would Internet latency matter? TCP ACKs should be done by the loadbalancing appliance, so they'll be low-latency for the application. TLS handshakes should also be offloaded to the appliance.

From what I've measured, code I've written performs around the same in production as it ran locally given similar hardware. If you're deploying to a VM with 3000 IOPS and 1/2 a CPU core, obviously it's going to run like garbage. If you wouldn't run your business on a raspberry pi 3, you probably shouldn't be running it on an AWS xlarge instance either.

Re: How Amazon uses chaos engineering to handle 80k requests per second

#52

Post by andrewmcwatters »

superfrank wrote: post #43
andrewmcwatters wrote: post #13

I would love to know what software stack, hardware, and uplink connections in total they utilize to accomplish a real-world 80k request per second throughput. How many instances do you guys think Amazon runs for its primary e-commerce front-end stack? In total, and per region? Assuming they have a multi-region rollout. If it's the real-deal, and not like people saying "Bun.js can serve 65k req/s+ ( cough cough to loc…

I used to work on a system that did about 55k/sec at peak. The service was internal it was only handling grpc calls which were coming from inside our VPC and it was written in Go. It's main job was was reading and writing to an SQL db that was sharded across 3 or 4 of the biggest instances AWS offered at the time (2017ish). Everything was Dockerized and I think we were using Docker Swarm for container orchestration.…

Sounds about in the ballpark I would expect, though!

Re: How Amazon uses chaos engineering to handle 80k requests per second

#53

Post by Karrot_Kream »

andrewmcwatters wrote: post #13

I would love to know what software stack, hardware, and uplink connections in total they utilize to accomplish a real-world 80k request per second throughput. How many instances do you guys think Amazon runs for its primary e-commerce front-end stack? In total, and per region? Assuming they have a multi-region rollout. If it's the real-deal, and not like people saying "Bun.js can serve 65k req/s+ ( cough cough to loc…

Having led API teams at a big tech where we handled similar (slightly lower) request numbers at the edge, I can tell you that the entire stack was engineered to be defensive and handle a certain amount of load. As other commenters have said, fan-out means that 80k qps at the edge means you're getting probably 10-100x that on some of the most heavily hit backend systems. A lot of the work we did was very aggressive caching and sharding. Autoscaling to handle load spikes.

Observability was our secret sauce. We would monitor everything. Our caches, NICs, our load balancers, etc. Cache hotspotting and DB problems were the problems that kept us up at night, though my teams didn't deal with much stateful data.

Re: How Amazon uses chaos engineering to handle 80k requests per second

#55

Post by winrid »

c7DJTLrn wrote: post #47
bananapub wrote: post #45

why would they need to do anything - including blogging - to handle 80kqps, that's ~one machine nowadays

Your average enterprise Java backend cannot handle that.

on what, a raspberry pi? with what db? with what indexes? with what threading strategy? with what...

Re: How Amazon uses chaos engineering to handle 80k requests per second

#56

Post by 0cf8612b2e1e »

beebmam wrote: post #12
latchkey wrote: post #2

I feel like Amazon search is one of the worst products I've ever used. It is a clusterf/ck of paid advertisements and obviously gamed results. I don't care how many requests/sec you get. If the results are horrible, what does it matter?

I'm sick of it being impossible to identify cheaply made products from high quality, durable products on Amazon. The rating system is flat out broken and there's an entire industry built around gaming those ratings. I'm at the point that I rarely ever buy products on Amazon anymore. It's a total disgrace. On an ethical level, I wish I had the ability to say "I only want to be presented with results that weren't made…

Is that Amazon’s fault? So many once reputable brands have been MBAed to death and are now indistinguishable from the bottom tier garbage. It is near impossible to do real product research on anything, anywhere.

Re: How Amazon uses chaos engineering to handle 80k requests per second

#57

Post by [deleted] »

andrewmcwatters wrote: post #13

I would love to know what software stack, hardware, and uplink connections in total they utilize to accomplish a real-world 80k request per second throughput. How many instances do you guys think Amazon runs for its primary e-commerce front-end stack? In total, and per region? Assuming they have a multi-region rollout. If it's the real-deal, and not like people saying "Bun.js can serve 65k req/s+ ( cough cough to loc…

[deleted]

Re: How Amazon uses chaos engineering to handle 80k requests per second

#58

Post by 1vuio0pswjnm7 »

_a_a_a_ wrote: post #31
aeturnum wrote: post #17

I'm pretty unclear on the "how" here - but from what I can understand in the article the search resilience team injected properly tagged synthetic traffic into their system to do testing? That does seem like the kind of practice that could be part of healthy holistic approach - but the article elides a ton of details. I suppose the idea is that it promotes AWS services (with the idea of suggesting that this kind of r…

> I suppose the idea is that it promotes AWS services an advert in disguise then?

Recruiting ad

Re: How Amazon uses chaos engineering to handle 80k requests per second

#59

Post by ilyt »

c7DJTLrn wrote: post #47
bananapub wrote: post #45

why would they need to do anything - including blogging - to handle 80kqps, that's ~one machine nowadays

Your average enterprise Java backend cannot handle that.

Yeah you're 10 years out of date on that too

Re: How Amazon uses chaos engineering to handle 80k requests per second

#60

Post by ilyt »

andrewmcwatters wrote: post #13

I would love to know what software stack, hardware, and uplink connections in total they utilize to accomplish a real-world 80k request per second throughput. How many instances do you guys think Amazon runs for its primary e-commerce front-end stack? In total, and per region? Assuming they have a multi-region rollout. If it's the real-deal, and not like people saying "Bun.js can serve 65k req/s+ ( cough cough to loc…

Well, to get actual impact you'd need infrastructure-wide tracing and that's hard.

Like, you could hit a cache and serve 95% of page from it. Or hit some long path that will burn half a second on 20 servers in the backend to serve some big query

But yeah title is useless

Post reply on HN