Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How Amazon uses chaos engineering to handle 80k requests per second

community.aws

Post reply on HN
Print view View original

71–80 of 119 posts

Re: How Amazon uses chaos engineering to handle 80k requests per second

#71

Post by ljm »

_a_a_a_ wrote: post #31
aeturnum wrote: post #17

I'm pretty unclear on the "how" here - but from what I can understand in the article the search resilience team injected properly tagged synthetic traffic into their system to do testing? That does seem like the kind of practice that could be part of healthy holistic approach - but the article elides a ton of details. I suppose the idea is that it promotes AWS services (with the idea of suggesting that this kind of r…

> I suppose the idea is that it promotes AWS services an advert in disguise then?

The solution to any problem in a cloud provider is another service the same provider offers, as far as they’re concerned.

Re: How Amazon uses chaos engineering to handle 80k requests per second

#72

Post by yazaddaruvala »

Tomis02 wrote: post #61
yazaddaruvala wrote: post #44

Earlier quoted context omitted.

> If it's the real-deal, and not like people saying "Bun.js can serve 65k req/s+ (cough cough to localhost,)" that's impressive. Not all req/s are made the same. Amazon search is made of 100s of services, and Amazon's search page loads 20 products per page, that means 80k search req/s translates to 1.6 MM product API req/s for example. FWIW a search request at Amazon hits roughly 100 unique search clusters (think of…

> Not all req/s are made the same. > Amazon search is made of 100s of services, and Amazon's search page loads 20 products per page, that means 80k search req/s translates to 1.6 MM product API req/s for example. That's their problem, no? Nobody's forcing them to have an architecture where a request propagates to hundreds of services.

> That's their problem, no? Nobody's forcing them to have an architecture where a request propagates to hundreds of services.

I mean, physics kinda is.

Just the volume of data that needs to be hosted, needs multiple nodes. ElasticSearch has some good general documentation of search engines if you want to learn more. In Amazon's case, a general query will hit a fanout of about 10,000x nodes - I wasn't even counting that fanout because it is all technically 1 service.

Meanwhile, Bun.js (no offense to Bun) is built for IO-bounded workloads and 65k req/s is great for that. However, executing the required Natural Language Processing (i.e. multiple ML models) would result in an exhausted CPU (even if Bun could distribute the compute across cores or compute the ML inference on a GPU). I'd be willing to bet even on a great CPU it gets throttled at 10 req/s (at most 100 req/s).

This really isn't some "They should have just done it all in Postgress, Bun.js can just use a nice ORM and be IO-bounded" type situation - which is a philosophy I very much agree with for 99.9999% of usecases.

Re: How Amazon uses chaos engineering to handle 80k requests per second

#73

Post by amzthrow »

NoPicklez wrote: post #64
amzthrow wrote: post #48

Earlier quoted context omitted.

I understand how you feel. It can be frustrating. Contrary to popular belief Amazon actually does put energy into making sure products are responsibly sourced. Products are de-listed if they’re found to come from unethical sources. To take that even further take a look at Climate Pledge Friendly. Those are products with (at least one) third party certification. These certifications don’t just further climate goals. S…

Buying from ethical sources is great, but that doesn't mean those products aren't crap either

Take a look at the certifications. It’s a signal.

Re: How Amazon uses chaos engineering to handle 80k requests per second

#74

Post by NBJack »

tree666 wrote: post #67

Your article shows you put in a lot of work, and was written well. But... A few thoughts... Service Owners usually tend to design and build their systems in a way which allows them to understand and know how the system will behave under the worst conditions. It is not difficult to test these conditions either. A bash script utilizing curl will suffice. If you have to use 'Chaos Engineering' to experiment your way int…

I might be misunderstanding what you consider one, but in my experience a bash script driving curl is great for some types of API or front-end load testing.

However, it won't necessarily help you know how your system will behave if S3 kicks the bucket in us-east-1 (again), your image host for that super-cool Kubernetes cluster suddenly throttles you during a critical restart, or your other service of choice went down due to an expired certificate.

If you however mean to use it to perform a denial of service on an endpoint you don't own, you're more hard-core than I thought.

Re: How Amazon uses chaos engineering to handle 80k requests per second

#75

Post by nijave »

yazaddaruvala wrote: post #23
latchkey wrote: post #2

I feel like Amazon search is one of the worst products I've ever used. It is a clusterf/ck of paid advertisements and obviously gamed results. I don't care how many requests/sec you get. If the results are horrible, what does it matter?

Disclaimer: Used to work at Amazon. Funny story, internally Amazon Search doesn't consider the ads products to be part of the "search results". It is tracked and accounted to Ads. The way Ads are handled on Amazon is really poorly done. The Ads teams claim to make a lot of money (and based on the internal accounting tricks they do), and as such have been pushing Amazon's leadership to go more into Ads, even tho every…

So... The advertising team is good at marketing?

Re: How Amazon uses chaos engineering to handle 80k requests per second

#76

Post by ndriscoll »

nonameiguess wrote: post #69
ndriscoll wrote: post #51

Earlier quoted context omitted.

Where are you saying the difference would exist? I haven't seen local network tests be worse than localhost (usually it's better since the client uses a lot of CPU itself). Why would Internet latency matter? TCP ACKs should be done by the loadbalancing appliance, so they'll be low-latency for the application. TLS handshakes should also be offloaded to the appliance. From what I've measured, code I've written performs…

These aren't requests for TCP ACKs to establish a session, nor even requests for a simple static resource. They're requests for the live status of an inventory of physical goods spread across thousands of distribution centers on six continents that are themselves gaining and losing thousands of products per second. A system that can return a reasonably accurate view of that state 80k times per second is not the same…

I'm not talking about just establishing a session. My questioning there was just why Internet vs. local would be different. On a local network, I've gotten 70k json CRUD requests out of a netty based service + postgresql with 4 cores and a single SSD.

I imagine search is more complex and expensive than CRUD, but 80k isn't something you can only do with a "hello world" tier application.

Re: How Amazon uses chaos engineering to handle 80k requests per second

#77

Post by NBJack »

bdamm wrote: post #46
hexo wrote: post #39

only 80k?

Can anyone calculate the dollars-per-request revenue and profit? Would be interesting to see how much it costs Amazon to make money and make some connection to the request rate.

Back in The Day, rumor had it the detail page hosting/rendering would easily max out a single machine after only a handful of queries per second. I regrettably can't verify the truth of that, or if it still holds, but there is a LOT going on with a given /dp . "How many of those requests translate to actual purchases?" is the next question.

Re: How Amazon uses chaos engineering to handle 80k requests per second

#78

Post by nijave »

amzthrow wrote: post #48
beebmam wrote: post #12

Earlier quoted context omitted.

I'm sick of it being impossible to identify cheaply made products from high quality, durable products on Amazon. The rating system is flat out broken and there's an entire industry built around gaming those ratings. I'm at the point that I rarely ever buy products on Amazon anymore. It's a total disgrace. On an ethical level, I wish I had the ability to say "I only want to be presented with results that weren't made…

I understand how you feel. It can be frustrating. Contrary to popular belief Amazon actually does put energy into making sure products are responsibly sourced. Products are de-listed if they’re found to come from unethical sources. To take that even further take a look at Climate Pledge Friendly. Those are products with (at least one) third party certification. These certifications don’t just further climate goals. S…

Amazon can simultaneously do due diligence on a subset of products while completely ignoring the rampant fraud and knockoffs of many other products.

Re: How Amazon uses chaos engineering to handle 80k requests per second

#79

Post by NBJack »

Tomis02 wrote: post #61
yazaddaruvala wrote: post #44

Earlier quoted context omitted.

> If it's the real-deal, and not like people saying "Bun.js can serve 65k req/s+ (cough cough to localhost,)" that's impressive. Not all req/s are made the same. Amazon search is made of 100s of services, and Amazon's search page loads 20 products per page, that means 80k search req/s translates to 1.6 MM product API req/s for example. FWIW a search request at Amazon hits roughly 100 unique search clusters (think of…

> Not all req/s are made the same. > Amazon search is made of 100s of services, and Amazon's search page loads 20 products per page, that means 80k search req/s translates to 1.6 MM product API req/s for example. That's their problem, no? Nobody's forcing them to have an architecture where a request propagates to hundreds of services.

I suspect it grew into that. Amazon has been around for almost 30 years. That's a long time to incrementally change a service ecosystem.

Re: How Amazon uses chaos engineering to handle 80k requests per second

#80

Post by nijave »

0cf8612b2e1e wrote: post #56
beebmam wrote: post #12

Earlier quoted context omitted.

I'm sick of it being impossible to identify cheaply made products from high quality, durable products on Amazon. The rating system is flat out broken and there's an entire industry built around gaming those ratings. I'm at the point that I rarely ever buy products on Amazon anymore. It's a total disgrace. On an ethical level, I wish I had the ability to say "I only want to be presented with results that weren't made…

Is that Amazon’s fault? So many once reputable brands have been MBAed to death and are now indistinguishable from the bottom tier garbage. It is near impossible to do real product research on anything, anywhere.

Surely a company with the resources of Amazon can determine fake reviews from real ones. Especially those bait-and-switch where the product is changed and half the reviews aren't even for the current listing.
Post reply on HN