Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How Amazon uses chaos engineering to handle 80k requests per second

community.aws

Post reply on HN
Print view View original

61–70 of 119 posts

Re: How Amazon uses chaos engineering to handle 80k requests per second

#61

Post by Tomis02 »

yazaddaruvala wrote: post #44
andrewmcwatters wrote: post #13

I would love to know what software stack, hardware, and uplink connections in total they utilize to accomplish a real-world 80k request per second throughput. How many instances do you guys think Amazon runs for its primary e-commerce front-end stack? In total, and per region? Assuming they have a multi-region rollout. If it's the real-deal, and not like people saying "Bun.js can serve 65k req/s+ ( cough cough to loc…

> If it's the real-deal, and not like people saying "Bun.js can serve 65k req/s+ (cough cough to localhost,)" that's impressive. Not all req/s are made the same. Amazon search is made of 100s of services, and Amazon's search page loads 20 products per page, that means 80k search req/s translates to 1.6 MM product API req/s for example. FWIW a search request at Amazon hits roughly 100 unique search clusters (think of…

> Not all req/s are made the same. > Amazon search is made of 100s of services, and Amazon's search page loads 20 products per page, that means 80k search req/s translates to 1.6 MM product API req/s for example.

That's their problem, no? Nobody's forcing them to have an architecture where a request propagates to hundreds of services.

Re: How Amazon uses chaos engineering to handle 80k requests per second

#62

Post by omgJustTest »

aeturnum wrote: post #17

I'm pretty unclear on the "how" here - but from what I can understand in the article the search resilience team injected properly tagged synthetic traffic into their system to do testing? That does seem like the kind of practice that could be part of healthy holistic approach - but the article elides a ton of details. I suppose the idea is that it promotes AWS services (with the idea of suggesting that this kind of r…

Oh... "Chaos systems"=="event-driven development". For those confused about an analogy to chaos theory... as far as i can tell, there isn't one. In physics a chaos system has small perturbations that lead to an instability. I would argue this is just a large perturbation that leads to certain instabilities. I would also classify this as "network systems fault tolerance" engineering.

As a stand-alone article it is fine, and is likely to trip the more fluff than stuff alarm on many people's bs-detectors.

Re: How Amazon uses chaos engineering to handle 80k requests per second

#63

Post by dmoy »

Tomis02 wrote: post #61
yazaddaruvala wrote: post #44

Earlier quoted context omitted.

> If it's the real-deal, and not like people saying "Bun.js can serve 65k req/s+ (cough cough to localhost,)" that's impressive. Not all req/s are made the same. Amazon search is made of 100s of services, and Amazon's search page loads 20 products per page, that means 80k search req/s translates to 1.6 MM product API req/s for example. FWIW a search request at Amazon hits roughly 100 unique search clusters (think of…

> Not all req/s are made the same. > Amazon search is made of 100s of services, and Amazon's search page loads 20 products per page, that means 80k search req/s translates to 1.6 MM product API req/s for example. That's their problem, no? Nobody's forcing them to have an architecture where a request propagates to hundreds of services.

> That's their problem, no? Nobody's forcing them to have an architecture where a request propagates to hundreds of services.

Well, Jeff Bezos circa 2002 or so did:

https://news.ycombinator.com/item?id=3102800

It's hard to remove that ethos now

Re: How Amazon uses chaos engineering to handle 80k requests per second

#64

Post by NoPicklez »

amzthrow wrote: post #48
beebmam wrote: post #12

Earlier quoted context omitted.

I'm sick of it being impossible to identify cheaply made products from high quality, durable products on Amazon. The rating system is flat out broken and there's an entire industry built around gaming those ratings. I'm at the point that I rarely ever buy products on Amazon anymore. It's a total disgrace. On an ethical level, I wish I had the ability to say "I only want to be presented with results that weren't made…

I understand how you feel. It can be frustrating. Contrary to popular belief Amazon actually does put energy into making sure products are responsibly sourced. Products are de-listed if they’re found to come from unethical sources. To take that even further take a look at Climate Pledge Friendly. Those are products with (at least one) third party certification. These certifications don’t just further climate goals. S…

Buying from ethical sources is great, but that doesn't mean those products aren't crap either

Re: How Amazon uses chaos engineering to handle 80k requests per second

#66

Post by coffeebeqn »

andrewmcwatters wrote: post #8
truetraveller wrote: post #3

Surprised that 80K/second is called "massive" for Amazon.com's main search feature.

It dawned on me that in web software, people talk about req/s from two entirely different perspectives and it's borderline fraud: req/s from localhost to localhost, and req/s from the Internet to any user. The latter is actually interesting. People saying you can get 10k req/s from Node.js is stupid. You're not actually getting that on say, a single low-end instance over the Internet, which is what most developers ar…

It’s not the best metric. Just responding to 80k req/sec with static in memory content is easy nowadays. If there are some complex database queries you have to finangle 80k/sec then that’s the interesting part

Re: How Amazon uses chaos engineering to handle 80k requests per second

#67

Post by tree666 »

Your article shows you put in a lot of work, and was written well. But... A few thoughts... Service Owners usually tend to design and build their systems in a way which allows them to understand and know how the system will behave under the worst conditions. It is not difficult to test these conditions either. A bash script utilizing curl will suffice.

If you have to use 'Chaos Engineering' to experiment your way into innovation, this is a sign you built your service wrong. What will Amazon Re-Invent next!? I am guess the wheel. Well written article though.

Re: How Amazon uses chaos engineering to handle 80k requests per second

#68

Post by txcwpalpha »

Tomis02 wrote: post #61
yazaddaruvala wrote: post #44

Earlier quoted context omitted.

> If it's the real-deal, and not like people saying "Bun.js can serve 65k req/s+ (cough cough to localhost,)" that's impressive. Not all req/s are made the same. Amazon search is made of 100s of services, and Amazon's search page loads 20 products per page, that means 80k search req/s translates to 1.6 MM product API req/s for example. FWIW a search request at Amazon hits roughly 100 unique search clusters (think of…

> Not all req/s are made the same. > Amazon search is made of 100s of services, and Amazon's search page loads 20 products per page, that means 80k search req/s translates to 1.6 MM product API req/s for example. That's their problem, no? Nobody's forcing them to have an architecture where a request propagates to hundreds of services.

Why is it anyone's "problem"? Nobody said they're being forced to do it this way - just that they are. And I guarantee that there are thousands of other companies out there that have an API-fanout model as well, and might be interested in how Amazon does it.

I don't get the hostility around this article. Nobody is forcing you to read it or to do it this way. If your system is architected in a different way where you can run your whole system on a single instance, then good for you! But Amazon presumably doesn't have that luxury, and others may not either.

Re: How Amazon uses chaos engineering to handle 80k requests per second

#69

Post by nonameiguess »

ndriscoll wrote: post #51
andrewmcwatters wrote: post #8

Earlier quoted context omitted.

It dawned on me that in web software, people talk about req/s from two entirely different perspectives and it's borderline fraud: req/s from localhost to localhost, and req/s from the Internet to any user. The latter is actually interesting. People saying you can get 10k req/s from Node.js is stupid. You're not actually getting that on say, a single low-end instance over the Internet, which is what most developers ar…

Where are you saying the difference would exist? I haven't seen local network tests be worse than localhost (usually it's better since the client uses a lot of CPU itself). Why would Internet latency matter? TCP ACKs should be done by the loadbalancing appliance, so they'll be low-latency for the application. TLS handshakes should also be offloaded to the appliance. From what I've measured, code I've written performs…

These aren't requests for TCP ACKs to establish a session, nor even requests for a simple static resource. They're requests for the live status of an inventory of physical goods spread across thousands of distribution centers on six continents that are themselves gaining and losing thousands of products per second. A system that can return a reasonably accurate view of that state 80k times per second is not the same thing as a system that can send 80k http responses with "Hello ${NAME}" per second.
Post reply on HN