Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How Amazon uses chaos engineering to handle 80k requests per second

community.aws

Post reply on HN
Print view View original

31–40 of 119 posts

Re: How Amazon uses chaos engineering to handle 80k requests per second

#31

Post by _a_a_a_ »

aeturnum wrote: post #17

I'm pretty unclear on the "how" here - but from what I can understand in the article the search resilience team injected properly tagged synthetic traffic into their system to do testing? That does seem like the kind of practice that could be part of healthy holistic approach - but the article elides a ton of details. I suppose the idea is that it promotes AWS services (with the idea of suggesting that this kind of r…

> I suppose the idea is that it promotes AWS services

an advert in disguise then?

Re: How Amazon uses chaos engineering to handle 80k requests per second

#32

Post by tpmx »

srcreigh wrote: post #14
truetraveller wrote: post #3

Surprised that 80K/second is called "massive" for Amazon.com's main search feature.

Google search is only 99k qps

Huh.

About a decade ago Opera Mini did 150k transcoded full pageloads/s (times about 30 inlines per pageload that was the average back then, so about 4.5 million requested/loaded/processed/compressed HTTP resources/s).

(All of the public Google Search numbers I've seen have seemed one or two orders of magnitudes too small. Or maybe most people don't use their search engine/browser as much as I do, so my perspective is skewed...)

Re: How Amazon uses chaos engineering to handle 80k requests per second

#34

Post by baz00 »

beebmam wrote: post #12
latchkey wrote: post #2

I feel like Amazon search is one of the worst products I've ever used. It is a clusterf/ck of paid advertisements and obviously gamed results. I don't care how many requests/sec you get. If the results are horrible, what does it matter?

I'm sick of it being impossible to identify cheaply made products from high quality, durable products on Amazon. The rating system is flat out broken and there's an entire industry built around gaming those ratings. I'm at the point that I rarely ever buy products on Amazon anymore. It's a total disgrace. On an ethical level, I wish I had the ability to say "I only want to be presented with results that weren't made…

The trick is to find your products somewhere else and then look them up on Amazon for a price comparison.

Re: How Amazon uses chaos engineering to handle 80k requests per second

#35

Post by catchnear4321 »

_a_a_a_ wrote: post #31
aeturnum wrote: post #17

I'm pretty unclear on the "how" here - but from what I can understand in the article the search resilience team injected properly tagged synthetic traffic into their system to do testing? That does seem like the kind of practice that could be part of healthy holistic approach - but the article elides a ton of details. I suppose the idea is that it promotes AWS services (with the idea of suggesting that this kind of r…

> I suppose the idea is that it promotes AWS services an advert in disguise then?

anything that uses the template “How X did Y to improve Z by METRIC” and also hosted on company x domain is an ad.

for that matter, any blog post by a company is an ad. maybe not to sell product, but to at least build exposure and familiarity with the brand.

Re: How Amazon uses chaos engineering to handle 80k requests per second

#36

Post by [deleted] »

adamckay wrote: post #26
andrewmcwatters wrote: post #8

Earlier quoted context omitted.

It dawned on me that in web software, people talk about req/s from two entirely different perspectives and it's borderline fraud: req/s from localhost to localhost, and req/s from the Internet to any user. The latter is actually interesting. People saying you can get 10k req/s from Node.js is stupid. You're not actually getting that on say, a single low-end instance over the Internet, which is what most developers ar…

> People saying you can get 10k req/s from Node.js is stupid. No, it's not, for exactly the reason you state: > You're not actually getting that on say, a single low-end instance over the Internet Some languages are, of course, more efficient, but it doesn't matter - you can get very good performance out of any language/runtime - it's all about your architecture and infrastructure.

[deleted]

Re: How Amazon uses chaos engineering to handle 80k requests per second

#37

Post by tpmx »

nextworddev wrote: post #20
srcreigh wrote: post #14

Earlier quoted context omitted.

Google search is only 99k qps

I’m sure there’s massive variance and seasonality around that number

From my experience with that scale of traffic (with Opera Mini at the time about 250M MAUs and 150k full pageloads/s):

There is surprisingly little seasonal variance. You have your weekly/daily traffic rhythms based on when your users are awake/active based on their geographical distribution and that's mostly it.

"World events" also have very little impact - they tend to barely make a dent in that massive background noise.

Before we had large volumes of traffic I thought we'd be seeing all sorts of unusual peaks, after a few years I realized growth at scale tends to become boring (but in a good way).

Re: How Amazon uses chaos engineering to handle 80k requests per second

#38

Post by tacozilla »

andrewmcwatters wrote: post #13

I would love to know what software stack, hardware, and uplink connections in total they utilize to accomplish a real-world 80k request per second throughput. How many instances do you guys think Amazon runs for its primary e-commerce front-end stack? In total, and per region? Assuming they have a multi-region rollout. If it's the real-deal, and not like people saying "Bun.js can serve 65k req/s+ ( cough cough to loc…

Amazon doesn't really have a "primary e-commerce front-end stack" in any concrete sense. They have hundreds/thousands of teams that deploy bits and pieces to a massive pipeline that ultimately makes up what you see on Amazon.com, but each team can have their own infrastructure backing things. Some teams might run everything off a dozen low-end EC2 instances while another sibling team has 3k+ instances; it's really all over the place, and that's ignoring specific events like Black Friday or Prime Day, etc. where teams need to prescale things in advance.

Source: Worked at Amazon/AWS for almost a decade.

Re: How Amazon uses chaos engineering to handle 80k requests per second

#40

Post by tacozilla »

baby_souffle wrote: post #16
andrewmcwatters wrote: post #13

I would love to know what software stack, hardware, and uplink connections in total they utilize to accomplish a real-world 80k request per second throughput. How many instances do you guys think Amazon runs for its primary e-commerce front-end stack? In total, and per region? Assuming they have a multi-region rollout. If it's the real-deal, and not like people saying "Bun.js can serve 65k req/s+ ( cough cough to loc…

> I would love to know what software stack, hardware, and uplink connections in total they utilize to accomplish a real-world 80k request per second throughput. How many instances do you guys think Amazon runs for its primary e-commerce front-end stack? In total, and per region? Assuming they have a multi-region rollout. > But I never see anyone talk about real-world numbers. Just synthetic poopoo. The number probabl…

While I can understand the cynicism, the real answer is a lot closer to something much more boring, which is most people just don't care about the actual numbers, and if they were to release them, while interesting to a small few, generally no one would actually care.

There's also a common misnomer that Amazon.com is somehow just this one giant app running on a set of servers, which isn't remotely how it's actually deployed, and that's before we spend time arguing whether a team's instances even count as "primary e-commerce front-end stack" or not. :P

Post reply on HN