Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Concurrency Limits by Netflix

github.com

Post reply on HN
Print view View original

21–30 of 30 posts

Re: Concurrency Limits by Netflix

#21

Post by nvarsj »

Anderkent wrote: post #17

This is a pretty cool design if your requests to a given endpoint are supposed to all take about the same time. It's not easy to see how you'd adjust it for things with more variance; perhaps rather than using the fastest seen mtt, you could look at your p99 over the last N minutes, and see if it's been changing?

I think you bring up the main problem w/ using tcp vegas. It's not clear to me this will work with heterogenous requests. If the typical request time distribution is long tailed, it might never increase the window size.

Re: Concurrency Limits by Netflix

#22

Post by ad-hominem »

nvarsj wrote: post #20
bvod wrote: post #18

Can anyone explain how they decide which requests to reject? The blog post just mentions that excess RPS gets rejected, but couldn't rejecting arbitrary requests cause other problems?

My guess is they use a 0 / small queue in front of the request pool. If queue is full (indicating the server is at its concurrency limit), it returns a 429 (which is sort of weird - return a 503 instead). I don't think that is part of the library though - the library just provides the low level bricks.

The 429 is to inform clients they should do something else ("it's your fault"). A 503 would indicate it was the server's problem and maybe they should retry. A server at capacity does not want the client to retry directly to it, it wants the client to look elsewhere (even if that means retrying against the load balancer). It's a small semantic difference, but since netflix controls all the clients and servers, it's kind of a preference thing.

Re: Concurrency Limits by Netflix

#23

Post by otterley »

tyrankh wrote: post #9
lkrubner wrote: post #7

I like the sounds of this: " Executor -- The BlockingAdaptiveExecutor adapts the size of an internal thread pool to match the concurrency limit based on measured latencies of Runnable commands and will block when the limit has been reached. " I'm often surprised this kind of auto-scaling thread pool is not a more common thing in Java land.

Agreed - not trying to trivialize the work here, but this thing seems solved with tools at hand. Perhaps I miss some of the subtlety in their usecase, though.

Which tools are you referring to, exactly? Some citations would be helpful.

Re: Concurrency Limits by Netflix

#24

Post by otterley »

baybal2 wrote: post #14

PROTIP: Managing congestion control on application level is not a good business. A better idea is to leave it to the edge and CDN, while the app level uses computationally cheap optimistic algos since the com in between the app and your edge will go over your own high quality infrastructure. This adds flexibility to allow for use of multiple algorithms in different load balancing regions (mobile as a lossy fabric is…

Can you refer us to any papers or analyses that support your claim?

Re: Concurrency Limits by Netflix

#25

Post by elandau »

nvarsj wrote: post #21
Anderkent wrote: post #17

This is a pretty cool design if your requests to a given endpoint are supposed to all take about the same time. It's not easy to see how you'd adjust it for things with more variance; perhaps rather than using the fastest seen mtt, you could look at your p99 over the last N minutes, and see if it's been changing?

I think you bring up the main problem w/ using tcp vegas. It's not clear to me this will work with heterogenous requests. If the typical request time distribution is long tailed, it might never increase the window size.

Even with heterogenous workload there normally is a uniform distribution of request types. Instead of generating complex statistics for average latency or tail latencies, especially for multimodal distributions, we just look at the minimum latencies as a proxy to identify queuing. So, when there is any queuing for whatever reason (increased RPS or latency in a dependent service) all latency measurements will show an increase, especially the minimum.

Re: Concurrency Limits by Netflix

#26

Post by elandau »

bvod wrote: post #18

Can anyone explain how they decide which requests to reject? The blog post just mentions that excess RPS gets rejected, but couldn't rejecting arbitrary requests cause other problems?

Requests are rejected essentially when an atomic counter of inflight requests hits the limit. It's important to note that the library doesn't actually keep any kind of queue of requests. That's really not necessary because every system already has a ton of queues in the form of socket buffers, executor queues, etc...

Yes, the basic implementation does reject arbitrary requests. We do have a partitioned limit strategy (currently in the experimental state, which is why it wasn't brought up in the techblog). The partitioned limiters lets you guarantee a portion of the limit to certain types of requests. For example, let's say you want to give priority to live vs batch traffic. Live gets 90% of the limit, batch gets 10%. If live requests only account for 50% of the limit then batch can use up to the remaining 50%. But if all of a sudden there's sustained increase in live traffic you're guaranteed that live requests will only be rejected once the exceed 90% of the limit.

Re: Concurrency Limits by Netflix

#27

Post by Bombthecat »

Anderkent wrote: post #17

This is a pretty cool design if your requests to a given endpoint are supposed to all take about the same time. It's not easy to see how you'd adjust it for things with more variance; perhaps rather than using the fastest seen mtt, you could look at your p99 over the last N minutes, and see if it's been changing?

Jeah, I'm also thinking about this problem. Thought this could help...

Most of the problematic backedends are not those with response time of 20ms. On almost every request. The backedends with problems are those which could reply in 10ms or 2 minutes ...

Re: Concurrency Limits by Netflix

#28

Post by fwef »

Would this work when connecting multiple threadpools (java executors)? Imagine I have a microservice where first threadpool downloads large files to disk (IO bound) and another threadpool that processes the downloaded data (CPU bound). Those two threadpools communicate with a bounded queue. Will using the concurrency-limit Executor allow me to get the best throughput in this scenario?

Re: Concurrency Limits by Netflix

#29

Post by nvarsj »

elandau wrote: post #25
nvarsj wrote: post #21

Earlier quoted context omitted.

I think you bring up the main problem w/ using tcp vegas. It's not clear to me this will work with heterogenous requests. If the typical request time distribution is long tailed, it might never increase the window size.

Even with heterogenous workload there normally is a uniform distribution of request types. Instead of generating complex statistics for average latency or tail latencies, especially for multimodal distributions, we just look at the minimum latencies as a proxy to identify queuing. So, when there is any queuing for whatever reason (increased RPS or latency in a dependent service) all latency measurements will show an…

"uniform distribution of request types" - okay, it makes sense in that context. Although if that assumption breaks down, your thread limits may become under or over provisioned.

I'm wondering though - how do you pick the right alpha and beta values? It seems like you need to do testing/validation to ensure you use the right values, right?

Sorry if I'm sounding critical by the way. I think this is a really cool project - thanks for open sourcing it!

Re: Concurrency Limits by Netflix

#30

Post by baybal2 »

otterley wrote: post #24
baybal2 wrote: post #14

PROTIP: Managing congestion control on application level is not a good business. A better idea is to leave it to the edge and CDN, while the app level uses computationally cheap optimistic algos since the com in between the app and your edge will go over your own high quality infrastructure. This adds flexibility to allow for use of multiple algorithms in different load balancing regions (mobile as a lossy fabric is…

Can you refer us to any papers or analyses that support your claim?

No
Post reply on HN