I'm not grasping the concept; Do I use this in production to manage concurrency, or do I use this in 'testing' to fine tune my system?
You use this on production to allow either the server or client to self determine optimal concurrency levels.
Concurrency Limits by Netflix
11–20 of 30 posts
Re: Concurrency Limits by Netflix
#12Re: Concurrency Limits by Netflix
#13I would have thought reactive streams or back pressure would be the usual way to deal with this issue.
Is this better ?
Re: Concurrency Limits by Netflix
#14PROTIP: Managing congestion control on application level is not a good business. A better idea is to leave it to the edge and CDN, while the app level uses computationally cheap optimistic algos since the com in between the app and your edge will go over your own high quality infrastructure.
This adds flexibility to allow for use of multiple algorithms in different load balancing regions (mobile as a lossy fabric is better to stay with conventional, desktop and server clients can use a some smarter throughput maximizing algo, and in countries with high percentages of connections being laggy DSLs, you can use something else )
Re: Concurrency Limits by Netflix
#15I would have thought reactive streams or back pressure would be the usual way to deal with this issue. Is this better ?
Netflix's circuit breaking OSS project, Hystrix, is commonly seen alongside Ratpack/rxjava where both reactive streams and back pressure are in play. I don't think you're wrong, but Netflix and others using their solutions like Hystrix and Hollow, are outside of what I'd consider "usual" problems and solutions.
Re: Concurrency Limits by Netflix
#16I would have thought reactive streams or back pressure would be the usual way to deal with this issue. Is this better ?
Netflix's circuit breaking OSS project, Hystrix, is commonly seen alongside Ratpack/rxjava where both reactive streams and back pressure are in play. I don't think you're wrong, but Netflix and others using their solutions like Hystrix and Hollow, are outside of what I'd consider "usual" problems and solutions.
I don't think this library is meant to be used w/ reactive streams. It talks a lot about limiting number of concurrent threads, so it sounds more like traditional RPC with a request pool that they are trying to size to inform clients to back off (by returning 429).
Re: Concurrency Limits by Netflix
#17This is a pretty cool design if your requests to a given endpoint are supposed to all take about the same time. It's not easy to see how you'd adjust it for things with more variance; perhaps rather than using the fastest seen mtt, you could look at your p99 over the last N minutes, and see if it's been changing?
Re: Concurrency Limits by Netflix
#18Can anyone explain how they decide which requests to reject? The blog post just mentions that excess RPS gets rejected, but couldn't rejecting arbitrary requests cause other problems?
Re: Concurrency Limits by Netflix
#19Can anyone explain how they decide which requests to reject? The blog post just mentions that excess RPS gets rejected, but couldn't rejecting arbitrary requests cause other problems?
I think their point is basically "It doesn't matter" - any client that sends a request which then gets rejected automatically retries and is bound to get a server that is up and has capacity. The retry happens so fast that even with just a naive retry implementation, the end-user won't even notice the interruption.
From https://medium.com/@NetflixTechBlog/performance-under-load-3...
> The discovered limit and number of concurrent requests can therefore vary from server to server, especially in a multi-tenant cloud environment. This can result in shedding by one server when there was enough capacity elsewhere. With that said, using client side load balancing a single client retry is nearly 100% successful at reaching an instance with available capacity. Better yet, there’s no longer a concern about retries causing DDOS and retry storms as services are able to shed traffic quickly in sub millisecond time with minimum impact to performance.
Edit: In terms of how they decide what to reject, from reading the blog post, there is a queue and there is a limit to how big the queue can be. Requests that come in while the queue is "full" get rejected immediately. They don't wait in the queue and timeout.
Re: Concurrency Limits by Netflix
#20Can anyone explain how they decide which requests to reject? The blog post just mentions that excess RPS gets rejected, but couldn't rejecting arbitrary requests cause other problems?
My guess is they use a 0 / small queue in front of the request pool. If queue is full (indicating the server is at its concurrency limit), it returns a 429 (which is sort of weird - return a 503 instead). I don't think that is part of the library though - the library just provides the low level bricks.