21–30 of 47 posts
In case anyone is interested, I wrote an async/await stream library for JavaScript/Node.js which supports backpressure management. It's heavily tested and used as part of SocketCluster (pub/sub SDK) https://socketcluster.io/ which is also heavily tested.
I like the article, but I am not sure that I agree with the terminology: I would not call "buffering" a form of back pressure. Imho there is really only one type of back pressure: the one the author calls "control". The other 2 are just ways to "release" pressure.
Backpressure is an implicit signal from consumer to producer that there is not enough capacity. It propagates backwards from flow like water in a network of pipes.
In their example, the conveyor belt keeps adding chocolates because there is no backpressure. The effect is that Lucy engages in load shedding, and the producer of the chocolates has no say or insight into what happens when that load has to be shed. If there was backpressure, the producer gets to choose what happens.
Isn't the ideal solution to make the throttled system faster? Like autoscale horizontally and/or vertically, sharding or just writing better code? Everything in this article is about to cope with back pressure but solving is frequently possible.
Never heard it outside of the context of flow control in networking. Not a thing in software outside of pieces communicating over a network, not using a protocol which has implicit flow control like TCP. E.g. words like "the lexer was producing tokens too fast, so the parser applied backpressure" have never been heard.
In other words, I/O is faster than CPU for this task.
A naive design where I read the data as fast as possible on a dedicated thread, and dump it into an unbounded queue that a thread-per-core consumes, will quickly run out of memory.
By putting a limit on queue depth, the queue can communicate back to the storage reader that it can't accept more data. This is backpressure. The reader in turn can decide whether to e.g. wait, slow down, discard etc. as appropriate for the use case.
Never heard it outside of the context of flow control in networking. Not a thing in software outside of pieces communicating over a network, not using a protocol which has implicit flow control like TCP. E.g. words like "the lexer was producing tokens too fast, so the parser applied backpressure" have never been heard.
I like the article, but I am not sure that I agree with the terminology: I would not call "buffering" a form of back pressure. Imho there is really only one type of back pressure: the one the author calls "control". The other 2 are just ways to "release" pressure.
Edit: I think the article also misses one of the most important ways to release pressure. And that is scaling throughput either horizontally (e.g. by adding more servers) or vertically (e.g. by optimization of software)
Isn't the ideal solution to make the throttled system faster? Like autoscale horizontally and/or vertically, sharding or just writing better code? Everything in this article is about to cope with back pressure but solving is frequently possible.
Sometimes you just have a server, not some elastic horizontal setup, or a budget for vertical scaling.
And "writing better code" is an optimization thing, not a general way to handle back-pressure (you can have back-pressure in a fully optimized program too, and even for non-optimal code you don't want to just have to iterate and refactor/improve the code every time there's back-pressure).
There is a subtler issue. Even if your average input and output rates are OK, the lumpiness (stochastic variations) in input and processing rates can cause queues to build up. In the simplest example, with "well behaved" arrival and processing rates, and a single server (an M/M/1 queue), the average queue length is 1/(1-mu) where the utilization mu = avg arrival rate / avg processing rate. So as the arrival rate appr…
Use a LIFO with a timeout. When you find a request that exceeds the timeout clear the lifo.
Earlier quoted context omitted.
Use a LIFO with a timeout. When you find a request that exceeds the timeout clear the lifo.
why lifo? can you elaborate?
problem is that the most common backpressure techniques like exponential back-off and sending a retry-after time in the response header have constraints on maximum backoff time they can do, in some scenarios that is much much less than the normal.
for example, imagine a scenario where a customer explores 10 items on Amazon, and then finally places an order, so 10rps for the product page and 1 rps for the order page. if order services goes down, slowly the customers get stuck on the order page and even with backpressure, your RPS keeps on growing on the order page. exponential backoff doesn't help as well
while dropping requests is a good idea, but that action is not designed by default every time, systems go into metastable state and you need the ability to control the throughput on producer side
you could solve it by keeping a different layer in between like load balancer or some gateway layer that is resilient against such throughput spikes and will let you control throughput on your service and slowly scale up the throughput as per your requirements (by user or by random)
for frontend devices, it gets exponentially harder to control the throughput. having an independent config API that can control the throughput is the best solution that I came across