31–40 of 47 posts
Earlier quoted context omitted.
I'll bet your backpressure mechanism can react at least an order of magnitude faster than your scaling mechanism.
Serverless systems are pretty decent at scaling quickly these days. The problem is rarely lack of servers in my experience, though. You usually run out of database connections or some other bottleneck first.
Earlier quoted context omitted.
why lifo? can you elaborate?
LIFO queues with a timeout, and a retry after an exponential backoff with jitter is/was kind of standard for implementing queues at Google. More info in the Google SRE book: https://sre.google/sre-book/addressing-cascading-failures/#x...
Earlier quoted context omitted.
why lifo? can you elaborate?
LIFO queues with a timeout, and a retry after an exponential backoff with jitter is/was kind of standard for implementing queues at Google. More info in the Google SRE book: https://sre.google/sre-book/addressing-cascading-failures/#x...
Earlier quoted context omitted.
LIFO queues with a timeout, and a retry after an exponential backoff with jitter is/was kind of standard for implementing queues at Google. More info in the Google SRE book: https://sre.google/sre-book/addressing-cascading-failures/#x...
The parent comment is asking why LIFO, and you’re responding “because Google does it.” I don’t think this response is helpful.
For more detail, the document cites an article from Facebook (https://dl.acm.org/doi/10.1145/2838344.2839461):
> Most services process queues in FIFO (first-in first-out) order. During periods of high queuing, however, the first-in request has often been sitting around for so long that the user may have aborted the action that generated the request. Processing the first-in request first expends resources on a request that is less likely to benefit a user than a request that has just arrived. Our services process requests using adaptive LIFO. During normal operating conditions, requests are processed in FIFO order, but when a queue is starting to form, the server switches to LIFO mode. Adaptive LIFO and CoDel play nicely together, as shown in figure 2. CoDel sets short timeouts, preventing long queues from building up, and adaptive LIFO places new requests at the front of the queue, maximizing the chance that they will meet the deadline set by CoDel. HHVM3, Facebook’s PHP runtime, includes an implementation of the Adaptive LIFO algorithm.
Never heard it outside of the context of flow control in networking. Not a thing in software outside of pieces communicating over a network, not using a protocol which has implicit flow control like TCP. E.g. words like "the lexer was producing tokens too fast, so the parser applied backpressure" have never been heard.
Things become more complicated when multi-threading and / or asynchronous IO gets involved. If the producer on the left side of the pipe has multiple threads, e.g., one writer thread and multiple worker threads, producing data for the writer thread. Then it has to invent its own signaling between the different threads, to avoid throwing away data, when its internal buffers fill up. This is effectively a kind of back pressure.
In your example with the lever and the parser: if the lexer is multi threaded, you either need unlimited buffer for the tokens or some signaling for the threads, to slow down when the parser cannot keep up. This example is a bit artificial, since most languages don’t allow parallel lexers. But with parallel compilers and one (incremental) linker, this becomes more realistic.
Isn't the ideal solution to make the throttled system faster? Like autoscale horizontally and/or vertically, sharding or just writing better code? Everything in this article is about to cope with back pressure but solving is frequently possible.
it doesn't matter. as long as there is a rate mismatch over a sufficiently long timescale, queues will build up and consume memory, latency will increase, and depending on how things are structured overall throughput will go down because of scheduling overhead and memory pressure. this is actually an inherent problem _within_ horizontally scaled services that have cross dependencies. so no, you should really implemen…
But seriously the question was "ideal solution". Not everyone can do it but just right-sizing compute and removing or offloading bottlenecks is entirely possible. So is just throwing in the towel and discarding excess load. Reddit does that all the time and their IPO was a big success. You can also scale to accommodate massive headroom by just throwing money at it. There are plenty of systems where response times are mandatory and any back pressure mitigation that slows traffic is unacceptable.
The article makes it seem like you have three options: (1) Buffer, (2) drop, or (3) “control” the producer (and gives examples of “telling” the producer to slow down.)
But the simplest thing to do is for a producer to not send a second request until the first one is done. If you have an upstream system you have to pipeline requests into, just don’t complete/ack a client request until the upstream system has acknowledged it. So your slow database server that only supports 10 concurrent writes can be limited 10 writes at a time, and your clients will see their request block until the database server serves their request.
The really hard part, and the reason why you’d need ad-hoc (often out-of-band) “signaling” to control the producer, comes when you decide you want to have unbounded concurrency. It’s tempting, because synchronous requests are slow! You can’t start the next request until the previous one is acknowledged! How inefficient! But unless you have a true need to do otherwise, it’s also the simplest and most reliable way of doing things. It’s how things like file copying work straight out of the box on just about every operating system: Read from one place, write to another, but don’t just keep reading forever: Writes block until they’re fully received, then you read the next block. Add some buffers as needed to make things a bit more efficient, but the abstraction is still synchronous.
Earlier quoted context omitted.
LIFO queues with a timeout, and a retry after an exponential backoff with jitter is/was kind of standard for implementing queues at Google. More info in the Google SRE book: https://sre.google/sre-book/addressing-cascading-failures/#x...
Why is jitter important in a queue?
For instance, a bunch of clients all make a request to a server at the same time, briefly saturating the server. If all the clients have the same timeout without jitter, they will all try again together at the same time once the timeout expires, saturating the server again and again. Jitter helps by « spreading » those clients in time, thus « diluting » the server load. The server can then process these requests without saturating.
I like the article, but I am not sure that I agree with the terminology: I would not call "buffering" a form of back pressure. Imho there is really only one type of back pressure: the one the author calls "control". The other 2 are just ways to "release" pressure.
I agree. The author has interpreted “backpressure” to mean pressure from behind, but my understanding of the term is the other direction, like water pooling up at a partially blocked drain. Water is prevented from entering the pipe because the pipe is at capacity. Backpressure is an implicit signal from consumer to producer that there is not enough capacity. It propagates backwards from flow like water in a network o…
Mozilla's documentation[1] similarly defines backpressure as "the process by which a single stream or a pipe chain regulates the speed of reading/writing".
The article confuses backpressure (the signal used for regulation of the flow) with the reason backpressure is needed (producers and consumers working at different speeds). It should be fairly clear from the metaphor, I would have thought: With a pipe of unbounded size there is no pressure. The pressure builds up in a fixed-size pipe when consumer is slower than producer, which in turn slows down the producer. (Or the pipe explodes, or springs a leak and has to drop data on the ground.)
[0] https://streams.spec.whatwg.org/#pipe-chains
[1] https://developer.mozilla.org/en-US/docs/Web/API/Streams_API...