41–50 of 59 posts
Tangential, but when dealing with queues, the first thing you want to do is have a basic grounding of queuing theory, and know whether you're optimising for throughput or worker utilisation (i.e. what are your SLAs and efficiency targets?). IME each goal involves fairly different metrics and scaling rules, so you'll want to know what you're prioritising.
Anyone know a good into to queuing theory?
The UNIX pipe really is an incredible concurrency invention which is not well understood, and attempts to work around its features turn into bugs. A buffer to accumulate data that blocks when it’s full allows you to handle bursty loads. It solves the back pressure problem of readers and writers operating at different speeds. It doesn’t over consume resources. It also solves the architecture problem of when to trigger…
Since the pipe is generally unidirectional, the sending process has no way of knowing whether the receiving process has received, or even more successfully processed, anything sent. For that, one needs to make a pipe in the opposite direction and that is not as easy anymore; it also requires building your own protocol to identify and acknowledge sent work items.
The UNIX pipe really is an incredible concurrency invention which is not well understood, and attempts to work around its features turn into bugs. A buffer to accumulate data that blocks when it’s full allows you to handle bursty loads. It solves the back pressure problem of readers and writers operating at different speeds. It doesn’t over consume resources. It also solves the architecture problem of when to trigger…
Most message queues want to be able to distribute work to multiple parallel consumers, either for performance, redundancy, re-deployment of the consumers, and so on. Like, sure, you can do that with named pipes, but they're not durable; their state can be lost in the event of a crash or reboot.
Other than like ... in-process, thread-to-thread message queues, almost all message queues are used because they provide some form of durability (either by persisting messages to disk, or just by virtue of running on a separate server/process than much more crash-prone producers/consumers).
> imperatively, rather than one end being imperative and the other being a declarative graph of callbacks (all those “reactive” libraries).
MQ client libraries are heavily reactive because they need to work with MQs that are network services. If every consumer that ever wanted to fetch a message had to issue a network RPC poll/timeout cycle, and they were all doing that constantly, that'd be both a lot of traffic for the MQ to handle and a lot of network chatter. Some MQs do use (or support) the RPC poll model, but especially among the more performant ones, most of them are push-based so clients can just sit in an epoll/select statement waiting for the message queue to send them traffic.
That said, you're totally right that plenty of MQ client frameworks/libraries are way over complicated and want to own your whole program's entry point and design. Callbacks are probably a necessary outgrowth of the underlying network behavior, but tortured whole-program-event-loop-ownership architecture is not.
> Even using a term like “back pressure” is a tell to me that someone is confused snd doing something architecturally wrong.
I don't think your other points support this; you said yourself that pipes' bounded nature is good, so what's the problem here?
The UNIX pipe really is an incredible concurrency invention which is not well understood, and attempts to work around its features turn into bugs. A buffer to accumulate data that blocks when it’s full allows you to handle bursty loads. It solves the back pressure problem of readers and writers operating at different speeds. It doesn’t over consume resources. It also solves the architecture problem of when to trigger…
The UNIX pipe has the (for many systems) undesirable negative property of losing data in the pipe when the receiving process terminates: Anything in the kernel buffer of the pipe gets lost. Since the pipe is generally unidirectional, the sending process has no way of knowing whether the receiving process has received, or even more successfully processed, anything sent. For that, one needs to make a pipe in the opposi…
So are most message queues.
If the producer needs to know when the consumer has finished work, that's not a queue; that's an RPC.
The UNIX pipe really is an incredible concurrency invention which is not well understood, and attempts to work around its features turn into bugs. A buffer to accumulate data that blocks when it’s full allows you to handle bursty loads. It solves the back pressure problem of readers and writers operating at different speeds. It doesn’t over consume resources. It also solves the architecture problem of when to trigger…
The UNIX pipe has the (for many systems) undesirable negative property of losing data in the pipe when the receiving process terminates: Anything in the kernel buffer of the pipe gets lost. Since the pipe is generally unidirectional, the sending process has no way of knowing whether the receiving process has received, or even more successfully processed, anything sent. For that, one needs to make a pipe in the opposi…
The UNIX pipe really is an incredible concurrency invention which is not well understood, and attempts to work around its features turn into bugs. A buffer to accumulate data that blocks when it’s full allows you to handle bursty loads. It solves the back pressure problem of readers and writers operating at different speeds. It doesn’t over consume resources. It also solves the architecture problem of when to trigger…
Bad take. Most of the different behaviors you criticize about message queue systems arise from them needing to a) distribute work b) durably, c) over the network. Most message queues want to be able to distribute work to multiple parallel consumers, either for performance, redundancy, re-deployment of the consumers, and so on. Like, sure, you can do that with named pipes, but they're not durable; their state can be l…
One of my takeaways is that if you do not handle on data transfer in the http request/response then you are choosing to exit the pipe model. Once you do that, yes all kinds of crazy async problems exist which require very complex tools to wrangle.
See the sibiling comment from Google engineer. It’s better to do work synchronously than delay it until later.
So now we get your list of requirements that MQ is for. Ok but how did we get here? What problem are we trying to solve? The answer is usually a performance problem for which this is a bandaid.
Another possible model for this problem is email. You can find some great threads here about large amounts of infrastructure being replaced with Unix email tools. Although I have yet to try it!
> If every consumer that ever wanted to fetch a message had to issue a network RPC poll/timeout cycle, and they were all doing that constantly, that'd be both a lot of traffic for the MQ to handle and a lot of network chatter
This makes no sense to me. A while loop blocked on a socket read with a producer with a write does not generate any extra network “chatter”.
Anytime you are using callbacks you are choosing not to have a thread with imperative logic.
> I don't think your other points support this
Correct that was not a conclusion.
What I’m trying to say is if you want to send data between two systems and you have producers and consumers at different rates, you don’t have to do anything special. The UNIX kernel is designed to solve this problem.
So when I hear that, my inclination is that we don’t understand the capabilities UNIX already offers and I’ve yet to be wrong.
Earlier quoted context omitted.
There are lots of good books and some great vids on youtube, but I'd start with this statement and work backward, because this is the non-obvious thing most bootcamp trained, promoted to CTO don't know: The single most important lesson from queuing theory for software systems is the non-linear relationship between utilisation and latency. As system utilisation approaches 1.0 (100% capacity), the average waiting time…
To explain this: if the system is 95% utilized, and a new request comes in, there's a 95% chance the system is already busy and the request has to wait. But after the currently in-progress request finishes, there's still a 95% chance the system is busy (with another request that was already queued behind it) and request X has to wait. After that one finishes, same thing. On average, request X has to wait for about 20…