this is not really a good guide to Message Queues, it's really only talking about them in context of one use of them. It doesn't really talk about the message queue at all and the basic differences between various message queue implementations. Your local AI chatbot is going to give you a much better overview, just take the title "Message Queues: A Simple Guide with Analogies" and it does a much better job
Message Queues: A Simple Guide with Analogies (2024)
21–29 of 29 posts
Re: Message Queues: A Simple Guide with Analogies (2024)
#22I’ve been thinking that defaulting to durable execution over lower‑level primitives like queues makes sense a lot of the time, what do you think? A lot of the "simple queue" use cases end up needing extra machinery like a transactional‑outbox pattern just to be reliable. Durable‑execution frameworks (DBOS/Temporal/etc.) give you retries, state, and consistency out of the box. Patterns like Sagas also tend to get stit…
Drawing the boundary at high throughput, huge fan-out and ultra-low-latency is correct - I'd also add that MQs are often used for pub/sub and signaling.
MQs are heavily optimized for reducing E2E latency between publishers and consumers in a way that DE engines are not, since DE engines usually rely on an ACID compliant database. Under load I've seen an order of magnitude difference in enqueue times (low single-digit milliseconds for the MQ p95 vs 10ms p95 for Postgres commit times). And AMQP has a number of routing features built-in (i.e. different exchange types) that you won't see in DE engines.
Another way to think about it is that message queues usually provide an optional message durability layer alongside signaling and pub/sub. So if you need a very simple queue with retries _and_ you need pub/sub, I'd be eyeing an MQ (or a DE execution engine that supports basic pub/sub, like Hatchet).
I wrote about our perspective on this here: https://hatchet.run/blog/durable-execution
( disclaimer - I'm one of the people behind https://github.com/hatchet-dev/hatchet )
Re: Message Queues: A Simple Guide with Analogies (2024)
#23This is surprisingly basic knowledge for ending up on the front page. It’s a good intro, but I’d love to read more about when to know it’s time to replace my synchronous inter service http requests with a queue. What metrics should I consider and what are the trade offs. I’ve learned some answers to this question over time, but these guys are theoretically message queue experts. I’d love to learn about more things to…
I think the article would be a little bit more useful to non-beginners if it included an update on the modern landscape of MQs. Are people still using apache kafka lol? it is a fine enough article as it is though!
Kafka is a distributed log system. Yes, people use Kafka as a message queue, but it's often a wrong tool for the job, it wasn't designed for that.
Re: Message Queues: A Simple Guide with Analogies (2024)
#24I’ve been thinking that defaulting to durable execution over lower‑level primitives like queues makes sense a lot of the time, what do you think? A lot of the "simple queue" use cases end up needing extra machinery like a transactional‑outbox pattern just to be reliable. Durable‑execution frameworks (DBOS/Temporal/etc.) give you retries, state, and consistency out of the box. Patterns like Sagas also tend to get stit…
It doesn't make sense to persist by default. If I send a message to my rendering thread's message queue that the window is fully occluded I never want that message to be persisted. If the process crashes the fact that the window was occluded back then has no relevance to the current insurance of the process.
Trying to persist things has a performance cost that you don't want to pay everytime you want a thread to communicate with another.
Do you think ArrayList or std::vector should be persisted by default?
Re: Message Queues: A Simple Guide with Analogies (2024)
#25While queues definitely play an important role in microservices architecture, I think it’s worth clarifying that they’re not unique to it. A queue can fit perfectly in a monolith depending on the use case. I regularly use queues for handling critical operations that might require retrying, for having better visibility into failed jobs, ensuring FIFO guarantees, and more. Queues are such a useful tool for building any…
Monoliths also have to scale to multiple servers eventually, so message queues are an important architectural component to understand regardless of the organization of your services.
Even without multiple servers a single server itself has many cores. So if you aren't using multiple threads you are leaving performance on the table.
Re: Message Queues: A Simple Guide with Analogies (2024)
#26Earlier quoted context omitted.
> This is surprisingly basic knowledge for ending up on the front page. Nothing wrong with that! Hacker News has a large audience of all skill levels. Well written explainers are always good to share, even for basic concepts.
Agree! In fact, I would appreciate more well written articles explaining basic concepts on the front page of Hacker News. It is always good to revisit some basic concepts, but it is even better to relearn them. I am surprised by how often I realize that my definition of a concept is wrong or just superficial.
Also it's nice to have a set of well-written explainers for when someone asks about a concept.
Re: Message Queues: A Simple Guide with Analogies (2024)
#27This is surprisingly basic knowledge for ending up on the front page. It’s a good intro, but I’d love to read more about when to know it’s time to replace my synchronous inter service http requests with a queue. What metrics should I consider and what are the trade offs. I’ve learned some answers to this question over time, but these guys are theoretically message queue experts. I’d love to learn about more things to…
> when to know it’s time to replace my synchronous inter service http requests with a queue
I've found that once it's inconveniently long for a synchronous client side request, it's less about the performance or metrics and more about reasoning. Some things are queue shaped, or async job shaped. The worker -> main app communication pattern can even remain sync http calls or not (like callback based or something), but if you have something that has high variance in timing or is a background thing then just kick it off to workers.
I'd also say start simple and only go to Kafka or some other high dev-time overhead solution when you start seeing Redis/Rabbit stop being sufficient. Odds are you can make the simple solution work.
Re: Message Queues: A Simple Guide with Analogies (2024)
#28Earlier quoted context omitted.
Monoliths also have to scale to multiple servers eventually, so message queues are an important architectural component to understand regardless of the organization of your services.
Even without multiple servers a single server itself has many cores. So if you aren't using multiple threads you are leaving performance on the table.
A single multithreaded process usually doesn’t need an external message queue for sharing, though.
I guess if you’re stuck with a single threaded language you would want a message queue though.
Re: Message Queues: A Simple Guide with Analogies (2024)
#29Earlier quoted context omitted.
Even without multiple servers a single server itself has many cores. So if you aren't using multiple threads you are leaving performance on the table.
A single multithreaded process usually doesn’t need an external message queue for sharing, though. I guess if you’re stuck with a single threaded language you would want a message queue though.
No one specified external message queue. You can have a message queue within the process itself that can deliver messages from one thread to another. There are different kinds of messages queues for this purpose depending on how many threads will be producing messages to the message queue and how many threads will be consuming messages from it. While using message queues may not be neccessary for multithreading it is a very common setup. The ability to have a message queue to schedule work on a background thread is very common between many programs.