NSQ – A realtime distributed messaging platform

Thanks!

This is something that I've always tried to stress when talking about NSQ...

The "message queue" is the most boring and uninteresting aspect of the system.

It's the combination of out-of-the-box tooling and conceptually simple primitives that really differentiate it from other systems.

I don't think I had heard about NSQ till now; how is it better than, say, a pub-sub queue on a beefy Redis server? (Or a cluster of servers, if you wish). Nothing beats Redis' simplicity, AFAICT.

It looks like there isn't any kind of user-definable shard/partition key available within the topics - a message within a topic could go to any client subscribing to a channel for that topic? Is that correct?

That's obviously fine for AWS Lambda-style single event processing (maps, filters, sinks), but isn't that going to make multiple event processing (reduces, sorts) really difficult? It rules out using in-process memory or local KV storage for maintaining the necessary state across the aggregation window - you're only left with pounding some remote database, not a great strategy as per http://radar.oreilly.com/2014/07/why-local-state-is-a-fundam...

For the applications I've dealt with, it's an upside that a workers distributed across a number of servers can process messages from producers distributed across a number of servers, with no user-defined partitioning. Kafka, I think, requires you to define partitions to have multiple workers consume a stream. In the NSQ case, if you need more throughput, just spin it up.

Somewhat on a tangent, one of the goals of the design was to avoid any servers in the middle of the "workers" (consumers) and producers, and making it such that any consuming or producing server dieing or experiencing a network partition doesn't slow down any producers or consumers who can still reach each other.

You're right, this isn't really focused on replacing map-reduce thing like some other message streaming systems are. For one, I can't imagine a "sorting" step using nsq. It's still used for somewhat hefty data-processing tasks, in multiple stages. Each stage has as many servers spun up as needed, consuming messages directly from the previous stage, and publishing to nsqd on localhost. They might consume and acknowledge multiple messages to produce one new message.

For reduced inter-server hops before hitting the database, you can run workers on the same host as producers. So you might have a X hosts producing messages (and running their own nsqd locally, of course), and have Y consumers on each host, configured to consume messages from localhost instead of using nsqlookupd to find all sources, and those consumers would make direct requests to the appropriate database host. With this setup you can still use nsqadmin to monitor the queue levels on all producer hosts, and you can still be archiving messages to disk on some other remote hosts.

+1 agree with that (although you can use the DB based persistent store).

The other obvious problem with AMQ is that it supports XA transactions which means people are tempted to use it, which in turn leads to much pain and suffering for all concerned..