Live data from Hacker News

NSQ – A realtime distributed messaging platform

nsq.io

41–50 of 64 posts

Re: NSQ – A realtime distributed messaging platform

#41
post #25
post #13

Earlier quoted context omitted.

MQTT is a wire protocol for connection a message sender and receiver, but leaves the actual messaging architecture (brokers, P2P, etc) unspecified. NSQ appears to include its own proprietary wire format, as well as a full im-memory messaging architecture.

Just want to add the NSQ wire format looks pretty efficient as well. Seems to be defined at the binary level which was one of the biggest wins for MQTT in terms of memory footprint. Although would be nice if a standard protocol was used instead of creating their own.

Binary formats have their drawbacks - the guy that designed the AMQP protocol called the wire protocol he designed for it an "expert mistake". Admittedly he was designing a spec rather than a product but worth understanding.

Source is a long but very interesting read about messaging, amqp, zeromq and others: http://www.imatix.com/articles:whats-wrong-with-amqp

Re: NSQ – A realtime distributed messaging platform

#43

Having played with NSQ off and on for the past few months, and having gone deeper with it over the past week or so in preparation for rolling out a production service, here are a few things that have really impressed me, in no particular order: It's super easy to run. A few command line params, if that, and I've got a local nsqd running that I can develop and test against. Great for those offline coding sessions on B…

Yeah, I love it. There are some good modules out there for installing it too.

https://github.com/simplereach/chef-nsq https://galaxy.ansible.com/list#/roles/2265

Re: NSQ – A realtime distributed messaging platform

#44
post #20

Earlier quoted context omitted.

Can one use NSQ as a replacement for Kafka?

Kafka offers persistence, which seems extremely rare in distributable queue. (Is it even available outside of Kafka?) On that note, what is everyone using these queues for that they can ignore durability?

Also, Kafka guarantees message order, where NSQ does not. This can be dealt with in the client, but something to note.

Re: NSQ – A realtime distributed messaging platform

#46
post #17

I'm glad these new projects are coming on, and this one seems to be very forthright about its limitations, but I'm just putting this in the bucket with all of the other messaging systems that provide a minimum feature set. I haven't seen much on the market recently that offers things like (first-class) persistence, guaranteed ordering, guaranteed delivery, or any of the other more complex distribution patterns. That'…

I would also like to see a queue system with better guarantees. We've been running RabbitMQ for a couple of years and are fed up with its lack of safety in the face of minor cluster failures.

I don't know about NSQ — it looks nice, but the lack of true persistence and ordering means it's not something I can use. For example, lack of ordering means you can't use it as an event bus for document-oriented changes.

I know it's not a popular sentiment these days, but I prefer to err on the side of correctness. I have had enough of systems that are supposedly SPOF-free but still fail for mysterious reasons and are nigh-impossible to debug once they do (ElasticSearch, ugh).

RabbitMQ's lack of transparency means it's harder to work with the contents of a queue. For example, if the queue processing has stalled for some mysterious reason, what's clogging it? With RabbitMQ, the only way to peek into a queue is to actually pop messages from it (using a tool such as rabbitmqadmin or amqp-dequeue).

Another problem with RabbitMQ is that messages go away after being acknowledged. I, for one, would like to see a bit of history. Sure, you can build this into individual apps, but it turns out this kind of history is beneficial for many kinds of apps. There's DLX, but you can't (afaik) set up rules to automatically copy acked messages into a history queue.

I can't say I like AMQP. Binary protocol, requires several layers of client tooling to work with. If anything goes weird, strace or tcpdump are of no use. It's also a very complex protocol whose interpretation and matrix of support features seems to change a lot.

I'm currently writing a small job manager in Go that's backed by PostgreSQL (at least initially) because I want something safer, more stable and more transparent. It's based on my realization that jobs are different from events, and should be treated differently.

For example, RabbitMQ has no conception of scheduling or prioritization, and messages are ephemeral objects that go away once they've been acknowledged, as opposed to jobs, which have a life before, during and after processing.

So far, it looks pretty good. Postgres can't process a bazillion messages per second, and it's not going to work for "big data", but it seems to scale decently enough.

Re: NSQ – A realtime distributed messaging platform

#49
post #17

I'm glad these new projects are coming on, and this one seems to be very forthright about its limitations, but I'm just putting this in the bucket with all of the other messaging systems that provide a minimum feature set. I haven't seen much on the market recently that offers things like (first-class) persistence, guaranteed ordering, guaranteed delivery, or any of the other more complex distribution patterns. That'…

Hello, in the last couple of months I've been working on a message queue of the kind you describe, which is, one that is more biased about providing a number of features already built in inside the broker itself, instead of delegating it to the client. I understand the case for the other approach taken by NSQ and other systems, it's just a matter of what you want to do.

However while my queue project supports persistence, synchronous replication, delayed jobs, at least once and at most once delivery semantics, automatic federation, there is one thing I don't support in the list of features you mentioned: ordering. And I believe there is a strong argument for not supporting ordering in certain kinds of message queues.

By implementing only a weak form of ordering (approximated wall-clock ordering, so that, usually jobs are served in roughly insertion order) we gain: availability (as in CAP, the system can continue with a single node), latency (even in the case of synchronous replication, you need to care just for a message to be replicated into N nodes, regardless of what those N nodes are), and functionality (the queue can auto-reissue messages after a specified retry-time, so at-least-once delivery is trivial to accomplish for consumers and producers). Depending on how the system is designed, to give up strict ordering also wins you scalability.

So I agree about your reasoning but my feeling is that message ordering is a big point that really changes how a message queue is shaped. My bet is that there are many problems where ordering is not needed but all the other features are.

Re: NSQ – A realtime distributed messaging platform

#50
post #37

It looks like there isn't any kind of user-definable shard/partition key available within the topics - a message within a topic could go to any client subscribing to a channel for that topic? Is that correct? That's obviously fine for AWS Lambda-style single event processing (maps, filters, sinks), but isn't that going to make multiple event processing (reduces, sorts) really difficult? It rules out using in-process…

For the applications I've dealt with, it's an upside that a workers distributed across a number of servers can process messages from producers distributed across a number of servers, with no user-defined partitioning. Kafka, I think, requires you to define partitions to have multiple workers consume a stream. In the NSQ case, if you need more throughput, just spin it up. Somewhat on a tangent, one of the goals of the…

Thanks, that's a very helpful explanation. (A partition key is optional in Kafka - the DefaultPartitioner just computes a random partition regardless of key.)

I am having trouble visualizing this in NSQ:

> [A server] might consume and acknowledge multiple messages to produce one new message

I can imagine a server consuming multiple messages and performing a single write to e.g. ElasticSearch or Cassandra - in this case it doesn't matter that the batched write is a random subset of messages. But I can't imagine consuming multiple messages and emitting another message - at least not without being able to reason about what partition of data was present on that server. For example, I can't detect an abandoned shopping cart because a single shopper's events will be scattered across all servers.

Maybe I'm thinking of this wrong - can you give a live use case for consuming multiple events and emitting a single message in NSQ?

Post reply on HN