51–60 of 64 posts
Previous discussion on bitly blog / reddit with some comparisons to zmq: http://www.reddit.com/r/programming/comments/117dsj/nsq_real... And also a great article that compares / contrasts NSQ along with a lot of other alternatives: http://www.bravenewgeek.com/dissecting-message-queues/
I'm glad these new projects are coming on, and this one seems to be very forthright about its limitations, but I'm just putting this in the bucket with all of the other messaging systems that provide a minimum feature set. I haven't seen much on the market recently that offers things like (first-class) persistence, guaranteed ordering, guaranteed delivery, or any of the other more complex distribution patterns. That'…
Hello, in the last couple of months I've been working on a message queue of the kind you describe, which is, one that is more biased about providing a number of features already built in inside the broker itself, instead of delegating it to the client. I understand the case for the other approach taken by NSQ and other systems, it's just a matter of what you want to do. However while my queue project supports persist…
DDS supports all of this including the QOS layer, but no request/reply unfortunately. The existing open source DDS implementations are massive code bases that I wouldn't dare go near to attempt extending, and DDS is heavily based on IDL, which I'd prefer avoiding. If I had the messaging library I could do the QOS + persistence layer, but so far I haven't seen anything that ticks all the boxes (ZMQ has the raw capabilities but I'd need to build out the publish/subscribe and request/reply protocol, which I honestly don't have the time/skill to do.)
Earlier quoted context omitted.
Hello, in the last couple of months I've been working on a message queue of the kind you describe, which is, one that is more biased about providing a number of features already built in inside the broker itself, instead of delegating it to the client. I understand the case for the other approach taken by NSQ and other systems, it's just a matter of what you want to do. However while my queue project supports persist…
Is your queue project published anywhere? Why did you choose to support persistence? I'm dying for a messaging library that supports only brokerless publish/subscribe with QOS support, request/reply and discovery. On top of that I want to build a persistence layer supporting 'retry' and 'replay' operations; this would allow me to build highly decoupled components, with the ability to do maintenance operations (retry)…
Having played with NSQ off and on for the past few months, and having gone deeper with it over the past week or so in preparation for rolling out a production service, here are a few things that have really impressed me, in no particular order: It's super easy to run. A few command line params, if that, and I've got a local nsqd running that I can develop and test against. Great for those offline coding sessions on B…
Thanks! This is something that I've always tried to stress when talking about NSQ... The "message queue" is the most boring and uninteresting aspect of the system. It's the combination of out-of-the-box tooling and conceptually simple primitives that really differentiate it from other systems.
Thanks!
Earlier quoted context omitted.
Thanks! This is something that I've always tried to stress when talking about NSQ... The "message queue" is the most boring and uninteresting aspect of the system. It's the combination of out-of-the-box tooling and conceptually simple primitives that really differentiate it from other systems.
The Features page says it is "horizontally scalable (no brokers, seamlessly add more nodes to the cluster)", but the Design page says you need to run a 'nsdq' daemon - which is a message broker - i.e. you cannot embed the daemon into your application?
That sentence is missing an important word. No centralized brokers, meaning the typical/recommended deployment topology is an nsqd node on all hosts producing messages.
Earlier quoted context omitted.
For the applications I've dealt with, it's an upside that a workers distributed across a number of servers can process messages from producers distributed across a number of servers, with no user-defined partitioning. Kafka, I think, requires you to define partitions to have multiple workers consume a stream. In the NSQ case, if you need more throughput, just spin it up. Somewhat on a tangent, one of the goals of the…
Thanks, that's a very helpful explanation. (A partition key is optional in Kafka - the DefaultPartitioner just computes a random partition regardless of key.) I am having trouble visualizing this in NSQ: > [A server] might consume and acknowledge multiple messages to produce one new message I can imagine a server consuming multiple messages and performing a single write to e.g. ElasticSearch or Cassandra - in this ca…
I suppose I was thinking that it might reduce "just a little", just adding stuff together as appropriate, but that wouldn't really be useful.
In practice, the way multiple events turn into a single event is that some are just thrown out, according to some static filter rule, or after checking a datastore.
The places where NSQ are typically used is where new data never stops flowing. So sometimes for this sort of thing, it updates a value in some datastore, and creates a new message for the next stage with the latest value, perhaps only if the latest value reaches some threshold, in a scheme where the value decays.
Can somebody comment on the advantages of using this over zeromq/nanomsg? Thanks!
Earlier quoted context omitted.
Thanks! This is something that I've always tried to stress when talking about NSQ... The "message queue" is the most boring and uninteresting aspect of the system. It's the combination of out-of-the-box tooling and conceptually simple primitives that really differentiate it from other systems.
The Features page says it is "horizontally scalable (no brokers, seamlessly add more nodes to the cluster)", but the Design page says you need to run a 'nsdq' daemon - which is a message broker - i.e. you cannot embed the daemon into your application?
That said, for a larger deployment you'd still probably want to run the lookupd as well, at which point, at least for a typical architecture, running nsqd standalone is probably fine. Embedding is darn handy for testing, though.
I don't think I had heard about NSQ till now; how is it better than, say, a pub-sub queue on a beefy Redis server? (Or a cluster of servers, if you wish). Nothing beats Redis' simplicity, AFAICT.
Redis pub/sub is ephemeral. If you aren't connected and listening, you missed it.