NSQ: Realtime distributed message processing at scale (in Go)
1–10 of 46 posts
Re: NSQ: Realtime distributed message processing at scale (in Go)
#2I love seeing how people tackle these sorts of problems. Here, the trade-off is duplicate messages reaching a client. I know others have tried a no-"ack" approach. I'm still working in the land of RabbitMQ solving all my business needs and I like it.
I'd love to see some numbers, reasons for decisions made, and suggested best practices for this solution other than "manual de-dupe."
Re: NSQ: Realtime distributed message processing at scale (in Go)
#3It's kind of like Storm. I'd love to see a comparison written by the authors.
Re: NSQ: Realtime distributed message processing at scale (in Go)
#4Was something like MQTT investigated prior to re-inventing the wheel with NSQ? From a cursory examination, I don't see any major advantages other than nice UI tools.
The wire protocol seems more complicated -- why distinct protocols for producers and consumers? Linebreak-delimited headers are error-prone and ought to be banished. Why transmit the hostname with sub requests -- is another system requesting jobs on behalf of workers?
Then the topic/channel distinction seems artificial when wildcards would suffice (and provide much more flexibility), eg topic/*/channels or topic/channels/# in the MQTT parlance. MQTT also has more fine-grained delivery guarantees via its QoS levels. All this with a header structure that's as small as two bytes.
edit: While MQTT is pubsub, as long as the system is under your control, you're free to change the semantics at the broker side from "broadcast messages to all consumers" to "rotate messages among consumers."
Re: NSQ: Realtime distributed message processing at scale (in Go)
#5We use ActiveMQ at work and are investigating other MQs too. I dont see any info or performance graphs w.r.t message size. That would be a useful addition.
Re: NSQ: Realtime distributed message processing at scale (in Go)
#6We use ActiveMQ at work and are investigating other MQs too. I dont see any info or performance graphs w.r.t message size. That would be a useful addition.
Re: NSQ: Realtime distributed message processing at scale (in Go)
#7Was something like MQTT investigated prior to re-inventing the wheel with NSQ? From a cursory examination, I don't see any major advantages other than nice UI tools. The wire protocol seems more complicated -- why distinct protocols for producers and consumers? Linebreak-delimited headers are error-prone and ought to be banished. Why transmit the hostname with sub requests -- is another system requesting jobs on beha…
Yes, it would be nice if someone from the team behind NSQ elaborates a bit on why they didn't go with an existing protocol like MQTT.
Re: NSQ: Realtime distributed message processing at scale (in Go)
#8This ensures that the only edge case that would result in message loss is an unclean shutdown of an nsqd process. In that case, any messages that were in memory (or any buffered writes not flushed to disk) would be lost.
I don't understand how you can call it a message delivery "guarantee" when you're susceptible to losing messages when a node dies.
One solution is to stand up redundant nsqd pairs (on separate hosts) that receive copies of the same portion of messages.
OK, delivery is only guaranteed if I run multiple independent sets of NSQd and write messages to both.
Regardless, it looks like an interesting project.
Re: NSQ: Realtime distributed message processing at scale (in Go)
#9This ensures that the only edge case that would result in message loss is an unclean shutdown of an nsqd process. In that case, any messages that were in memory (or any buffered writes not flushed to disk) would be lost. I don't understand how you can call it a message delivery "guarantee" when you're susceptible to losing messages when a node dies. One solution is to stand up redundant nsqd pairs (on separate hosts)…
How would you design a system that didn't have this problem at the individual node level?
Re: NSQ: Realtime distributed message processing at scale (in Go)
#10This ensures that the only edge case that would result in message loss is an unclean shutdown of an nsqd process. In that case, any messages that were in memory (or any buffered writes not flushed to disk) would be lost. I don't understand how you can call it a message delivery "guarantee" when you're susceptible to losing messages when a node dies. One solution is to stand up redundant nsqd pairs (on separate hosts)…
How would you design a system that didn't have this problem at the individual node level?
Sequence numbers on messages sent by producers coupled with some sort of persistence by the sender. This way when a gap is detected some sort of resend protocol can be implemented to achieve the delivery guarantee.