Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

NSQ: Realtime distributed message processing at scale (in Go)

github.com

Post reply on HN
Print view View original

11–20 of 46 posts

Re: NSQ: Realtime distributed message processing at scale (in Go)

#11

Post by jehiah »

wheaties wrote: post #2

I love seeing how people tackle these sorts of problems. Here, the trade-off is duplicate messages reaching a client. I know others have tried a no-"ack" approach. I'm still working in the land of RabbitMQ solving all my business needs and I like it. I'd love to see some numbers, reasons for decisions made, and suggested best practices for this solution other than "manual de-dupe."

Some of the background behind our decisions are on our blog post http://word.bitly.com/post/33232969144/nsq

In terms of manual de-duping, we strive internally for idempotent message processing so it's fairly irrelevant, but to handle cases where it matters, all of our messages have unique id's added to them outside of NSQ.

The actual cases where messages are handled multiple times is limited to when a client disappeared during message processing (a hard restart) or it passed the allowable time window to respond and was given to another client.

If there are any specific numbers you are curious about, please ask.

Re: NSQ: Realtime distributed message processing at scale (in Go)

#12

Post by matticakes »

jallmann wrote: post #4

Was something like MQTT investigated prior to re-inventing the wheel with NSQ? From a cursory examination, I don't see any major advantages other than nice UI tools. The wire protocol seems more complicated -- why distinct protocols for producers and consumers? Linebreak-delimited headers are error-prone and ought to be banished. Why transmit the hostname with sub requests -- is another system requesting jobs on beha…

one of the developers here...

The protocols that exist in NSQ now are designed to be the simplest implementation that worked.

You've correctly pointed out some of the issues. At this stage, the distinction of producer vs consumer was mostly so that you could publish at all without having to use the HTTP interface. For our use cases, in particular taking advantage of the /mput endpoint, we aren't even using the TCP based publishing protocol.

Re: your point on sending metadata with SUB commands. I agree its a bit ugly. We actually intend on improving that aspect by instead sending the data in the form of an IDENTIFY type command upon initial connection. That information is used in the various administrative UIs and endpoints.

I'm going to do a bit more reading on MQTT, thanks for the link.

Re: NSQ: Realtime distributed message processing at scale (in Go)

#13

Post by matan_a »

We're currently using Apache Kafka which is working out great, but does include a broker as a middleman. This is good for us since it's designed to accumulate unconsumed data (or even roll it back) and still provide good performance - rather than having that happen on the producers or consumers.

It would be cool to see some stats on throughput and latency relative to # of producers / consumers and amount of data currently accumulated in the producers (since there is no middleman).

Re: NSQ: Realtime distributed message processing at scale (in Go)

#14

Post by philjr »

jlgreco wrote: post #9
bgentry wrote: post #8

This ensures that the only edge case that would result in message loss is an unclean shutdown of an nsqd process. In that case, any messages that were in memory (or any buffered writes not flushed to disk) would be lost. I don't understand how you can call it a message delivery "guarantee" when you're susceptible to losing messages when a node dies. One solution is to stand up redundant nsqd pairs (on separate hosts)…

How would you design a system that didn't have this problem at the individual node level?

A similar design to TCP/IP - sequence numbers, acknowledgements & resending capability on sender side

Re: NSQ: Realtime distributed message processing at scale (in Go)

#16

Post by jlgreco »

dclusin wrote: post #10
jlgreco wrote: post #9

Earlier quoted context omitted.

How would you design a system that didn't have this problem at the individual node level?

Sequence numbers on messages sent by producers coupled with some sort of persistence by the sender. This way when a gap is detected some sort of resend protocol can be implemented to achieve the delivery guarantee.

So the sender just keeps trying until it gets an ACK. Fair enough, though I imagine you'd want to run multiple nodes even with this.

Re: NSQ: Realtime distributed message processing at scale (in Go)

#19

Post by matticakes »

jordanlewis wrote: post #3

It's kind of like Storm. I'd love to see a comparison written by the authors.

Agreed, a comparison would be helpful for highlighting the tradeoffs behind the choices we made.

We do talk about the evolution of our infrastructure and the genesis of NSQ in our blog post, http://word.bitly.com/post/33232969144/nsq.

Post reply on HN