NSQ: Realtime distributed message processing at scale (in Go) • Hacker News Board

Re: NSQ: Realtime distributed message processing at scale (in Go)

#41

Post by dikbrouwer » Wed, Oct 10, 2012, 5:35 AM UTC

matticakes wrote: post #37

dikbrouwer wrote: post #36

Earlier quoted context omitted.

Right, I guess my question more specifically is: ZeroMQ allows you to built applications like this in (seemingly) just a few lines of code. Did you guys consider building it on top of ZeroMQ, and if so, what made your decide to not use it? Nothing against what you've built btw, it looks great, but am just curious

The truth is we wanted to use ZeroMQ initially. It would have been really awesome to have all that flexibility on the client side. The ZeroMQ documentation is fantastic as well. It inspired and helped shape some of the design choices we made. The further along we got from design to implementation it became obvious that it would be important to "own" the socket. Generally speaking, this is exactly what ZeroMQ prevents…

Makes a lot of sense, thanks for sharing!

Re: NSQ: Realtime distributed message processing at scale (in Go)

#42

Post by rabbitmq » Wed, Oct 10, 2012, 7:33 AM UTC

matticakes wrote: post #33

rabbitmq wrote: post #31

Earlier quoted context omitted.

Master-slave etc are HA techniques for replicating state eg queues (or "channels" as nsq calls them?). "Decentralized and distributed" has nothing to do with state, master/slave or any of that; it's a topological property of your system. You can certainly build a decentralized and distributed system using a set of brokers, regardless of whether they support HA.

That's fair, you certainly can use rabbitmq in a distributed and decentralized fashion. You bring up an important point though... What is important is the topology we're promoting and the platform we've built to support it. The actual queue is less important and it made sense for us to own that piece to achieve our goals.

So, why not just deliver that topology using off the shelf components? Rabbit, Kafka, ZeroMQ, ... the toolbox is deep. This is not meant as a criticism, I am simply trying to understand what I am missing when I look at your design :-)

Re: NSQ: Realtime distributed message processing at scale (in Go)

#43

Post by rabbitmq » Wed, Oct 10, 2012, 7:37 AM UTC

jallmann wrote: post #4

Was something like MQTT investigated prior to re-inventing the wheel with NSQ? From a cursory examination, I don't see any major advantages other than nice UI tools. The wire protocol seems more complicated -- why distinct protocols for producers and consumers? Linebreak-delimited headers are error-prone and ought to be banished. Why transmit the hostname with sub requests -- is another system requesting jobs on beha…

shameless rabbitmq mqtt plug here :-) http://www.rabbitmq.com/blog/2012/09/12/mqtt-adapter/

Re: NSQ: Realtime distributed message processing at scale (in Go)

#44

Post by noselasd » Wed, Oct 10, 2012, 9:07 AM UTC

dikbrouwer wrote: post #36

jeremyjh wrote: post #35

Earlier quoted context omitted.

ZeroMQ is a library, it is embedded in the application code and can support a peer-to-peer messaging topology. It also has support for Broker patterns but you have to develop the Broker application yourself; you could build a solution such as this one using it, but the zmq library itself really handles lower-level concerns with a high degree of flexibility but no out-of-the-box functionality. zmq can be thought of as…

Right, I guess my question more specifically is: ZeroMQ allows you to built applications like this in (seemingly) just a few lines of code. Did you guys consider building it on top of ZeroMQ, and if so, what made your decide to not use it? Nothing against what you've built btw, it looks great, but am just curious

So - this has nothing to do with NSQ - just my $0.02. Maybe zeromq has changed a lot since I looked at it quite a while back. The trouble we constantly ran into was zeromq is really just a fancy socket library - this means that you have to program all the logic yourself if you want certain things to happen if messages cannot be delivered (e.g. log the fact at a minimum), or get queued abnormally long, or get acks from the other side etc. Which is fine - you have to do all this yourself with plain sockets unless you just need a fire & forget system(albeit with the smarts of retrying and failover and some convenience of pub-sub). However we ran into problems where we could not get the info out of zmq that we wanted, such as the fact a peer had failed, the current rtt, and other things - leaving the gain we'd get from zmq at practically nothing.

Re: NSQ: Realtime distributed message processing at scale (in Go)

#45

Post by 4ad » Wed, Oct 10, 2012, 10:25 AM UTC

overbroad wrote: post #40

jehiah wrote: post #38

Earlier quoted context omitted.

Thanks for the reminder. Getting binary builds up is on our to-do list, so check back soon for those. In the mean time, installing Go is pretty easy (you can use brew, or the official OSX package (assuming you are on OSX) http://golang.org/doc/install ) and we've tried to leave clear steps for building NSQ here https://github.com/bitly/nsq/blob/master/INSTALLING.md (there really are very few dependencies other than g…

Cheers. I have installed Go before and messed around with it. I agree it's pretty good as far as being self-contained and straightforward. I just don't have the energy to keep chasing a moving target. For my purposes, I'm more interested in stuff that is stable and rarely changes. If you can produce binaries for UNIX (Linux, BSD, etc), OSX and Windows, I'll be impressed. That's something I was interested in doing wit…

Since the Go1 release, Go is not a moving target anymore. Stable releases are rare and there are binary packages available if you are so inclined.

Go compilers, being derived from Plan 9, are always cross compiling. To cross compile you just set and export GOARCH and GOOS to your target, for example:

  GOOS=windows GOARCH=386 CGO_ENABLED=0 go build foo.bar/baz

Would build foo.bar/baz for 32 bit Windows from any system that has Go and ran make.bash --no-clean with those variables set. More interesting is building for ARM:

  GOOS=linux GOARCH=arm CGO_ENABLED=0 go build foo.bar/baz

Re: NSQ: Realtime distributed message processing at scale (in Go)

#46

Post by old_sound » Wed, Oct 10, 2012, 12:35 PM UTC

jehiah wrote: post #11

wheaties wrote: post #2

I love seeing how people tackle these sorts of problems. Here, the trade-off is duplicate messages reaching a client. I know others have tried a no-"ack" approach. I'm still working in the land of RabbitMQ solving all my business needs and I like it. I'd love to see some numbers, reasons for decisions made, and suggested best practices for this solution other than "manual de-dupe."

Some of the background behind our decisions are on our blog post http://word.bitly.com/post/33232969144/nsq In terms of manual de-duping, we strive internally for idempotent message processing so it's fairly irrelevant, but to handle cases where it matters, all of our messages have unique id's added to them outside of NSQ. The actual cases where messages are handled multiple times is limited to when a client disappea…

When people say "billions of messages per day" do they realize that 1 billion messages =~ 11500 msgs/sec ? RabbitMQ as well as any self respecting message broker is able to handle that on a single machine