Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

NSQ: Realtime distributed message processing at scale (in Go)

github.com

Post reply on HN
Print view View original

31–40 of 46 posts

Re: NSQ: Realtime distributed message processing at scale (in Go)

#31

Post by rabbitmq »

matticakes wrote: post #29
old_sound wrote: post #28

Earlier quoted context omitted.

I'm really curious to know why would you build this tool instead of using, say, RabbitMQ? DISCLAIMER: I wrote the RabbitMQ in Action book.

We looked into AMQP based solutions. Our understanding is that slaving, master-slave failover, or other strategies are used to mitigate the fact that there is a broker responsible for a given stream of messages. We wanted a brokerless, decentralized, and distributed system. NSQ has no broker or SPOF so it is able to route around failures. That said, I think RabbitMQ is a good tool depending on your requirements. I ca…

Master-slave etc are HA techniques for replicating state eg queues (or "channels" as nsq calls them?). "Decentralized and distributed" has nothing to do with state, master/slave or any of that; it's a topological property of your system. You can certainly build a decentralized and distributed system using a set of brokers, regardless of whether they support HA.

Re: NSQ: Realtime distributed message processing at scale (in Go)

#33

Post by matticakes »

rabbitmq wrote: post #31
matticakes wrote: post #29

Earlier quoted context omitted.

We looked into AMQP based solutions. Our understanding is that slaving, master-slave failover, or other strategies are used to mitigate the fact that there is a broker responsible for a given stream of messages. We wanted a brokerless, decentralized, and distributed system. NSQ has no broker or SPOF so it is able to route around failures. That said, I think RabbitMQ is a good tool depending on your requirements. I ca…

Master-slave etc are HA techniques for replicating state eg queues (or "channels" as nsq calls them?). "Decentralized and distributed" has nothing to do with state, master/slave or any of that; it's a topological property of your system. You can certainly build a decentralized and distributed system using a set of brokers, regardless of whether they support HA.

That's fair, you certainly can use rabbitmq in a distributed and decentralized fashion.

You bring up an important point though... What is important is the topology we're promoting and the platform we've built to support it. The actual queue is less important and it made sense for us to own that piece to achieve our goals.

Re: NSQ: Realtime distributed message processing at scale (in Go)

#34

Post by knodi »

andreypopp wrote: post #24

Interesting if bitly guys had a look at beanstalkd before implementing nsqd.

Beanstalkd is not stable it has a lot issues even in it current version. If you doing to be sending couple million messages per day beanstalkd is not for you.

Re: NSQ: Realtime distributed message processing at scale (in Go)

#35

Post by jeremyjh »

dikbrouwer wrote: post #32

How does this compare to ZeroMQ?

ZeroMQ is a library, it is embedded in the application code and can support a peer-to-peer messaging topology. It also has support for Broker patterns but you have to develop the Broker application yourself; you could build a solution such as this one using it, but the zmq library itself really handles lower-level concerns with a high degree of flexibility but no out-of-the-box functionality. zmq can be thought of as something as simple as an improved socket library but with support for async, storing & high-water marks when there are no attached consumers, various distribution strategies, queues and lots more.

Re: NSQ: Realtime distributed message processing at scale (in Go)

#36

Post by dikbrouwer »

jeremyjh wrote: post #35
dikbrouwer wrote: post #32

How does this compare to ZeroMQ?

ZeroMQ is a library, it is embedded in the application code and can support a peer-to-peer messaging topology. It also has support for Broker patterns but you have to develop the Broker application yourself; you could build a solution such as this one using it, but the zmq library itself really handles lower-level concerns with a high degree of flexibility but no out-of-the-box functionality. zmq can be thought of as…

Right, I guess my question more specifically is: ZeroMQ allows you to built applications like this in (seemingly) just a few lines of code. Did you guys consider building it on top of ZeroMQ, and if so, what made your decide to not use it? Nothing against what you've built btw, it looks great, but am just curious

Re: NSQ: Realtime distributed message processing at scale (in Go)

#37

Post by matticakes »

dikbrouwer wrote: post #36
jeremyjh wrote: post #35

Earlier quoted context omitted.

ZeroMQ is a library, it is embedded in the application code and can support a peer-to-peer messaging topology. It also has support for Broker patterns but you have to develop the Broker application yourself; you could build a solution such as this one using it, but the zmq library itself really handles lower-level concerns with a high degree of flexibility but no out-of-the-box functionality. zmq can be thought of as…

Right, I guess my question more specifically is: ZeroMQ allows you to built applications like this in (seemingly) just a few lines of code. Did you guys consider building it on top of ZeroMQ, and if so, what made your decide to not use it? Nothing against what you've built btw, it looks great, but am just curious

The truth is we wanted to use ZeroMQ initially. It would have been really awesome to have all that flexibility on the client side.

The ZeroMQ documentation is fantastic as well. It inspired and helped shape some of the design choices we made.

The further along we got from design to implementation it became obvious that it would be important to "own" the socket. Generally speaking, this is exactly what ZeroMQ prevents you from doing (and rightly so, it aims to abstract all of that away).

The choice to use Go had an impact here as well. Language features like channels and the breadth of the standard library made it really easy to translate our NSQ design into working code, offsetting the benefit of ZeroMQ's abstractions.

Re: NSQ: Realtime distributed message processing at scale (in Go)

#38

Post by jehiah »

overbroad wrote: post #27

I'm too lazy to install Go just to try this (I would need to, right?) but reading through the docs this looks far more appealing to me than tent.io. Nice work, at least with respect to the general design. Flexible.

Thanks for the reminder. Getting binary builds up is on our to-do list, so check back soon for those.

In the mean time, installing Go is pretty easy (you can use brew, or the official OSX package (assuming you are on OSX) http://golang.org/doc/install) and we've tried to leave clear steps for building NSQ here https://github.com/bitly/nsq/blob/master/INSTALLING.md (there really are very few dependencies other than go itself)

Re: NSQ: Realtime distributed message processing at scale (in Go)

#40

Post by overbroad »

jehiah wrote: post #38
overbroad wrote: post #27

I'm too lazy to install Go just to try this (I would need to, right?) but reading through the docs this looks far more appealing to me than tent.io. Nice work, at least with respect to the general design. Flexible.

Thanks for the reminder. Getting binary builds up is on our to-do list, so check back soon for those. In the mean time, installing Go is pretty easy (you can use brew, or the official OSX package (assuming you are on OSX) http://golang.org/doc/install ) and we've tried to leave clear steps for building NSQ here https://github.com/bitly/nsq/blob/master/INSTALLING.md (there really are very few dependencies other than g…

Cheers. I have installed Go before and messed around with it. I agree it's pretty good as far as being self-contained and straightforward. I just don't have the energy to keep chasing a moving target. For my purposes, I'm more interested in stuff that is stable and rarely changes.

If you can produce binaries for UNIX (Linux, BSD, etc), OSX and Windows, I'll be impressed. That's something I was interested in doing with Go (is it possible to cross-compile?) but never managed to learn.

Post reply on HN