NSQ: Realtime distributed message processing at scale (in Go) • Hacker News Board

Re: NSQ: Realtime distributed message processing at scale (in Go)

#21

Post by onetwothreefour » Tue, Oct 09, 2012, 7:43 PM UTC

jlgreco wrote: post #9

bgentry wrote: post #8

This ensures that the only edge case that would result in message loss is an unclean shutdown of an nsqd process. In that case, any messages that were in memory (or any buffered writes not flushed to disk) would be lost. I don't understand how you can call it a message delivery "guarantee" when you're susceptible to losing messages when a node dies. One solution is to stand up redundant nsqd pairs (on separate hosts)…

How would you design a system that didn't have this problem at the individual node level?

This is a solved problem in most messaging protocols.

Re: NSQ: Realtime distributed message processing at scale (in Go)

#22

Post by dclusin » Tue, Oct 09, 2012, 7:48 PM UTC

jlgreco wrote: post #16

dclusin wrote: post #10

Earlier quoted context omitted.

Sequence numbers on messages sent by producers coupled with some sort of persistence by the sender. This way when a gap is detected some sort of resend protocol can be implemented to achieve the delivery guarantee.

So the sender just keeps trying until it gets an ACK. Fair enough, though I imagine you'd want to run multiple nodes even with this.

Correct. Reliability and Availability are two separate but related goals.

Re: NSQ: Realtime distributed message processing at scale (in Go)

#23

Post by jasonmoo » Tue, Oct 09, 2012, 8:08 PM UTC

Hey it's awesome to see you guys tackling a larger project in Go. I've been writing some smaller services/tools with it in my day job, and I like it a lot.

Thanks for sharing your work.

Re: NSQ: Realtime distributed message processing at scale (in Go)

#24

Post by andreypopp » Tue, Oct 09, 2012, 8:16 PM UTC

Interesting if bitly guys had a look at beanstalkd before implementing nsqd.

Re: NSQ: Realtime distributed message processing at scale (in Go)

#25

Post by matticakes » Tue, Oct 09, 2012, 8:49 PM UTC

andreypopp wrote: post #24

Interesting if bitly guys had a look at beanstalkd before implementing nsqd.

beanstalkd is a great project and it certainly inspired some of our thinking. I think of beanstalkd as a more fully functional alternative to simplequeue (what we built/used prior to NSQ - more details in our blog post http://word.bitly.com/post/33232969144/nsq).

Some of the goals of NSQ transcended just replacing our specific daemon that buffered and delivered messages (most importantly the interactions with the lookup service). Because of that, we felt that owning that piece would make it easier to achieve those goals.

Additionally, one of the most important properties of nsqd (the queue component of NSQ) is that data is pushed to the client rather than polled (like in beanstalkd).

Re: NSQ: Realtime distributed message processing at scale (in Go)

#26

Post by interg12 » Tue, Oct 09, 2012, 9:02 PM UTC

Anyone at Disqus care to weigh in on this?

Re: NSQ: Realtime distributed message processing at scale (in Go)

#27

Post by overbroad » Tue, Oct 09, 2012, 9:02 PM UTC

I'm too lazy to install Go just to try this (I would need to, right?) but reading through the docs this looks far more appealing to me than tent.io. Nice work, at least with respect to the general design. Flexible.

Re: NSQ: Realtime distributed message processing at scale (in Go)

#28

Post by old_sound » Tue, Oct 09, 2012, 9:18 PM UTC

matticakes wrote: post #12

jallmann wrote: post #4

Was something like MQTT investigated prior to re-inventing the wheel with NSQ? From a cursory examination, I don't see any major advantages other than nice UI tools. The wire protocol seems more complicated -- why distinct protocols for producers and consumers? Linebreak-delimited headers are error-prone and ought to be banished. Why transmit the hostname with sub requests -- is another system requesting jobs on beha…

one of the developers here... The protocols that exist in NSQ now are designed to be the simplest implementation that worked. You've correctly pointed out some of the issues. At this stage, the distinction of producer vs consumer was mostly so that you could publish at all without having to use the HTTP interface. For our use cases, in particular taking advantage of the /mput endpoint, we aren't even using the TCP ba…

I'm really curious to know why would you build this tool instead of using, say, RabbitMQ?

DISCLAIMER: I wrote the RabbitMQ in Action book.

Re: NSQ: Realtime distributed message processing at scale (in Go)

#29

Post by matticakes » Tue, Oct 09, 2012, 9:37 PM UTC

old_sound wrote: post #28

matticakes wrote: post #12

Earlier quoted context omitted.

one of the developers here... The protocols that exist in NSQ now are designed to be the simplest implementation that worked. You've correctly pointed out some of the issues. At this stage, the distinction of producer vs consumer was mostly so that you could publish at all without having to use the HTTP interface. For our use cases, in particular taking advantage of the /mput endpoint, we aren't even using the TCP ba…

I'm really curious to know why would you build this tool instead of using, say, RabbitMQ? DISCLAIMER: I wrote the RabbitMQ in Action book.

We looked into AMQP based solutions. Our understanding is that slaving, master-slave failover, or other strategies are used to mitigate the fact that there is a broker responsible for a given stream of messages.

We wanted a brokerless, decentralized, and distributed system. NSQ has no broker or SPOF so it is able to route around failures.

That said, I think RabbitMQ is a good tool depending on your requirements. I can imagine a broker proving useful in situations where you may want strict ordering or no-duplication. Those were tradeoffs we were willing to make for fault tolerance and availability.

Also, given the fact that we were already operating infrastructure built on simplequeue (which is also distributed and brokerless) we found it more appealing to evolve rather than replace.

Re: NSQ: Realtime distributed message processing at scale (in Go)

#30

Post by jallmann » Tue, Oct 09, 2012, 9:59 PM UTC

matticakes wrote: post #29

old_sound wrote: post #28

Earlier quoted context omitted.

I'm really curious to know why would you build this tool instead of using, say, RabbitMQ? DISCLAIMER: I wrote the RabbitMQ in Action book.

We looked into AMQP based solutions. Our understanding is that slaving, master-slave failover, or other strategies are used to mitigate the fact that there is a broker responsible for a given stream of messages. We wanted a brokerless, decentralized, and distributed system. NSQ has no broker or SPOF so it is able to route around failures. That said, I think RabbitMQ is a good tool depending on your requirements. I ca…

A broker is an implementation detail; there's nothing stopping you from going peer-to-peer with $protocol and having publishers manage their own subscribers. Some sort of registry is still needed though; nslookupd in your case.

There is a bit of an impedance mismatch since now you have to go out-of-band to fetch information. Not saying it's a bad thing; that's essentially also the role of DNS and LDAP, among others, so it is a fairly common pattern.