Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

NSQ – A realtime distributed messaging platform designed to operate at scale

github.com

Post reply on HN
Print view View original

41–50 of 57 posts

Re: NSQ – A realtime distributed messaging platform designed to operate at scale

#41

Post by be_erik »

StavrosK wrote: post #32
be_erik wrote: post #29

Earlier quoted context omitted.

Absolutely, but you can't rely on it to be persistent like Kafka. We use it extensively and are incredibly happy with it, but we follow best practices around not putting state in messages, making changes idempotent, and ensuring that we can always replay a message if needed. We've yet to lose any messages in production, but it could happen and we're okay with the tradeoffs between that and the operational complexity…

Yeah, at-least-once is definitely the way to go, sounds like you've got a good architecture around that. Another nice thing I've heard with Kafka is that it can store all messages since the beginning of time (if you want) and you can replay them to retrieve all your MQ-related state. Does NSQ do that, do you know?

NSQ does nothing like this out of the box. You could create a channel for each topic that accumulates messages, but those messages would not be distributed across the cluster and after consuming them once they would be effectively gone. If that feature is a hard requirement NSQ wouldn't fit your use case.

Re: NSQ – A realtime distributed messaging platform designed to operate at scale

#42

Post by ngrilly »

TheHydroImpulse wrote: post #23
doh wrote: post #20

Earlier quoted context omitted.

Have you ever looked at any proprietary solutions like Google's PubSub? We're running on PubSub for over year now and outside of some unplanned downtimes it's scaling very well. But as we're looking to branch out out of GCP we are looking at Kafka as an alternative. Could you comment on particular problems and challenges that you ran into? For the context, we're currently sending around 60k messages/sec and around 1k…

The biggest issue with PubSub and Amazon's alternative is the cost. Being capped at a per-message cost would be a no go. If you can get away with using PubSub or the like it would be far easier than to manage your own Kafka deployment (correctly). If data loss is unacceptable then Kafka is basically the only open-source solution that is known for not losing data (if done correctly of course). NSQ was great but lacked…

> If data loss is unacceptable then Kafka is basically the only open-source solution that is known for not losing data

What about RabbitMQ?

Re: NSQ – A realtime distributed messaging platform designed to operate at scale

#43

Post by mafro »

ngrilly wrote: post #42
TheHydroImpulse wrote: post #23

Earlier quoted context omitted.

The biggest issue with PubSub and Amazon's alternative is the cost. Being capped at a per-message cost would be a no go. If you can get away with using PubSub or the like it would be far easier than to manage your own Kafka deployment (correctly). If data loss is unacceptable then Kafka is basically the only open-source solution that is known for not losing data (if done correctly of course). NSQ was great but lacked…

> If data loss is unacceptable then Kafka is basically the only open-source solution that is known for not losing data What about RabbitMQ?

RabbitMQ does not persist messages to disk. A common mistake with RMQ is treating it like something of a database.

You would need to run RMQ in HA mode with multiple brokers to have any chance of not losing data.

Re: NSQ – A realtime distributed messaging platform designed to operate at scale

#44

Post by ngrilly »

mafro wrote: post #43
ngrilly wrote: post #42

Earlier quoted context omitted.

> If data loss is unacceptable then Kafka is basically the only open-source solution that is known for not losing data What about RabbitMQ?

RabbitMQ does not persist messages to disk. A common mistake with RMQ is treating it like something of a database. You would need to run RMQ in HA mode with multiple brokers to have any chance of not losing data.

What about this: https://www.rabbitmq.com/persistence-conf.html ?

Re: NSQ – A realtime distributed messaging platform designed to operate at scale

#45

Post by rafaeljesus »

doh wrote: post #27
molszanski wrote: post #25

Earlier quoted context omitted.

Do you have anythings to say about nats.io?

We're using nats for synchronous communication, sending around 10k messages each second through it. Must say that the stability is great, even with larger payloads (over 10MB in size). We're running it in production for couple of weeks now and haven't had any issues. The main limitation is that there is no federation and massive clustering. You can have a pretty robust cluster, but each node can only forward once, wh…

You mean synchronous communication using request-reply/rpc like?

Re: NSQ – A realtime distributed messaging platform designed to operate at scale

#46

Post by rafaeljesus »

be_erik wrote: post #30
Roritharr wrote: post #5

Earlier quoted context omitted.

I think the biggest question is, do you think its feasible to directly start with Kafka instead of NSQ or does Kafka just require a much stronger/larger team to operate than NSQ?

We debated the same question and went with NSQ for now. We might need some of the guarantees that Kafka makes in the longterm or for some specific use cases, but for a no thrills distributed messaging platform that is incredibly simple to operate at scale, NSQ is pretty fantastic. Building client libraries is also a joy. I blogged about it a bit here: https://product.reverb.com/how-to-write-an-nsq-consumer-in-g...

Nice one! Just as a info, I wrapped producer/consumer in a pkg https://github.com/rafaeljesus/nsq-event-bus also it exposes request-reply/rpc like

Re: NSQ – A realtime distributed messaging platform designed to operate at scale

#47

Post by doh »

rafaeljesus wrote: post #45
doh wrote: post #27

Earlier quoted context omitted.

We're using nats for synchronous communication, sending around 10k messages each second through it. Must say that the stability is great, even with larger payloads (over 10MB in size). We're running it in production for couple of weeks now and haven't had any issues. The main limitation is that there is no federation and massive clustering. You can have a pretty robust cluster, but each node can only forward once, wh…

You mean synchronous communication using request-reply/rpc like?

Correct

Re: NSQ – A realtime distributed messaging platform designed to operate at scale

#48

Post by user454545 »

TheHydroImpulse wrote: post #23
doh wrote: post #20

Earlier quoted context omitted.

Have you ever looked at any proprietary solutions like Google's PubSub? We're running on PubSub for over year now and outside of some unplanned downtimes it's scaling very well. But as we're looking to branch out out of GCP we are looking at Kafka as an alternative. Could you comment on particular problems and challenges that you ran into? For the context, we're currently sending around 60k messages/sec and around 1k…

The biggest issue with PubSub and Amazon's alternative is the cost. Being capped at a per-message cost would be a no go. If you can get away with using PubSub or the like it would be far easier than to manage your own Kafka deployment (correctly). If data loss is unacceptable then Kafka is basically the only open-source solution that is known for not losing data (if done correctly of course). NSQ was great but lacked…

It's not marketed as a messaging platform, but it sounds like Apache NiFi [1] may fulfill some of your needs, with many of the specialized tooling you described, already built in. NiFi is very tunable to support differing needs ("loss tolerance vs. guaranteed deliveries", "low latency vs. high throughput", ...). It is built to scale, it includes enterprise security features, and you can track every piece of data ever sent through it (if you want to.) It includes an outstanding web-based GUI, where you can immediately change the settings on all of your distributed nodes, through a simple and complete interface. It features an extension interface, but it contains many battle-tested commonly-used plugins (Kafka, HTTP(s), SSH, HDFS, ...) so that you can gradually integrate it into your environment.

NiFi has come up a few times on HN, but I really don't think it gets the attention it deserves --- I don't know how it would perform against Kafka, NATS, NSQ, *MQ, or other messaging platforms, and unfortunately, I don't have any metrics to share. But when I see that many users are taking these messaging platforms, and building additional tooling/processes to meet needs that are already built into NiFi, I think it shines as a very competitive open-source option.

@TheHydroImpulse: Thank you for sharing your insights in this post, and explaining why your organization made these selections. Have you considered or evaluated NiFi?

[1] https://nifi.apache.org/

(Disclaimer: All posts are my own, and not sponsored by the Department of Defense.)

Re: NSQ – A realtime distributed messaging platform designed to operate at scale

#49

Post by doh »

ngrilly wrote: post #42
TheHydroImpulse wrote: post #23

Earlier quoted context omitted.

The biggest issue with PubSub and Amazon's alternative is the cost. Being capped at a per-message cost would be a no go. If you can get away with using PubSub or the like it would be far easier than to manage your own Kafka deployment (correctly). If data loss is unacceptable then Kafka is basically the only open-source solution that is known for not losing data (if done correctly of course). NSQ was great but lacked…

> If data loss is unacceptable then Kafka is basically the only open-source solution that is known for not losing data What about RabbitMQ?

We used RabbitMQ extensively for almost two years but the problems we were encountering along the way weren't worth it. We ended up talking to the dev team too often to solve catastrophic issues that took down our whole production for hours.

We reconsidered using it again for a synchronous RPC communication as we were replacing gRPC, but ended up going with nats.io instead. It does have less fearures but we are able to squeze much more juice on a smaller stack.

Re: NSQ – A realtime distributed messaging platform designed to operate at scale

#50

Post by derekperkins »

doh wrote: post #49
ngrilly wrote: post #42

Earlier quoted context omitted.

> If data loss is unacceptable then Kafka is basically the only open-source solution that is known for not losing data What about RabbitMQ?

We used RabbitMQ extensively for almost two years but the problems we were encountering along the way weren't worth it. We ended up talking to the dev team too often to solve catastrophic issues that took down our whole production for hours. We reconsidered using it again for a synchronous RPC communication as we were replacing gRPC, but ended up going with nats.io instead. It does have less fearures but we are able…

Why were you replacing gRPC?
Post reply on HN