Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

NSQ – A realtime distributed messaging platform designed to operate at scale

github.com

Post reply on HN
Print view View original

51–57 of 57 posts

Re: NSQ – A realtime distributed messaging platform designed to operate at scale

#51

Post by doh »

derekperkins wrote: post #50
doh wrote: post #49

Earlier quoted context omitted.

We used RabbitMQ extensively for almost two years but the problems we were encountering along the way weren't worth it. We ended up talking to the dev team too often to solve catastrophic issues that took down our whole production for hours. We reconsidered using it again for a synchronous RPC communication as we were replacing gRPC, but ended up going with nats.io instead. It does have less fearures but we are able…

Why were you replacing gRPC?

gRPC is a great, but has a ton of small problems including a catastrophic documentation (in some cases we had to read byte code to figure out what to do).

The biggest issue for us was however was that there is no middle server that could handle an route connections to available workers. We were using haproxy which worked ok but far from great. It was very hard to figure out how many servers need to run at any given point and thus a ton of our requests were ending up with UNAVAILABLE response.

Essentially what we needed is a synchronous RPC over PubSub which gRPC doesn't offer.

Re: NSQ – A realtime distributed messaging platform designed to operate at scale

#52

Post by StavrosK »

be_erik wrote: post #41
StavrosK wrote: post #32

Earlier quoted context omitted.

Yeah, at-least-once is definitely the way to go, sounds like you've got a good architecture around that. Another nice thing I've heard with Kafka is that it can store all messages since the beginning of time (if you want) and you can replay them to retrieve all your MQ-related state. Does NSQ do that, do you know?

NSQ does nothing like this out of the box. You could create a channel for each topic that accumulates messages, but those messages would not be distributed across the cluster and after consuming them once they would be effectively gone. If that feature is a hard requirement NSQ wouldn't fit your use case.

I see, thank you. I'm a bit confused by "after consuming them once they would be effectively gone". Surely NSQ supports more than one subscribers, and won't remove messages after they've been delivered to a single client?

Re: NSQ – A realtime distributed messaging platform designed to operate at scale

#53

Post by StavrosK »

kasey_junk wrote: post #34
StavrosK wrote: post #10

Earlier quoted context omitted.

Can someone summarize the promises? Specifically, would NSQ work well as an easier-to-operate, Kafka alternative, or are there low-throughput use cases it's just not suitable for?

People tend to talk about Kafka in the same conversations as messaging systems because it can support messaging use cases, but that leads to a mental model of what Kafka is (and conversely what messaging systems tend to be) that is incorrect. Its better to think of Kafka as a distributed log service than a messaging broker. When described this way, don't think log as in "the things humans look at to debug application…

That's a great comparison, thank you. Basically, the "log structure vs in-memory queue" distinction at the start crystallizes the differences, thanks for the reply.

Re: NSQ – A realtime distributed messaging platform designed to operate at scale

#54

Post by kasey_junk »

StavrosK wrote: post #52
be_erik wrote: post #41

Earlier quoted context omitted.

NSQ does nothing like this out of the box. You could create a channel for each topic that accumulates messages, but those messages would not be distributed across the cluster and after consuming them once they would be effectively gone. If that feature is a hard requirement NSQ wouldn't fit your use case.

I see, thank you. I'm a bit confused by "after consuming them once they would be effectively gone". Surely NSQ supports more than one subscribers, and won't remove messages after they've been delivered to a single client?

It supports more than one subscriber as long as you have a separate channel for each. But if the channel gets created after some messages have been delivered, they will not be delivered to the new channel, only the messages after that.

To do replay with nsq (one way is) you'd attach their file consumer to its own channel as the first operation on a topic. Then in application code you'd need to read from that file whenever you need to do a replay (and you'd need to know when/where to read from/to).

Re: NSQ – A realtime distributed messaging platform designed to operate at scale

#55

Post by mafro »

ngrilly wrote: post #44
mafro wrote: post #43

Earlier quoted context omitted.

RabbitMQ does not persist messages to disk. A common mistake with RMQ is treating it like something of a database. You would need to run RMQ in HA mode with multiple brokers to have any chance of not losing data.

What about this: https://www.rabbitmq.com/persistence-conf.html ?

I've never seen that option before - I stand corrected. I'd be interested to see how having it switched on affects performance. Also I expect there are still few guarantees about data loss in the event of failure.

Re: NSQ – A realtime distributed messaging platform designed to operate at scale

#56

Post by ngrilly »

mafro wrote: post #55
ngrilly wrote: post #44

Earlier quoted context omitted.

What about this: https://www.rabbitmq.com/persistence-conf.html ?

I've never seen that option before - I stand corrected. I'd be interested to see how having it switched on affects performance. Also I expect there are still few guarantees about data loss in the event of failure.

Here is some details:

https://www.rabbitmq.com/confirms.html#publisher-confirms-la...

To fully guarantee that a message won't be lost, it looks like you need to declare the queue as durable + to mark your message as persistent + use publisher confirm. And it looks like this costs you several hundreds milliseconds of latency.

Re: NSQ – A realtime distributed messaging platform designed to operate at scale

#57

Post by StavrosK »

kasey_junk wrote: post #54
StavrosK wrote: post #52

Earlier quoted context omitted.

I see, thank you. I'm a bit confused by "after consuming them once they would be effectively gone". Surely NSQ supports more than one subscribers, and won't remove messages after they've been delivered to a single client?

It supports more than one subscriber as long as you have a separate channel for each. But if the channel gets created after some messages have been delivered, they will not be delivered to the new channel, only the messages after that. To do replay with nsq (one way is) you'd attach their file consumer to its own channel as the first operation on a topic. Then in application code you'd need to read from that file whe…

That makes sense, thank you.
Post reply on HN