NSQ is my go-to queue whenever I'm working on any kind of service, even more so when that service might be standalone. Consider deploying a service on an isolated box which frequently pushes data into the cloud. Connectivity to the cloud might be intermittent, so you just store every work item to the locally running nsq and you're protected against network outages as well as your data-producing process crashing. Since NSQ is so easy to spin up and easy to interact with (Publish works via plain HTTP), it's not a lot of overhead to add it to your process.
Scaling NSQ to 750B Messages (2016)
11–15 of 15 posts
Re: Scaling NSQ to 750B Messages (2016)
#12Earlier quoted context omitted.
What pushed you to move to Kafka?
Reliability, scalability and tooling (in that order). Basically Kafka provides what NSQ is lacking: https://nsq.io/overview/features_and_guarantees.html Messages are durable (across nodes, not just persisted to the filesystem). Messages are delivered in-order (for a partition). Consumers know all partitions of a topic from startup, it's not eventually consistent. NSQ has been great honestly, it was just designed with…
Re: Scaling NSQ to 750B Messages (2016)
#13Earlier quoted context omitted.
Reliability, scalability and tooling (in that order). Basically Kafka provides what NSQ is lacking: https://nsq.io/overview/features_and_guarantees.html Messages are durable (across nodes, not just persisted to the filesystem). Messages are delivered in-order (for a partition). Consumers know all partitions of a topic from startup, it's not eventually consistent. NSQ has been great honestly, it was just designed with…
Was language a deciding factor? NSQ being Go while Kafka is Java.
Re: Scaling NSQ to 750B Messages (2016)
#14Been using nats for a few projects lately and really enjoying it. One of the many great golang projects. Resource utilization compared to kafka is tiny. Re: msq: Looks like segment are using kafka these days despite the praise for nsq in this dated article.
Out of curiosity, are you using the durable, NATS-Streaming server, or just the ephemeral NATS-server? I’d be interested to hear more about the performance differences you see.
When we put it together a few years ago the cloistering wasn't very mature and the streaming version with persistence wasn't completed. Would look into this over kafka if only for the simplicity in management and track record of go for performance and reliability.
Re: Scaling NSQ to 750B Messages (2016)
#15Earlier quoted context omitted.
Was language a deciding factor? NSQ being Go while Kafka is Java.
Not at all. We mostly use Go internally for backend services, so if anything that was a point for NSQ.