Scaling NSQ to 750B Messages (2016)

#1

Post by tepidandroid » Tue, Aug 13, 2019, 12:29 AM UTC

Scaling NSQ to 750B Messages (2016)
segment.com

Re: Scaling NSQ to 750B Messages (2016)

#2

Post by bruth » Tue, Aug 13, 2019, 1:04 AM UTC

Note this is from 2016

Re: Scaling NSQ to 750B Messages (2016)

#3

Post by tgtweak » Tue, Aug 13, 2019, 1:27 AM UTC

Been using nats for a few projects lately and really enjoying it. One of the many great golang projects. Resource utilization compared to kafka is tiny.

Re: msq: Looks like segment are using kafka these days despite the praise for nsq in this dated article.

Re: Scaling NSQ to 750B Messages (2016)

#4

Post by tepidandroid » Tue, Aug 13, 2019, 1:29 AM UTC

bruth wrote: post #2

Note this is from 2016

thanks, I updated the title

Re: Scaling NSQ to 750B Messages (2016)

#5

Post by jonathanoliver » Tue, Aug 13, 2019, 1:35 AM UTC

tgtweak wrote: post #3

Been using nats for a few projects lately and really enjoying it. One of the many great golang projects. Resource utilization compared to kafka is tiny. Re: msq: Looks like segment are using kafka these days despite the praise for nsq in this dated article.

As of 2017 they were migrating everything over to Kafka: https://news.ycombinator.com/item?id=14451148

Re: Scaling NSQ to 750B Messages (2016)

#6

Post by quxbar » Tue, Aug 13, 2019, 3:39 AM UTC

Jehiah and Matt (the original creators of NSQ) are really talented engineers, I'm glad their work is getting some love.

Re: Scaling NSQ to 750B Messages (2016)

#7

Post by wcdolphin » Tue, Aug 13, 2019, 3:55 AM UTC

tgtweak wrote: post #3

Been using nats for a few projects lately and really enjoying it. One of the many great golang projects. Resource utilization compared to kafka is tiny. Re: msq: Looks like segment are using kafka these days despite the praise for nsq in this dated article.

Out of curiosity, are you using the durable, NATS-Streaming server, or just the ephemeral NATS-server? I’d be interested to hear more about the performance differences you see.

Re: Scaling NSQ to 750B Messages (2016)

#8

Post by caust1c » Tue, Aug 13, 2019, 4:35 AM UTC

I'm working at Segment now as a Software Engineer in Infrastructure. We just went through a big effort to remove NSQ (in favor of Kafka). Happy to answer any questions.

Re: Scaling NSQ to 750B Messages (2016)

#9

Post by coulix » Tue, Aug 13, 2019, 4:44 AM UTC

caust1c wrote: post #8

I'm working at Segment now as a Software Engineer in Infrastructure. We just went through a big effort to remove NSQ (in favor of Kafka). Happy to answer any questions.

What pushed you to move to Kafka?

Re: Scaling NSQ to 750B Messages (2016)

#10

Post by caust1c » Tue, Aug 13, 2019, 4:57 AM UTC

coulix wrote: post #9

caust1c wrote: post #8

I'm working at Segment now as a Software Engineer in Infrastructure. We just went through a big effort to remove NSQ (in favor of Kafka). Happy to answer any questions.

What pushed you to move to Kafka?

Reliability, scalability and tooling (in that order).

Basically Kafka provides what NSQ is lacking: https://nsq.io/overview/features_and_guarantees.html

Messages are durable (across nodes, not just persisted to the filesystem). Messages are delivered in-order (for a partition). Consumers know all partitions of a topic from startup, it's not eventually consistent.

NSQ has been great honestly, it was just designed with different use cases in mind.