[deleted]
NSQ – A realtime distributed messaging platform designed to operate at scale
11–20 of 57 posts
Re: NSQ – A realtime distributed messaging platform designed to operate at scale
#12Earlier quoted context omitted.
I think the biggest question is, do you think its feasible to directly start with Kafka instead of NSQ or does Kafka just require a much stronger/larger team to operate than NSQ?
If you are starting off just use what you/the team is comfortable with. I mean, unless you think you will need capacity for over 25,000 messages per second.
25k messages total, or per second?
Re: NSQ – A realtime distributed messaging platform designed to operate at scale
#13Re: NSQ – A realtime distributed messaging platform designed to operate at scale
#14Segment is probably the biggest NSQ user right now, and they're moving to Kafka - any employees want to weigh in? :)
Engineer @ Segment
NSQ has served us pretty well but long term persistence has been a massive concern to us. If any of our NSQ nodes go down it's a big problem.
Kafka has been far more complicated to operate in production and developing against it requires more thought than NSQ (where you can just consume from a topic/channel, ack the message and be done). More to that, if you want more capacity you can just scale up your services and be done. With Kafka we had to plan how many partitions we needed and autoscaling has become a bit trickier.
We now have critical services running against Kafka and started moving our whole pipeline to it as well. It's a slow process but we're getting there.
We've had to build some tooling to operate Kafka and ramp up everyone else on how to use it. To be fair, we've also had to build tooling for NSQ, specifically nsq-lookup to allow us to scale up.
We have an nsq-go library that we use in production along with some tooling: https://github.com/segmentio/nsq-go
Re: NSQ – A realtime distributed messaging platform designed to operate at scale
#15what are the typical use cases for NSQ ?
A bit of an older article but still useful: https://segment.com/blog/scaling-nsq/
Re: NSQ – A realtime distributed messaging platform designed to operate at scale
#16Earlier quoted context omitted.
Kafka and NSQ have widely variant promises around things like durability, order, etc. In most use cases you can get NSQ like behavior out of Kafka and the inverse isn't true. Kafka's performance and added gaurantees come at the expense of being harder to operate.
Can someone summarize the promises? Specifically, would NSQ work well as an easier-to-operate, Kafka alternative, or are there low-throughput use cases it's just not suitable for?
I think it's less about throughput and more about durability guarantees. Kafka producers can specify the number of 'acks' (brokers which have written the message to disk) when they produce a message and the request will only return successfully if that can be done.
Re: NSQ – A realtime distributed messaging platform designed to operate at scale
#17Segment is probably the biggest NSQ user right now, and they're moving to Kafka - any employees want to weigh in? :)
Engineer @ Segment NSQ has served us pretty well but long term persistence has been a massive concern to us. If any of our NSQ nodes go down it's a big problem. Kafka has been far more complicated to operate in production and developing against it requires more thought than NSQ (where you can just consume from a topic/channel, ack the message and be done). More to that, if you want more capacity you can just scale up…
Out of interest, what kind of tooling did you build for Kafka?
Re: NSQ – A realtime distributed messaging platform designed to operate at scale
#18Earlier quoted context omitted.
Kafka and NSQ have widely variant promises around things like durability, order, etc. In most use cases you can get NSQ like behavior out of Kafka and the inverse isn't true. Kafka's performance and added gaurantees come at the expense of being harder to operate.
Can someone summarize the promises? Specifically, would NSQ work well as an easier-to-operate, Kafka alternative, or are there low-throughput use cases it's just not suitable for?
NSQ has been (IMO) far far easier to operate than Kafka has been. With Kafka you need a Zookeeper cluster in addition to your Kafka brokers. Not to mention developing against NSQ is pretty simple whereas Kafka you need to think about partitions and offsets.
If you're worried about data loss, Kakfa can be what you're looking for (but takes a lot to learn how to operate it correctly).
Re: NSQ – A realtime distributed messaging platform designed to operate at scale
#19Earlier quoted context omitted.
Engineer @ Segment NSQ has served us pretty well but long term persistence has been a massive concern to us. If any of our NSQ nodes go down it's a big problem. Kafka has been far more complicated to operate in production and developing against it requires more thought than NSQ (where you can just consume from a topic/channel, ack the message and be done). More to that, if you want more capacity you can just scale up…
Out of interest, what kind of tooling did you build for Kafka?
We started deploying our Kafka cluster as a set of N EC2 instances but we started running into a bunch of issues (rolling the cluster, rolling an instance without moving partitions around, moving partitions around, etc...)
Now we run Kafka through ECS and wrote some tooling to manage rolling the cluster and replacing brokers. krollout(1) (currently private) basically prevents partitions from becoming unavailable while rolling.
Now that multiple teams are using Kakfa we started exploring how to scale up. Each team may have different requirements and isolation can become an issue. Likely more tooling will need to be built around this.
Re: NSQ – A realtime distributed messaging platform designed to operate at scale
#20Segment is probably the biggest NSQ user right now, and they're moving to Kafka - any employees want to weigh in? :)
Engineer @ Segment NSQ has served us pretty well but long term persistence has been a massive concern to us. If any of our NSQ nodes go down it's a big problem. Kafka has been far more complicated to operate in production and developing against it requires more thought than NSQ (where you can just consume from a topic/channel, ack the message and be done). More to that, if you want more capacity you can just scale up…
Have you ever looked at any proprietary solutions like Google's PubSub? We're running on PubSub for over year now and outside of some unplanned downtimes it's scaling very well. But as we're looking to branch out out of GCP we are looking at Kafka as an alternative.
Could you comment on particular problems and challenges that you ran into?
For the context, we're currently sending around 60k messages/sec and around 1k of them contains data larger than 10kb.