Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

RabbitMQ vs. Kafka – An Architect’s Dilemma (Part 1)

eranstiller.com

Post reply on HN
Print view View original

131–140 of 173 posts

Re: RabbitMQ vs. Kafka – An Architect’s Dilemma (Part 1)

#131

Post by ssd532 »

jpgvm wrote: post #97
mianos wrote: post #96

Earlier quoted context omitted.

While pulsar on paper seems a superior solution, in my experience it is very still very immature and very buggy. I really want to use it over kafka but I would not bet my business on it. I am not a fan of Kafka, it's kinda old, and the code is a bit messy, a lot of the once only semantic problems 100% solved by Pulser are sorta kinda in Kfaka these days. All the newer stuff like built in RAFT makes it competitive wit…

I have run all 3 at big scale. Kafka is still great as long as everyone using it understands it's a stream, not a queue and using it like a queue is going to get them burnt. I don't touch RabbitMQ with a 30ft pole anymore, too many lost days or nights to split brains and other chaos. Pulsar has mostly replaced Kafka for me because I don't need to worry about people coming along and changing requirements after the fac…

I am contemplating this exact topic for my project at this moment. It would be great if you can briefly explain what, as per your understanding, stream vs queue semantic are. I am studying it and got somewhat confusing discussions on the internet and in person forums.

Re: RabbitMQ vs. Kafka – An Architect’s Dilemma (Part 1)

#132

Post by usrusr »

onetimeuse92304 wrote: post #129

If you have a throughput problem you are most likely doing it wrong. If your knee jerk reaction is to scale your messaging up you may want to reconsider. Messaging systems are usually hard to scale up and always very costly to do so compared to the amount of data they are transferring. The simplest thing you can do is to realise WHY you are using messages. Messages are there to trigger a process. Usually, you don't n…

If your use case can stomach the added failure modes this implies, yeah, that's what you can do.

Re: RabbitMQ vs. Kafka – An Architect’s Dilemma (Part 1)

#133

Post by doctor_eval »

zerbinxx wrote: post #107
pnpnp wrote: post #103

Earlier quoted context omitted.

I think they call that part of NATS “Jetstream” if I’m not mistaken. I haven’t used it, but I believe it has some form of message persistence. I have used it mostly for message-first services, and found subject-based messaging a breath of fresh air to decouple services. You can do the same thing with RabbitMQ topic exchanges, but it requires quite a bit more hand-waving.

Jetstream does indeed have message persistence: I can issue queries like “get messages on topic since 5 minutes ago” - I do this a ton. However, that seems to be the extent of the storage/query API that it exposes for historical messages. I’m quite a big fan, and would recommend it with the caveat that Jetstream is considerably more complex than simple nats and I get the feeling I’m barely scratching the surface with…

I’m really interested in a Kafka like message broker in the Go ecosystem and look forward to checking it out for whatever my next project ends up being.

Re: RabbitMQ vs. Kafka – An Architect’s Dilemma (Part 1)

#134

Post by FridgeSeal »

whalesalad wrote: post #30
FridgeSeal wrote: post #22

I’m personally a fan of Kafka. I think the design of persisting the messages, and tracking offsets for progress instead of message acknowledgments is a much cleaner and more versatile design. You can get all the same advantages of message acknowledgments, but now you can also replay queues, let different applications use the messages (handy for cross cutting event/notification systems) and you get better scaling prop…

> You can get all the same advantages of message acknowledgments, but now you can also replay queues with rmq you can reject/nack a message and have it put back on the queue. rmq is not well suited for long term historical retention inside queues a-la kafka's logs but it is possible to do. > let different applications use the messages (handy for cross cutting event/notification systems) rmq also does a publish once a…

> with rmq you can reject/nack a message and have it put back on the queue

I know other systems have semi-similar mechanisms, however most of them retain the “someone is the sole owner of this message” style design, which I think is fundamentally limiting. Owning application dies, is it acked or not? Acks but never gets around to putting it back on the queue? Who takes priority if 2 separate applications wish to watch the same stream of events?

I think Kafka’s “nobody owns it, acks are consumer group level” give you the same advantages for the application itself, without a number of the more difficult complications.

> rmq also does a publish once and fanout to multiple queues to support this

Which is probably fine for small volume or velocity topics, but is going to cause all sorts of load issues at higher scale.

Re: RabbitMQ vs. Kafka – An Architect’s Dilemma (Part 1)

#135

Post by waynesonfire »

ryanjshaw wrote: post #55
raducu wrote: post #26

Earlier quoted context omitted.

> who can't just now pluck messages off a queue to process The problem is you cannot mark individual messages as read, for a given consumer&partition you can only update the offset for a partition. If a certain message processing takes very long, all other messages in that partition will have to wait. Also, with kafka, the max read concurrency is equal to the number of partitions, for something like rabbitMq it is mu…

> The problem is you cannot mark individual messages as read, for a given consumer&partition you can only update the offset for a partition. Hence "smart clients". If you MUST process every message at least once, you will anyway be tracking messages individually on the client (e.g. a DB or file system plus logic for idempotent message processing) and thus disable auto-offset commits back to the cluster for your consu…

If you have idempotent messages, why can't you use auto offset committing?

Re: RabbitMQ vs. Kafka – An Architect’s Dilemma (Part 1)

#136

Post by protortyp »

jpgvm wrote: post #81

If you want features of RabbitMQ (specifically queue like behavior) but the scalability of Kafka then you probably want Apache Pulsar. To elaborate on that a bit the main things Pulsar gives you are: 1. Still underlying distributed stream based architecture, this is what makes it able to do Kafka like things. 2. Broker side management of subscription state which allow out of order acknowledgement, this means you can…

We also switched to Pulsar after running some benchmarks for our use cases. We use these services primarily as worker queues for image tasks that require low latency. And Pulsar turned out to have a 20x lower latency than Kafka in our setup.

Re: RabbitMQ vs. Kafka – An Architect’s Dilemma (Part 1)

#137

Post by Beached »

onetimeuse92304 wrote: post #129

If you have a throughput problem you are most likely doing it wrong. If your knee jerk reaction is to scale your messaging up you may want to reconsider. Messaging systems are usually hard to scale up and always very costly to do so compared to the amount of data they are transferring. The simplest thing you can do is to realise WHY you are using messages. Messages are there to trigger a process. Usually, you don't n…

1) some designs can't tolerate the producer sending messages at such a delay. 3) s3 is not cheap storage, it is significantly higher cost than most on prem solutions when talking about large scale storage. (pedabytes scale)

Re: RabbitMQ vs. Kafka – An Architect’s Dilemma (Part 1)

#138

Post by sgarland »

JohnMakin wrote: post #56
NovemberWhiskey wrote: post #40

Earlier quoted context omitted.

> I am not an expert in either and have only worked with Kafka. > I’ve managed a lot of rabbitmq deployments ... ?

You do not need to be an expert in something's internal workings to manage/monitor a deployment. Surely this does not need to be explained further.

> I think you have it backwards

You do need to be an expert when you start telling other people they’re wrong, though.

Re: RabbitMQ vs. Kafka – An Architect’s Dilemma (Part 1)

#140

Post by onetimeuse92304 »

usrusr wrote: post #132
onetimeuse92304 wrote: post #129

If you have a throughput problem you are most likely doing it wrong. If your knee jerk reaction is to scale your messaging up you may want to reconsider. Messaging systems are usually hard to scale up and always very costly to do so compared to the amount of data they are transferring. The simplest thing you can do is to realise WHY you are using messages. Messages are there to trigger a process. Usually, you don't n…

If your use case can stomach the added failure modes this implies, yeah, that's what you can do.

Everything has some cons, some failure modes.

All engineering is about knowing, understanding and making tradeoffs.

In my practice, I am happy if I can get rid of hard problems (my messaging platform being unable to process X messages per second) and replace them with relatively easier problems (my persistence might sometimes fail and then I can't send a message).

I would argue that distributed persistence solutions are usually more reliable than messaging platforms and also what is a very large throughput for a messsaging solution is usually nothing much for monsters that are engineered to take much larger volumes of data. And so, in my experience, reducing load to messaging and increasing load to persistence is net positive for the overall reliability.

Post reply on HN