Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

RabbitMQ vs. Kafka – An Architect’s Dilemma (Part 1)

eranstiller.com

Post reply on HN
Print view View original

101–110 of 173 posts

Re: RabbitMQ vs. Kafka – An Architect’s Dilemma (Part 1)

#101

Post by xupybd »

bob1029 wrote: post #79

I've seen Tibco Rendezvous used in manufacturing. ~300 megabytes per hour of raw log generated 24/7/365 by tools and control systems in a factory setting. Probably on the order of 10k+ participants in the pub/sub network. If you are running something like a factory where thousands of independent systems need to communicate in some way, this kind of tech starts to look like the only option. If you are orchestrating th…

I use NATS to create a secure pipe between buildings and the cloud. I don't need speed but I do need routing and I do need a security layer. Nats just works and took very little setup.

Re: RabbitMQ vs. Kafka – An Architect’s Dilemma (Part 1)

#102

Post by inkyoto »

kuchenbecker wrote: post #94
bob1029 wrote: post #79

I've seen Tibco Rendezvous used in manufacturing. ~300 megabytes per hour of raw log generated 24/7/365 by tools and control systems in a factory setting. Probably on the order of 10k+ participants in the pub/sub network. If you are running something like a factory where thousands of independent systems need to communicate in some way, this kind of tech starts to look like the only option. If you are orchestrating th…

Our prod cluster generates that about every minute at O(1M) qps. We JUST turned on remote Logs because until now Kafka didn't have capacity.

TIBCO Rendezvous is tech from 1998/99, millions of messages per second didn't exist at the time. Only NYSE and NASDAQ were capable of producing millions of events back then (still not by minute nor by second).

TIBCO Rendezvous was one of the first successful large scale, low latency and near real-time pub/sub implementations, and it had a very efficient, Avro like, wire level serialisation format that made messages very compact and efficient to deliver. It was very popular in finance, banking and manufacturing, and is all but legacy now.

Re: RabbitMQ vs. Kafka – An Architect’s Dilemma (Part 1)

#103

Post by pnpnp »

doctor_eval wrote: post #76
pnpnp wrote: post #69

Earlier quoted context omitted.

Just my 2c but for anyone unaware, you should check out NATS. It combines the best of both Kafka and RabbitMQ IMO.

I thought NATS didn't actually store messages, am I mistaken? Looking at Wikipedia ( https://en.wikipedia.org/wiki/NATS_Messaging ) I see that I'm technically right, it's JetStream that does the storage layer - but it's part of the NATS server. From memory, I really liked the philosophy of NATS but found the nomenclature confusing.

I think they call that part of NATS “Jetstream” if I’m not mistaken. I haven’t used it, but I believe it has some form of message persistence.

I have used it mostly for message-first services, and found subject-based messaging a breath of fresh air to decouple services. You can do the same thing with RabbitMQ topic exchanges, but it requires quite a bit more hand-waving.

Re: RabbitMQ vs. Kafka – An Architect’s Dilemma (Part 1)

#104

Post by inkyoto »

rafaelturk wrote: post #53
rafaelturk wrote: post #48

Nice post! RabbitMq is battle tested, exceptionally fast and low resources app. Capable of handling millions of transactions/second. RabbitMQ will handle vast majority of usecases. I'm puzzled why often startups, or even banks use Kafka, soley because is hype. Kafka on the order hand requires massive CPUs, Memory, often requiring its own K8S cluster just to be alive.

If your have a clean event-driven architecture, ie messages are completely agnostic and decoupled from one-another you don't need Kafka.

Event-driven architecture is an architectural principle, and Kafka, RabbitMQ/ActiveMQ, Pulsar, NATS and so forth are implementations that support the event-driven architectural principle. Yet, all of them range in a variety, complexity and extent of features they provide which may or may not be a good fit for a particular use case.

Traditional message brokers (RabbitMQ and similar) do support the event-driven architecture, yet the data they handle is ephemeral. Once a message has been processed, it is gone forever. Connecting a new raw data source is not an established practice and requires a technical «adapter» of sorts to be built. High concurrency levels is problematic for scenarios where the strict message processing ordering is required: the traditional message brokers do not handle it well in highly parallel scenarios out of the box.

Kafka and similar also support event-driven architectures, yet they allow the data to be processed multiple times – by existing (i.e. a data replay) and, most importantly, new or unknown at the time consumers (note: this is distinct from the data replay!). This is allows to plug existing data source(s) into a data streaming platform (Kafka) and incrementally add new data consumers and processors over the time with the datasets being available intact. This is an important distinction. Kafka and similar also improve on the strict processing order guarantee by allowing a message source (a Kafka topic) to be explicitly partitioned out and guaranteeing that the message order will be retained and enforced for a consumer group receiving messages from that partition.

To recap, traditional message brokers are a good fit for handling the ephemeral data, and data streaming platforms are a good fit for connecting data sources and allowing the data to be ingested multiple times. Both implement and support event-driven architectures in a variety of scenarios.

Re: RabbitMQ vs. Kafka – An Architect’s Dilemma (Part 1)

#105

Post by mianos »

jpgvm wrote: post #97
mianos wrote: post #96

Earlier quoted context omitted.

While pulsar on paper seems a superior solution, in my experience it is very still very immature and very buggy. I really want to use it over kafka but I would not bet my business on it. I am not a fan of Kafka, it's kinda old, and the code is a bit messy, a lot of the once only semantic problems 100% solved by Pulser are sorta kinda in Kfaka these days. All the newer stuff like built in RAFT makes it competitive wit…

I have run all 3 at big scale. Kafka is still great as long as everyone using it understands it's a stream, not a queue and using it like a queue is going to get them burnt. I don't touch RabbitMQ with a 30ft pole anymore, too many lost days or nights to split brains and other chaos. Pulsar has mostly replaced Kafka for me because I don't need to worry about people coming along and changing requirements after the fac…

Your comments seem to closely reflect mine. I'll have to take your advice and give it another go. It's everything all the others want to be. The architecture with distributed bookkeeper underneath seems so much more advanced.

Re: RabbitMQ vs. Kafka – An Architect’s Dilemma (Part 1)

#106

Post by raducu »

ryanjshaw wrote: post #55
raducu wrote: post #26

Earlier quoted context omitted.

> who can't just now pluck messages off a queue to process The problem is you cannot mark individual messages as read, for a given consumer&partition you can only update the offset for a partition. If a certain message processing takes very long, all other messages in that partition will have to wait. Also, with kafka, the max read concurrency is equal to the number of partitions, for something like rabbitMq it is mu…

> The problem is you cannot mark individual messages as read, for a given consumer&partition you can only update the offset for a partition. Hence "smart clients". If you MUST process every message at least once, you will anyway be tracking messages individually on the client (e.g. a DB or file system plus logic for idempotent message processing) and thus disable auto-offset commits back to the cluster for your consu…

You've made very good points about smart clients, but at some point one has to ponder if it's worth it or one should just not use kafka in the first place.

I've seen databases used as messaging queues and if it was up to me, I'd never do that. It's usually "but we already have kafka + db, why burden ourselves with another messaging technology?", which is fair.

> You can stream messages into a buffer and process them in parallel, and commit the low watermark offset whenever it changes, as described above. I've implemented this in .NET with Channels and saturate the CPUs with no problem.

That is very nice -- certainly seems better than just batch processing of kafka messages, but you're still just kicking the can down the road. How large do allow the buffer to become and what do you do when it's getting too large?

You probably use a DLQ.

Don't get me wrong, I think the buffer idea probably works most of the time.

Re: RabbitMQ vs. Kafka – An Architect’s Dilemma (Part 1)

#107

Post by zerbinxx »

pnpnp wrote: post #103
doctor_eval wrote: post #76

Earlier quoted context omitted.

I thought NATS didn't actually store messages, am I mistaken? Looking at Wikipedia ( https://en.wikipedia.org/wiki/NATS_Messaging ) I see that I'm technically right, it's JetStream that does the storage layer - but it's part of the NATS server. From memory, I really liked the philosophy of NATS but found the nomenclature confusing.

I think they call that part of NATS “Jetstream” if I’m not mistaken. I haven’t used it, but I believe it has some form of message persistence. I have used it mostly for message-first services, and found subject-based messaging a breath of fresh air to decouple services. You can do the same thing with RabbitMQ topic exchanges, but it requires quite a bit more hand-waving.

Jetstream does indeed have message persistence: I can issue queries like “get messages on topic since 5 minutes ago” - I do this a ton. However, that seems to be the extent of the storage/query API that it exposes for historical messages. I’m quite a big fan, and would recommend it with the caveat that Jetstream is considerably more complex than simple nats and I get the feeling I’m barely scratching the surface with it.

Re: RabbitMQ vs. Kafka – An Architect’s Dilemma (Part 1)

#108

Post by zerbinxx »

dventimi wrote: post #77

> Apache Kafka is not an implementation of a message broker. Instead, it is a distributed streaming platform. Unlike RabbitMQ, which is based on queues and exchanges, Kafka’s storage layer is implemented using a partitioned transaction log. Kafka also... This seems like an important passage, drawing the crucial and long-awaited distinction between RabbitMQ and Kafka, and yet without having defined a "partitioned tran…

This is sadly commonplace in tech blogs: rather than taking a great opportunity to hook the reader, the writer will drop a vocab term in bold and move on.

Re: RabbitMQ vs. Kafka – An Architect’s Dilemma (Part 1)

#109

Post by mherdeg »

vorpalhex wrote: post #95
kumarvvr wrote: post #93

Earlier quoted context omitted.

Could you expand on this a bit more? I am curious.

NATs has a decent-ish guide here: https://docs.nats.io/nats-concepts/overview/compare-nats A few things they get wrong mostly about Rabbit: + RabbitMQ does support replay, and also has a memory only mode which will support persistance in a cluster + RabbitMQ doesn't have that sensitive of a latency between cluster members (no more sensitive than NATS in some setups). + RabbitMQ also supports Prometheus A good (but in…

Thanks for this link -- really interesting.

I like the suggestion to rethink whether you actually need to be doing asynchronous computing with a message broker/queue/stream or whether you can represent your work another way.

Re: RabbitMQ vs. Kafka – An Architect’s Dilemma (Part 1)

#110

Post by olavgg »

relay23 wrote: post #86
EdwardDiego wrote: post #66

Earlier quoted context omitted.

Pulsar can have both MQ semantics and pub/sub semantics. In pub/sub it's sorta like "Kafka with all bits people found it necessary to build later already built in", e.g., a proxy, schema registry, connectors, replication, tiered storage, all out of the box. It also has lightweight streaming functions built-in, but they operate per record, so good for lightweight transforms/routing, not for stream aggregations etc. It…

Massive and complex platform.. at a certain point why not just run 2 different platforms that are best of breed for each.

You can just run Pulsar standalone which is super-simple, it is a great start and scales just fine for one machine. Once you outscale one machine, you can refactor Pulsar into a distributed setup. The advantage is that everything is still familiar.
Post reply on HN