Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

RabbitMQ vs. Kafka – An Architect’s Dilemma (Part 1)

eranstiller.com

Post reply on HN
Print view View original

161–170 of 173 posts

Re: RabbitMQ vs. Kafka – An Architect’s Dilemma (Part 1)

#161

Post by whalesalad »

FridgeSeal wrote: post #134
whalesalad wrote: post #30

Earlier quoted context omitted.

> You can get all the same advantages of message acknowledgments, but now you can also replay queues with rmq you can reject/nack a message and have it put back on the queue. rmq is not well suited for long term historical retention inside queues a-la kafka's logs but it is possible to do. > let different applications use the messages (handy for cross cutting event/notification systems) rmq also does a publish once a…

> with rmq you can reject/nack a message and have it put back on the queue I know other systems have semi-similar mechanisms, however most of them retain the “someone is the sole owner of this message” style design, which I think is fundamentally limiting. Owning application dies, is it acked or not? Acks but never gets around to putting it back on the queue? Who takes priority if 2 separate applications wish to watc…

> Who takes priority if 2 separate applications wish to watch the same stream of events?

each app would get its own queue, the messages would hit a fanout exchange that would route the same message to both queues.

Re: RabbitMQ vs. Kafka – An Architect’s Dilemma (Part 1)

#162

Post by hamandcheese »

snoman wrote: post #123
hamandcheese wrote: post #65

Earlier quoted context omitted.

> Thats kind of like saying dont use any software libraries because they all use the standard lib indirectly so you may as well just use that? This is decent advice, IMO. The cost of dependency management is often vastly understated.

That’s Not Invented Here syndrome, and it’s decidedly bad advice. The cost of dependency management may be understated but it’s always less than the cost of reimplementing everything found in established libraries.

Being judicious in which dependencies you take on is not the same as Not Invented Here syndrome. Code is usually a liability, not an asset.

Re: RabbitMQ vs. Kafka – An Architect’s Dilemma (Part 1)

#163

Post by rcombatwombat »

zerbinxx wrote: post #107
pnpnp wrote: post #103

Earlier quoted context omitted.

I think they call that part of NATS “Jetstream” if I’m not mistaken. I haven’t used it, but I believe it has some form of message persistence. I have used it mostly for message-first services, and found subject-based messaging a breath of fresh air to decouple services. You can do the same thing with RabbitMQ topic exchanges, but it requires quite a bit more hand-waving.

Jetstream does indeed have message persistence: I can issue queries like “get messages on topic since 5 minutes ago” - I do this a ton. However, that seems to be the extent of the storage/query API that it exposes for historical messages. I’m quite a big fan, and would recommend it with the caveat that Jetstream is considerably more complex than simple nats and I get the feeling I’m barely scratching the surface with…

NATS JetStream also implements subject-based addressing at the stream level (unlike Kafka where 1 stream = 1 topic, and you can only use the message's key for distribution, not for addressing).

So you can for example ask for [the first/the last/all] of the messages on a particular subject, or on a hierarchy of subjects by using wildcards. All the filtering is done at the server level.

Re: RabbitMQ vs. Kafka – An Architect’s Dilemma (Part 1)

#164

Post by rcombatwombat »

kumarvvr wrote: post #93
vorpalhex wrote: post #14

Earlier quoted context omitted.

Different set of promises. NATS is great but has a different tradeoff bargain from Rabbit or Kafka.

Could you expand on this a bit more? I am curious.

Also see https://www.youtube.com/watch?v=C4BnJ5QLeTY

Re: RabbitMQ vs. Kafka – An Architect’s Dilemma (Part 1)

#165

Post by kuchenbecker »

blindriver wrote: post #115
kuchenbecker wrote: post #94

Earlier quoted context omitted.

Our prod cluster generates that about every minute at O(1M) qps. We JUST turned on remote Logs because until now Kafka didn't have capacity.

My previous company's Kafka cluster was handing 20 million messages per second 5 years ago, and dozens of petabytes of data per day. Maybe your particular cluster that didn't have the capacity to handle 1M qps, but Kafka easily had that capacity years ago.

It was quota and hardware, not ability. This is a single service onboarding and they need the hardware.

And at that scale, we need to grep the logs so the downstream need the ability to process that volume, which it couldn't until about 2 years ago.

Re: RabbitMQ vs. Kafka – An Architect’s Dilemma (Part 1)

#166

Post by kuchenbecker »

inkyoto wrote: post #102
kuchenbecker wrote: post #94

Earlier quoted context omitted.

Our prod cluster generates that about every minute at O(1M) qps. We JUST turned on remote Logs because until now Kafka didn't have capacity.

TIBCO Rendezvous is tech from 1998/99, millions of messages per second didn't exist at the time. Only NYSE and NASDAQ were capable of producing millions of events back then (still not by minute nor by second). TIBCO Rendezvous was one of the first successful large scale, low latency and near real-time pub/sub implementations, and it had a very efficient, Avro like, wire level serialisation format that made messages v…

$$$ not capability. We have ~50 hosts that generate up to TB per day in just logs and 50k hosts that generate O(200mb/day). The large hosts ssh and grep works surprisingly well, but the smaller hosts is the real benefit.

Hard to justify a team of 7 burning several million in just Logging infra costs.

Re: RabbitMQ vs. Kafka – An Architect’s Dilemma (Part 1)

#167

Post by rcombatwombat »

inkyoto wrote: post #104
rafaelturk wrote: post #53

Earlier quoted context omitted.

If your have a clean event-driven architecture, ie messages are completely agnostic and decoupled from one-another you don't need Kafka.

Event-driven architecture is an architectural principle , and Kafka, RabbitMQ/ActiveMQ, Pulsar, NATS and so forth are implementations that support the event-driven architectural principle. Yet, all of them range in a variety, complexity and extent of features they provide which may or may not be a good fit for a particular use case. Traditional message brokers (RabbitMQ and similar) do support the event-driven archit…

NATS with JetStream provides _both_ queuing like a traditional message broker and multiple data replay from offset (plus KV, and request/reply)

Re: RabbitMQ vs. Kafka – An Architect’s Dilemma (Part 1)

#168

Post by zerbinxx »

doctor_eval wrote: post #133
zerbinxx wrote: post #107

Earlier quoted context omitted.

Jetstream does indeed have message persistence: I can issue queries like “get messages on topic since 5 minutes ago” - I do this a ton. However, that seems to be the extent of the storage/query API that it exposes for historical messages. I’m quite a big fan, and would recommend it with the caveat that Jetstream is considerably more complex than simple nats and I get the feeling I’m barely scratching the surface with…

I’m really interested in a Kafka like message broker in the Go ecosystem and look forward to checking it out for whatever my next project ends up being.

It’s pretty cool. I would personally suggest Kafka or RabbitMQ depending on your needs, as Jetstream has proven to require a lot of ops engineering to remain stable in production

Re: RabbitMQ vs. Kafka – An Architect’s Dilemma (Part 1)

#169

Post by 59nadir »

snoman wrote: post #123
hamandcheese wrote: post #65

Earlier quoted context omitted.

> Thats kind of like saying dont use any software libraries because they all use the standard lib indirectly so you may as well just use that? This is decent advice, IMO. The cost of dependency management is often vastly understated.

That’s Not Invented Here syndrome, and it’s decidedly bad advice. The cost of dependency management may be understated but it’s always less than the cost of reimplementing everything found in established libraries.

One HTTP connection per message (if this is what the original poster meant) is probably a bad idea whether you implement it yourself or not.

Also, let's be honest: The phalanx of developers that violently argue for importing everything and never implementing anything yourselves is way bigger than people who argue for the opposite; I don't think we need to worry about the latter making things worse as much as the former.

We've seen what the world turns into in both scenarios, I would argue, and at least with the first one we got software that ran decently and developers who knew how to actually do things. We have overall much safer languages now so their home grown solutions won't have the same safety issues that they've historically had.

With the importer crowd we've gotten software that feels like molasses and an industry full of people who know only the surface of everything and more importantly become tied to frameworks and specific APIs because they never actually go beyond any surface APIs.

As with most tradeoffs in software there is a good middle-ground, but we won't ever get there if we don't have people who argue for making things yourselves.

Re: RabbitMQ vs. Kafka – An Architect’s Dilemma (Part 1)

#170

Post by debadyutirc »

supermatt wrote: post #8
robertlagrant wrote: post #4

> one is a message broker, and the other is a distributed streaming platform I think this is an odd way of putting it. One is smart messaging; dumb clients. The other is dumb messaging; smart clients. It turns out the latter (i.e. Kafka) scales wonderfully so you can send more data, but you add complexity to your clients, who can't just now pluck messages off a queue to process, or have messages retry upon the first…

> One is smart messaging; dumb clients. The other is dumb messaging; smart clients. All the smartness of the messaging can be implemented in the smart clients. Then you can expose that as a smart messaging api to dumb clients. The most obvious example is kafka streams which exposes a "simple" api rather than dealing directly with kafka, but obviously you could create a less featurefull wrapper than that.

That's a neat way to put it!

- RabbitMQ: Smart messaging, dumb client - Kafka: Dumb messaging, smart client

Have you heard of Fluvio? Fluvio: Smart messaging, Smart Client, Stateful Streaming

Kafka + Flink in Rust + WASM Git Repo - https://github.com/infinyon/fluvio

Post reply on HN