Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

RabbitMQ vs. Kafka – An Architect’s Dilemma (Part 1)

eranstiller.com

Post reply on HN
Print view View original

141–150 of 173 posts

Re: RabbitMQ vs. Kafka – An Architect’s Dilemma (Part 1)

#141

Post by onetimeuse92304 »

Beached wrote: post #137
onetimeuse92304 wrote: post #129

If you have a throughput problem you are most likely doing it wrong. If your knee jerk reaction is to scale your messaging up you may want to reconsider. Messaging systems are usually hard to scale up and always very costly to do so compared to the amount of data they are transferring. The simplest thing you can do is to realise WHY you are using messages. Messages are there to trigger a process. Usually, you don't n…

1) some designs can't tolerate the producer sending messages at such a delay. 3) s3 is not cheap storage, it is significantly higher cost than most on prem solutions when talking about large scale storage. (pedabytes scale)

1) If you can't accept and process the the load of messages with your messaging the discussion of whether 100ms is or is not acceptable delay is very much pointless.

Messaging middleware is by design not suited well for architecting real time systems. If you require real time guarantees you would benefit from some other communication channel.

2)

3) S3 is orders of magnitude cheaper compared to messaging platforms like RabbitMQ or Kafka. If you take load off of your RabitMQ or Kafka and put it on S3 you should see a significant reduction in cost.

S3 might be more expensive than other persistence solutions, true. Just choose whatever else you have. I used S3 as an example because it is extremely easy to implement and get going.

Again, all engineering is about tradeoffs. You compare two solutions, they will always have some cons. You just decide which cons you can live with. If your platform can't process messages at all and you don't know how to scale it up to do so that's pretty large problem in my book.

Re: RabbitMQ vs. Kafka – An Architect’s Dilemma (Part 1)

#142

Post by [deleted] »

relay23 wrote: post #85
whalesalad wrote: post #16

One is a tomato, the other is an orange. From a distance they might look alike but they really are two completely different tools. This is a pretty solid explanation of the differences with good illustrations. Rabbit can do everything Kafka does - and much more - in a more configurable manner. Kafka is highly optimized for essentially one use case and does that well. Nothing in life is free, there are trade-offs ever…

>> Rabbit can do everything Kafka does - and much more - in a more configurable manner. Sure, if you're doing like 10's of MB/s. RMQ is fast compared to AK if you're not adding durability, persistence, etc. Try to run gigabytes per second through it though, or stretch across regions, or meet RTO when the broker gets overloaded and crashes.. Get your shovel ready! ;) Kafka itself is dumb but scalable and resilient, it…

[deleted]

Re: RabbitMQ vs. Kafka – An Architect’s Dilemma (Part 1)

#143

Post by KaiserPro »

officialchicken wrote: post #61
KaiserPro wrote: post #45

Earlier quoted context omitted.

> now you can also replay queues yeahnah, that leads to people treating queues like databases (I'm looking at you new york times, you know what you did wrong) its either a queue, or a pubsub, either way its ephemeral. Once its gone, it should stay gone. thats what database, object stores or filesystems are for. Kafka is a beast, has lots of bells and whistles and grinds to a halt when you look at it funny. Yes, it ca…

> (I'm looking at you new york times, you know what you did wrong) You're going to have to be a tiny bit more specific here. NYT is THE factory of wrongness for sure. In every dimension. Are we talking "yellow cake" wrong, or somewhere else on the severity of f'up scale...

https://www.confluent.io/blog/publishing-apache-kafka-new-yo...

^ this.

All they needed was a database, or possibly a DB that supports row signing. I mean actually they could have done it with git. They don't publish that many stories an hour.

Everything about this setup is just plain wrong, and to then boast about it, absolute madness.

Re: RabbitMQ vs. Kafka – An Architect’s Dilemma (Part 1)

#144

Post by KaiserPro »

serallak wrote: post #113
KaiserPro wrote: post #45

Earlier quoted context omitted.

> now you can also replay queues yeahnah, that leads to people treating queues like databases (I'm looking at you new york times, you know what you did wrong) its either a queue, or a pubsub, either way its ephemeral. Once its gone, it should stay gone. thats what database, object stores or filesystems are for. Kafka is a beast, has lots of bells and whistles and grinds to a halt when you look at it funny. Yes, it ca…

What would you choose today ?

It depends on the context.

Currently I'm using DDS, specifically from eprosma. I would avoid that implementation unless you're using java.

I really like NATS. However I would probably use what every is bundled with the cloud system I'm using, unless its super critical.

MQTT is quite nice for things, as is rabbit.

Re: RabbitMQ vs. Kafka – An Architect’s Dilemma (Part 1)

#145

Post by fnxjdjc »

bob1029 wrote: post #79

I've seen Tibco Rendezvous used in manufacturing. ~300 megabytes per hour of raw log generated 24/7/365 by tools and control systems in a factory setting. Probably on the order of 10k+ participants in the pub/sub network. If you are running something like a factory where thousands of independent systems need to communicate in some way, this kind of tech starts to look like the only option. If you are orchestrating th…

[dead]

Re: RabbitMQ vs. Kafka – An Architect’s Dilemma (Part 1)

#146

Post by sharms »

_a_a_a_ wrote: post #130
blindriver wrote: post #115

Earlier quoted context omitted.

My previous company's Kafka cluster was handing 20 million messages per second 5 years ago, and dozens of petabytes of data per day. Maybe your particular cluster that didn't have the capacity to handle 1M qps, but Kafka easily had that capacity years ago.

I have to ask, what value is this adding business-wise to store so much?

Not OP but I think it isn't always about storing, but having a log of events which get routed, processed, and aggregated in many cases.

Re: RabbitMQ vs. Kafka – An Architect’s Dilemma (Part 1)

#147

Post by continuitylimit »

robertlagrant wrote: post #4

> one is a message broker, and the other is a distributed streaming platform I think this is an odd way of putting it. One is smart messaging; dumb clients. The other is dumb messaging; smart clients. It turns out the latter (i.e. Kafka) scales wonderfully so you can send more data, but you add complexity to your clients, who can't just now pluck messages off a queue to process, or have messages retry upon the first…

It is not odd, it is basically accurate. You are making a fetish of the S-C interaction but the essential matter is that Kafka is designed to store & distribute logs, whereas Rabbit is designed to route & send messages. The ‘store’ bit is very much a part of Kafka’s mission statement but not for Rabbit.

Re: RabbitMQ vs. Kafka – An Architect’s Dilemma (Part 1)

#148

Post by continuitylimit »

bob1029 wrote: post #79

I've seen Tibco Rendezvous used in manufacturing. ~300 megabytes per hour of raw log generated 24/7/365 by tools and control systems in a factory setting. Probably on the order of 10k+ participants in the pub/sub network. If you are running something like a factory where thousands of independent systems need to communicate in some way, this kind of tech starts to look like the only option. If you are orchestrating th…

It’s not apparent as a dilemma until the said architect has spent years -convinced- that the grand edifice is “good architecture”, and finally matured as a practitioner. Only after that phase passes is there an actual ego-driven dilemma, strictly speaking.

Re: RabbitMQ vs. Kafka – An Architect’s Dilemma (Part 1)

#149

Post by continuitylimit »

jpgvm wrote: post #97
mianos wrote: post #96

Earlier quoted context omitted.

While pulsar on paper seems a superior solution, in my experience it is very still very immature and very buggy. I really want to use it over kafka but I would not bet my business on it. I am not a fan of Kafka, it's kinda old, and the code is a bit messy, a lot of the once only semantic problems 100% solved by Pulser are sorta kinda in Kfaka these days. All the newer stuff like built in RAFT makes it competitive wit…

I have run all 3 at big scale. Kafka is still great as long as everyone using it understands it's a stream, not a queue and using it like a queue is going to get them burnt. I don't touch RabbitMQ with a 30ft pole anymore, too many lost days or nights to split brains and other chaos. Pulsar has mostly replaced Kafka for me because I don't need to worry about people coming along and changing requirements after the fac…

Pulsar is a very interesting architectural case study. The cost of the greater clarity and flexibility is the greater management burden. The server-side functions are nice (and remind of JEE MBeans) but the direct challenge to Kafka is the decoupling of storage from servers via Bookkeeper (which adds the lower layer cluster management burden) which addresses the rebalancing headaches with Kafka type of solution (where the server and storage are unified).

Re: RabbitMQ vs. Kafka – An Architect’s Dilemma (Part 1)

#150

Post by bhouston »

relay23 wrote: post #88
bhouston wrote: post #5

If you use Confluent Kafka, the billing is pretty high. About 4 years ago it was much cheaper, but then they completely revamped the pricing to something ridiculous. I found that switching to Google Pub/Sub, at least if it meets your needs, is cheaper.

It's cheaper until you get to a sizeable workload, and the P90+ latency is ridiculous.. the kafka api is weak and when you're not using kafka api you're limited on integration tools unless you want to be super locked in to GCP.

The lock-in argument is a non starter with me. I never see people move between clouds (well it happens but it is incredibly rare) and it isn’t because of lock in but rather they are pretty close to equivalent. And if you wanted to go on prem you can replace the messaging system, it isn’t one of the hardest steps of going on prem.
Post reply on HN