Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Redis vs. Kafka vs. RabbitMQ

blog.devgenius.io

Post reply on HN
Print view View original

41–44 of 44 posts

Re: Redis vs. Kafka vs. RabbitMQ

#41

Post by morelisp »

nitrogen wrote: post #33
dcminter wrote: post #21

Earlier quoted context omitted.

Unlike a queue it is common for events to be: Kept indefinitely for future re-consumption. Partitioned with different consumers seeing different slices of the data.

Kept indefinitely for future re-consumption. Are there any stories of this saving someone from a potential disaster? My experience has been that this only causes bugs, such as resending hundreds of thoudands of out-of-date emails.

Independently of application logic (which also sometimes uses it), it's the primary mechanism in Kafka for handling high-throughput consumers while still recovering from failure. Consumers grab a batch of e.g. 1000 events, checkpoint every X seconds while processing them; if they die the events are still there and they restart from the last checkpoint.

It also means every message in Kafka is "addressable" via topic/partition/offset which lets you refer to "foreign" messages etc.

Re: Redis vs. Kafka vs. RabbitMQ

#42

Post by mtberatwork »

cebert wrote: post #40
mtberatwork wrote: post #27

Earlier quoted context omitted.

Maybe I'm missing something but this setup sounds like a re-creation of Beanstalkd. https://beanstalkd.github.io/

I haven’t heard of Beanstalkd before. I am just starting to read there site, but this sounds like a distributed job processing system? There’s some potentially some overlap. Our goal was to have a serverless form of event sourceing. We don’t have 1000s of subscribers to events (we aren’t FB), so our approach is working ok for us. Thanks for informing me about Beanstalkd, I will read up on it more.

If you look at the concept of "tubes" in beanstalkd, it seems to be quite similar to what you are doing in AWS. Each of your independent queues is just a "tube" in beanstalkd. Quite honestly though, you can do this same functionality with just about any message broker. No need to run all these separate services.

Re: Redis vs. Kafka vs. RabbitMQ

#43

Post by dcminter »

nitrogen wrote: post #33
dcminter wrote: post #21

Earlier quoted context omitted.

Unlike a queue it is common for events to be: Kept indefinitely for future re-consumption. Partitioned with different consumers seeing different slices of the data.

Kept indefinitely for future re-consumption. Are there any stories of this saving someone from a potential disaster? My experience has been that this only causes bugs, such as resending hundreds of thoudands of out-of-date emails.

It's not usually intended as a disaster recovery strategy; it's more often intended so that the event stream can be ingested in future by completely new consumers.

Retention policies and compaction exist where you don't actually want to keep the data, but the capability is one of the distinguishing features.

You can make an event log look like a database or a queue or a cache - but if you do that you should definitely consider whether you are using the right tool for the job.

Re: Redis vs. Kafka vs. RabbitMQ

#44

Post by raducu »

dtech wrote: post #39
morelisp wrote: post #35

Earlier quoted context omitted.

> You mean pre-commit to the maximum parallelism by setting the number of partitions? Yes. Then depending on the data it can be difficult-to-impossible to scale past that. Scaling down , of course you can always start fewer consumers, but unless you have many more partitions than consumers or partitions as a multiple of consumers, the load will be unbalanced. > But you get so much more out of kafka with consumer grou…

How difficult it is to scale depends on your requirements, but if those aren't strict it's as easy as adjusting a parameter. In my experience it's not hard at all. If you can live with not strictly ordered messages around scale time you just rescale. If you can live with some latency you stop producing, wait until lag is zero, scale, and then start producing again. Plus if you pick a partition number with nice diviso…

I really like your number of partitions with nice divisors, I thought of it myself and enthusiastically raved about it to an architect (which I really respect for a lot of reasons) but he insisted on some formula about throughput and I dropped the idea. Truth be told, I work for a bank with fixed number of servers.
Post reply on HN