Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

What If We Could Rebuild Kafka from Scratch?

morling.dev

Post reply on HN
Print view View original

101–110 of 229 posts

Re: What If We Could Rebuild Kafka from Scratch?

#101

Post by enether »

olavgg wrote: post #59

How many of the Apache Kafka issues are adressed by switching to Apache Pulsar? I skipped learning Kafka, and jumped right into Pulsar. It works great for our use case. No complaints. But I wonder why so few use it?

There’s inherently a lot of path-dependent network effects in open source software.

Just because something is 10-30% better in certain cases almost never warrants its adoption, if on the other side you get much less human expertise, documentation/resources and battle tested testimonies.

This, imo, is the story of most Kafka competitors

Re: What If We Could Rebuild Kafka from Scratch?

#102

Post by mountainriver »

ketzo wrote: post #100
mountainriver wrote: post #94

Earlier quoted context omitted.

I greatly prefer redis streams. Not all the same features, but if you just need basic streams, redis has the dead simple implementation I always wanted. Not to mention you then also have a KV store. Most problems can be solved with redis + Postgres

Actually thinking about building something with Redis streams next week. Any particular advice/sharp edges/etc?

Honestly no, which was my favorite part. So easy to spin up, "just works" the way you expect. It's night and day from a lot of the other solutions in the ecosystem.

Re: What If We Could Rebuild Kafka from Scratch?

#103

Post by mike_hearn »

wvh wrote: post #37
nitwit005 wrote: post #17

> When producing a record to a topic and then using that record for materializing some derived data view on some downstream data store, there’s no way for the producer to know when it will be able to "see" that downstream update. For certain use cases it would be helpful to be able to guarantee that derived data views have been updated when a produce request gets acknowledged, allowing Kafka to act as a log for a tru…

The problem is that you don't know who's listening. You don't want all possible interested parties to hammer the database. Hence the events in between. Arguably, I'd not use Kafka to store actual data, just to notify in-flight.

In some databases that's not a problem. Oracle has a built in horizontally scalable message queue engine that's transactional with the rest of the database. You can register a series of SELECT queries and be notified when the results have (probably) changed, either via direct TCP server push or via a queued message for pickup later. It's not polling based, the transaction engine knows what query predicates to keep an eye out for.

Disclosure: I work part time for Oracle Labs and know about these features because I'm using them in a project at the moment.

Re: What If We Could Rebuild Kafka from Scratch?

#104

Post by enether »

axegon_ wrote: post #35
elvircrn wrote: post #31

Surprised there's no mention of Redpanda here.

Having used both Kafka and Redpanda on several occasions, I'd pick Redpanda any day of the week without a second thought. Easier to setup, easier to maintain, a lot less finicky ans uses a fraction of the resources.

In what way is it materially easier to maintain and less finicky? I read a lot about this but I haven’t seen a concise bullet point list of why, which leads me to naturally distrust such claims. Ditto for the resources - Kafka is usually bottlenecked on disk/network, and whether it’s c++ or not doesn’t solve that

Re: What If We Could Rebuild Kafka from Scratch?

#105

Post by enether »

rvz wrote: post #81

Every time another startup falls for the Java + Kafka arguments, it keeps the AWS consultants happier. Fast forward into 2025, there are many performant, efficient and less complex alternatives to Kafka that save you money, instead of burning millions in operational costs "to scale". Unless you are at a hundred million dollar revenue company, choosing Kafka in 2025 is doesn't make sense anymore.

Can you share three? Without concrete suggestions, this is just a disparaging narrative (that a lot of vendors use)

Re: What If We Could Rebuild Kafka from Scratch?

#106

Post by Zambyte »

mountainriver wrote: post #94
vim-guru wrote: post #16

https://nats.io is easier to use than Kafka and already solves several of the points in this post I believe, like removing partitions, supporting key-based streams, and having flexible topic hierarchies.

I greatly prefer redis streams. Not all the same features, but if you just need basic streams, redis has the dead simple implementation I always wanted. Not to mention you then also have a KV store. Most problems can be solved with redis + Postgres

> Not to mention you then also have a KV store.

https://docs.nats.io/nats-concepts/jetstream/key-value-store

Re: What If We Could Rebuild Kafka from Scratch?

#107

Post by DrFalkyn »

munksbeer wrote: post #58
Ozzie_osman wrote: post #4

I feel like everyone's journey with Kafka ends up being pretty similar. Initially, you think "oh, an append-only log that can scale, brilliant and simple" then you try it out and realize it is far, far, from being simple.

I'm not a fan or an anti-fan of kafka, but I do wonder about the hate it gets. We use it for streaming tick data, system events, order events, etc, into kdb. We write to kafka and forget. The messages are persisted, and we don't have to worry if kdb has an issue. Out of band consumers read from the topics and persist to kdb. In several years of doing this we haven't really had any major issues. It does the job we wan…

What happens if the Kafka node fails ?

Re: What If We Could Rebuild Kafka from Scratch?

#108

Post by singron »

galeaspablo wrote: post #80

Agreed. The head of line problem is worth solving for certain use cases. But today, all streaming systems (or workarounds) with per message key acknowledgements incur O(n^2) costs in either computation, bandwidth, or storage per n messages. This applies to Pulsar for example, which is often used for this feature. Now, now, this degenerate time/space complexity might not show up every day, but when it does, you’re toa…

Check out the parallel consumer: https://github.com/confluentinc/parallel-consumer

It processes unrelated keys in parallel within a partition. It has to track what offsets have been processed between the last committed offset of the partition and the tip (i.e. only what's currently processed out of order). When it commits, it saves this state in the commit metadata highly compressed.

Most of the time, it was only processing a small number of records out of order so this bookkeeping was insignificant, but if one key gets stuck, it would scale to at least 100,000 offsets ahead, at which point enough alarms would go off that we would do something. That's definitely a huge improvement to head of line blocking.

Re: What If We Could Rebuild Kafka from Scratch?

#109

Post by Zambyte »

rmoff wrote: post #61
boomskats wrote: post #48

Earlier quoted context omitted.

Also remember that NATS was donated to the CNCF a while back, and as a result people built a huge ecosystem around it. Easy to forget.

Relevant: https://www.cncf.io/blog/2025/04/24/protecting-nats-and-the-...

Wow, what a poor move. Really surprising having worked with the folks at Synadia directly.
Post reply on HN