What If We Could Rebuild Kafka from Scratch?

#1

Post by mpweiher » Fri, Apr 25, 2025, 5:34 AM UTC

What If We Could Rebuild Kafka from Scratch?
morling.dev

Re: What If We Could Rebuild Kafka from Scratch?

#2

Post by hardwaresofton » Fri, Apr 25, 2025, 6:46 AM UTC

See also: Warpstream, which was so good it got acquired by Confluent.

Feels like there is another squeeze in that idea if someone “just” took all their docs and replicated the feature set. But maybe that’s what S2 is already aiming at.

Wonder how long warpstream docs, marketing materials and useful blogs will stay up.

Re: What If We Could Rebuild Kafka from Scratch?

#3

Post by ActorNightly » Fri, Apr 25, 2025, 6:51 AM UTC

[flagged]

Re: What If We Could Rebuild Kafka from Scratch?

#4

Post by Ozzie_osman » Fri, Apr 25, 2025, 7:02 AM UTC

I feel like everyone's journey with Kafka ends up being pretty similar. Initially, you think "oh, an append-only log that can scale, brilliant and simple" then you try it out and realize it is far, far, from being simple.

Re: What If We Could Rebuild Kafka from Scratch?

#5

Post by Spivak » Fri, Apr 25, 2025, 7:07 AM UTC

Once you start asking to query the log by keys, multi-tenancy trees of topics, synchronous commits-ish, and schemas aren't we just in normal db territory where the kafka log becomes the query log. I think you need to go backwards and be like what is the feature a rdbms/nosql db can't do and go from there. Because the wishlist is looking like CQRS with the front queue being durable but events removed once persisted in the backing db where the clients query events from the db.

The backing db in this wishlist would be something in the vein of Aurora to achieve the storage compute split.

Re: What If We Could Rebuild Kafka from Scratch?

#6

Post by butterlettuce » Fri, Apr 25, 2025, 7:10 AM UTC

ActorNightly wrote: post #3

[flagged]

Step 2, use Python for everything

Re: What If We Could Rebuild Kafka from Scratch?

#7

Post by sam_lowry_ » Fri, Apr 25, 2025, 7:19 AM UTC

ActorNightly wrote: post #3

[flagged]

Java APIs are indeed awfully complex.

Re: What If We Could Rebuild Kafka from Scratch?

#8

Post by mirekrusin » Fri, Apr 25, 2025, 7:21 AM UTC

butterlettuce wrote: post #6

ActorNightly wrote: post #3

[flagged]

Step 2, use Python for everything

Good choice, leaves space to rewrite in rust later, right?

Re: What If We Could Rebuild Kafka from Scratch?

#9

Post by Ygg2 » Fri, Apr 25, 2025, 7:21 AM UTC

butterlettuce wrote: post #6

ActorNightly wrote: post #3

[flagged]

Step 2, use Python for everything

Step 3, Rewrite it in Rust.

Step 4, Rewrite it in Zig.

Step 5, due to security issues rewrite it in Java 63.

Re: What If We Could Rebuild Kafka from Scratch?

#10

Post by carlmr » Fri, Apr 25, 2025, 7:24 AM UTC

Ozzie_osman wrote: post #4

I feel like everyone's journey with Kafka ends up being pretty similar. Initially, you think "oh, an append-only log that can scale, brilliant and simple" then you try it out and realize it is far, far, from being simple.

I'm wondering how much of that is bad developer UX and defaults, and how much of that is inherent complexity in the problem space.

Like the article outlines, partitions are not that useful for most people. Instead of removing them, how about having them behind a feature flag, i.e. not on by default. That would ease 99% of users problems.

The next point in the article which to me resonates is the lack of proper schema support. That's just bad UX again, not inherent complexity of the problem space.

On testing side, why do I need to spin up a Kafka testcontainer, why is there no in-memory kafka server that I can use for simple testing purposes.