Live data from Hacker News

What If We Could Rebuild Kafka from Scratch?

morling.dev

41–50 of 229 posts

Re: What If We Could Rebuild Kafka from Scratch?

#41

https://nats.io is easier to use than Kafka and already solves several of the points in this post I believe, like removing partitions, supporting key-based streams, and having flexible topic hierarchies.

I came here to say just that. Nats solves a lot of those challenges, like different ways to query and preserve messages, hierarchical data, decent authn/authz options for multi-tenancy, much lighter and easier to set up, etc. It has more of a messaging and k/v store feel than the log Kafka is, so while there's some overlap, I don't think they fit the exact same use cases. Nats is fast, but I haven't seen any benchmarks for specifically the bulk write-once append log situation Kafka is usually used for.

Still, if a hypothetical new Kafka would incorporate some of Nats' features, that would be a good thing.

Re: What If We Could Rebuild Kafka from Scratch?

#42

> When producing a record to a topic and then using that record for materializing some derived data view on some downstream data store, there’s no way for the producer to know when it will be able to "see" that downstream update. For certain use cases it would be helpful to be able to guarantee that derived data views have been updated when a produce request gets acknowledged, allowing Kafka to act as a log for a tru…

Of course, if you don't have separate downstream and upstream datastores, you don't have anything to do in the first place.

Re: What If We Could Rebuild Kafka from Scratch?

#43
post #15

Object storage for Kafka? Wouldn't this 10x the latency and cost? I feel like Kafka is a victim of it's own success, it's excellent for what it was designed, but since the design is simple and elegant, people have been using it for all sorts of things for which it was not designed. And well, of course it's not perfect for these use cases.

Warpstream already does Kafka with object storage.

WarpStream has been acquired by Confluent.

Re: What If We Could Rebuild Kafka from Scratch?

#44
post #11

Earlier quoted context omitted.

Good choice, leaves space to rewrite in rust later, right?

The fact that this is so common I think yearns for a language that is basically python and rust smashed together where a project would have some code in the python side and some code in the rust side intermixed fluidly like how you can drop to asm in C. Don't even really try to make the two halves of the language similar. An embedded interpreter and JIT in rust basically but jostled around a bit to make it more cohes…

You can already do that with a macro: https://crates.io/crates/inline-python

But I'm doubtful that it's going to make things simpler if one can't even decide on a language.

Re: What If We Could Rebuild Kafka from Scratch?

#45

I know it’s not what the article is about but I really wish we could rebuild Franz Kafka and hear what he thought about the tech dystopia we are in. >I cannot make you understand. I cannot make anyone understand what is happening inside me. I cannot even explain it to myself. -Franz Kafka, The Metamorphosis

They named it like that for a reason ;)

Re: What If We Could Rebuild Kafka from Scratch?

#46
post #28
post #12

Earlier quoted context omitted.

> Step 1, stop using Java. I've seen these comments for over 15 years yet for some "unknown", "silly" reason java keeps being used for really,really useful software like kafka.

But it's also the reason for why these Apache projects systematically get displaced by better and faster C, C++, Rust or Go alternatives.

It would make sense for a highly succesful but stable java project to be replaced like that, but since I'm in the java world, it's usually replaced with another java project.

I could provide examples myself, but I'm not convinced it's about java vs c++ or go: hadoop, cassandra, zookeeper

Re: What If We Could Rebuild Kafka from Scratch?

#47
post #14
post #10

Earlier quoted context omitted.

I'm wondering how much of that is bad developer UX and defaults, and how much of that is inherent complexity in the problem space. Like the article outlines, partitions are not that useful for most people. Instead of removing them, how about having them behind a feature flag, i.e. not on by default. That would ease 99% of users problems. The next point in the article which to me resonates is the lack of proper schema…

I think it's just horrible software built on great ideas sold on a false premise (this is a generic message queue and if you don't use this you cannot "scale").

It's not just about the scaling, it's about solving the "doing two things" problem.

If you take action a, then action b, your system will throw 500s fairly regularly between those two steps, leaving your user in an inconsistent state. (a = pay money, b = receive item). Re-ordering the steps will just make it break differently.

If you stick both actions into a single event ({userid} paid {money} for {item}) then "two things" has just become "one thing" in your system. The user either paid money for item, or didn't. Your warehouse team can read this list of events to figure out which items to ship, and your payments team can read this list of events to figure out users' balances and owed taxes.

(You could do the one-thing-instead-of-two-things using a DB instead of Kafka, but then you have to invent some kind of pub-sub so that callers know when to check for new events.)

Also it's silly waiting around to see exceptions build up in your dev logs, or for angry customers to reach out via support tickets. When your implementation depends on publishing literal events of what happened, you can spin up side-cars which verify properties of your system in (soft) real-time. One side-car could just read all the ({userid} paid {money} for {item}) events and ({item} has been shipped) events. It's a few lines of code to match those together and all of a sudden you have a monitor of "Whose items haven't been shipped?". Then you can debug-in-bulk (before the customers get angry and reach out) rather than scour the developer logs for individual userIds to try to piece together what happened.

Also, read this thread https://news.ycombinator.com/item?id=43776967 from a day ago, and compare this approach to what's going on in there, with audit trails, soft-deletes and updated_at fields.

Re: What If We Could Rebuild Kafka from Scratch?

#48

https://nats.io is easier to use than Kafka and already solves several of the points in this post I believe, like removing partitions, supporting key-based streams, and having flexible topic hierarchies.

Also remember that NATS was donated to the CNCF a while back, and as a result people built a huge ecosystem around it. Easy to forget.

Re: What If We Could Rebuild Kafka from Scratch?

#49
Ah the siren call of the ground-up rewrite. I didn’t know how deep the assumption of hard disks underpinning everything is baked into its design.

But don’t public cloud providers already all have cloud-native event sourcing? If that’s what you need, just use that instead of Kafka.

Re: What If We Could Rebuild Kafka from Scratch?

#50

Earlier quoted context omitted.

Yeah... It took 4 years to properly integrate Kafka into our pipelines. Everything, like everything is complicated with it: cluster management, numerous semi-tested configurations, etc. My final conclusion with it is that the project just doesn't really know what it wants to be. Instead it tries to provide everything for everybody, and ends up being an unbelievably complicated mess. You know, there are systems that k…

systemd knows very well what it wants to be, they just don't tell anyone. it's real goal is to make Linux administration as useless as windows so RH can sell certifications. tell me the output of systemctl is not as awful as opening the windows service panel.

Tell me systemctl output isn't more beneficial than per distro bash-mess
Post reply on HN