Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

What If We Could Rebuild Kafka from Scratch?

morling.dev

Post reply on HN
Print view View original

181–190 of 229 posts

Re: What If We Could Rebuild Kafka from Scratch?

#181

Post by jsumrall »

olavgg wrote: post #59

How many of the Apache Kafka issues are adressed by switching to Apache Pulsar? I skipped learning Kafka, and jumped right into Pulsar. It works great for our use case. No complaints. But I wonder why so few use it?

I've been down this path, and if my experience is more common, then it really boils down to the classic "Nobody gets fired for buying IBM", and here IBM -> Confluent.

StreamNative seems like an excellent team, and I hope they succeed. But as another comment has written, something (puslar) being better (than kafka) has to either be adopted from the start, or be a big enough improvement to change— and as difficult and feature-poor that Kafka is, it still gets the job done.

I can rant longer about this topic but Pulsar _should_ be more popular, but unfortunately Confluent has dominated here and rent-seeking this field into the ground.

Re: What If We Could Rebuild Kafka from Scratch?

#183

Post by Misdicorl »

singron wrote: post #161
Misdicorl wrote: post #135

Earlier quoted context omitted.

I suppose it depends on your message volume. To me, processing 100k messages and then getting a page however long later as the broker (or whatever) falls apart sounds much worse than head of line blocking and seeing the problem directly in my consumer. If I need to not do head of line blocking, I can build whatever failsafe mechanisms I need for the problematic data and defer to some other queueing system (typically,…

It's interesting you say that, since this turned an infra scaling problem into a worker problem for us. Previously, we would get terrible head-of-line throughput issues, so we would use an egregious number of partitions to try to alleviate that. Lots of partitions is hard to manage since resizing topics is operationally tedious and it puts a lot of strain on brokers. But no matter how many partitions you have, the he…

Yeah I'd say kafka is not a great technology if your median and 99ths (or 999ths if volume is large enough) are wildly different which sounds like your situation. I use kafka in contexts where 99ths going awry usually aren't key dependent so I don't have the issues you see.

I tend to prefer other queueing mechanisms in those cases, although I still work hard to make 99ths and medians align as it can still cause issues (especially for monitoring)

Re: What If We Could Rebuild Kafka from Scratch?

#185

Post by [deleted] »

gregfurman wrote: post #171
debadyutirc wrote: post #152

This is a question we asked 6 years ago. What if we wrote it in Rust. And leveraged and WASM. We have been at it for the past 6 years. https://github.com/infinyon/fluvio For the past 2 years we have also been building Flink using Rust and WASM. https://github.com/infinyon/stateful-dataflow-examples/

Fluvio looks awesome! Any chance you’re going to be reviving support for the Kafka wire protocol? https://github.com/infinyon/fluvio/issues/4259

[deleted]

Re: What If We Could Rebuild Kafka from Scratch?

#186

Post by Micoloth »

debadyutirc wrote: post #152

This is a question we asked 6 years ago. What if we wrote it in Rust. And leveraged and WASM. We have been at it for the past 6 years. https://github.com/infinyon/fluvio For the past 2 years we have also been building Flink using Rust and WASM. https://github.com/infinyon/stateful-dataflow-examples/

Interesting!

How would you say your project compares to Arroyo?

Re: What If We Could Rebuild Kafka from Scratch?

#187

Post by lamp_book »

nitwit005 wrote: post #140
salomonk_mur wrote: post #76

Earlier quoted context omitted.

Yeah... Not happening when you have scores of clients running down your database. The reason message queue systems exist is scale. Good luck sending a notification at 9am to your 3 million users and keeping your database alive in the sudden influx of activity. You need to queue that load.

Kafka is itself a database. Sending a message requires what is essentially a database insert. You're still doing a DB commit either way.

It's more of a commit log/write-ahead log/replication stream than a DBMS - consider that DBMSs typically include these in addition to their primary storage.

Re: What If We Could Rebuild Kafka from Scratch?

#188

Post by pas »

galeaspablo wrote: post #137
dpflan wrote: post #132

Earlier quoted context omitted.

Can you elaborate on how you have “adapted…message causality topologies to cope with consuming mechanisms” in relation to the example of a bank account? The causality topology being what here, couldn’t one day MoneyIn should come before else there can be now true MoneyOut?

Right on, great question. Some examples: Example Option 1 You give up on the guarantees across partition keys (bank accounts), and you accept that balances will not reflect a causally consistent state of the past. E.g., Bob deposits 100, Bob sends 50 to Alice. Balances: Bob 0 Alice 50 # the source system was never in this state Bob 100 Alice 50 # the source system was never in this state Bob 50 Alice 50 # eventually…

hmmm... could this be solved by "vector clocks"? if producers are emitting something that depends on a previous event they send the id of the previous event. (so like capabilities, you need proof of "data access".)

or the problem is that again this is O(n^2)? (because then the consumers now need to buffer [potentially] n key streams (and then search for them every time - so "n" times)?

Post reply on HN