Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Walrus – a Kafka alternative written in Rust

github.com

Post reply on HN
Print view View original

31–40 of 57 posts

Re: Show HN: Walrus – a Kafka alternative written in Rust

#31

Post by mrkeen »

gethly wrote: post #22

I never understood the popularity of Kafka. It's just a queue with persistent storage(ie. not in-memory queu with ram-size limited capacity) after all.

A queue with persistent storage is like a ledger whose entries don't vanish when you read them, or a git branch whose commits stick around for longer than 24-72 hours.

It's popular because it didn't have any competition while it built up its ecosystem. And even though there's competitors now, I haven't had time to check them out, and they still brand themselves as "Kafka-alternatives".

Re: Show HN: Walrus – a Kafka alternative written in Rust

#33

Post by thinkharderdev »

EdwardDiego wrote: post #13
teleforce wrote: post #7

For Kafka alternative written in C++ there's Redpanda [1],[2]. Redpanda claim of better performance but benchmarks showed no clear winner [3]. It will be interesting to test them together on the performance benchmarks. I've got the feeling it's not due to programming language implementation of Scala/Java (Kafka), C++ (Redpanda) and Rust (Walrus). It's the very architecture of Kafka itself due to the notorious head of…

> It's the very architecture of Kafka itself due to the notorious head of line problem Except a consumer can discard an unprocessable record? I'm not certain I understand how HOL applies to Kafka, but keen to learn more :)

> Except a consumer can discard an unprocessable record?

It's not the unproccessable records that are the problem it is the records that are very slow to process (for whatever reason).

Re: Show HN: Walrus – a Kafka alternative written in Rust

#34

Post by zellyn »

nubskr wrote: post #30
zellyn wrote: post #28

Earlier quoted context omitted.

The Developer Voices interview where Kris Jenkins talks to Ryan Worl is one of the best, and goes into a surprising amount of detail: https://www.youtube.com/watch?v=xgzmxe6cj6A tl;dr they write to s3 once every 250ms to save costs. IIRC, they contend that when you keep things organized by writing to different files for each topic, it's the Linux disk cache being clever that turns the tangle of disk block arrangement…

s3 charges per 1,000 Update requests, not sure how it's sustainable to do it every 250ms tbh, especially in multi tenant mode where you can have thousands of 'active' blocks being written to

Guess it beats doing it every 250ms for every topic…

Re: Show HN: Walrus – a Kafka alternative written in Rust

#36

Post by enether »

For Rust-based Kafka alternatives, I like Tansu[1]. It at least provides Kafka API parity, and critically also gives users a pluggable backend (embedded SQLite, S3 for low cost diskless type workloads and Postgres because just use Postgres)

It’s nice to try and out innovate Kafka, but I fear the network effect can’t be beaten unless the alternative is 10x better.

Something like Warpstream’s architecture[2] had a shot at dethroning Kafka, but critically even they adopted the Kafka API. Sure enough, Apache Kafka introduced a competing feature[3] within two years of warpstreams launch too.

[1] - https://github.com/tansu-io/tansu [2] - https://www.warpstream.com/ [3] - https://topicpartition.io/blog/kip-1150-diskless-topics-in-a...

Re: Show HN: Walrus – a Kafka alternative written in Rust

#38

Post by kylecazar »

lionkor wrote: post #8

Fun anecdote; a couple years ago I started writing a Kafka alternative in C++ with a friend. I got pretty far, but abandoned the project. We called it `tuberculosis`, or `tube` for short; of course, that is what killed Kafka.

"Consumption" works too :)

Assuming topics are consumed in your version, a la Kafka.

Post reply on HN