Show HN: Walrus – a Kafka alternative written in Rust
1–10 of 57 posts
Re: Show HN: Walrus – a Kafka alternative written in Rust
#2As someone who myself worked on a hobby-level Rust based Kafka alternative that used Raft for metadata coordination for ~8 months: nice work!
Wasn't immediately clear to me if the data-plane level replication also happens through Raft or something home-rolled? Getting consistency and reliability right with something home-rolled is challenging.
Notes:
- Would love to see it in an S3-backed mode, either entirely diskless like WarpStream or as tiered storage.
- Love the simplified API. If possible, adding a Kafka compatible API interface is probably worth it to connect to the broader ecosystem.
Best of luck!
Re: Show HN: Walrus – a Kafka alternative written in Rust
#3There's also Iggy https://github.com/apache/iggy
Never tried it, but looks promising
Re: Show HN: Walrus – a Kafka alternative written in Rust
#4As someone who myself worked on a hobby-level Rust based Kafka alternative that used Raft for metadata coordination for ~8 months: nice work! Wasn't immediately clear to me if the data-plane level replication also happens through Raft or something home-rolled? Getting consistency and reliability right with something home-rolled is challenging. Notes: - Would love to see it in an S3-backed mode, either entirely diskle…
It says on the github page
" It provides fault-tolerant streaming with automatic leadership rotation, segment-based partitioning, and Raft consensus for metadata coordination."
Re: Show HN: Walrus – a Kafka alternative written in Rust
#5Nice! How does it compare to Redpanda, NATS, etc?
Re: Show HN: Walrus – a Kafka alternative written in Rust
#6As someone who myself worked on a hobby-level Rust based Kafka alternative that used Raft for metadata coordination for ~8 months: nice work! Wasn't immediately clear to me if the data-plane level replication also happens through Raft or something home-rolled? Getting consistency and reliability right with something home-rolled is challenging. Notes: - Would love to see it in an S3-backed mode, either entirely diskle…
It says on the github page " It provides fault-tolerant streaming with automatic leadership rotation, segment-based partitioning, and Raft consensus for metadata coordination." So I guess that's a "yes" to raft?
GP asked about data plane consensus, not metadata/control plane.
Re: Show HN: Walrus – a Kafka alternative written in Rust
#7For Kafka alternative written in C++ there's Redpanda [1],[2].
Redpanda claim of better performance but benchmarks showed no clear winner [3].
It will be interesting to test them together on the performance benchmarks.
I've got the feeling it's not due to programming language implementation of Scala/Java (Kafka), C++ (Redpanda) and Rust (Walrus).
It's the very architecture of Kafka itself due to the notorious head of line problem (check the top most comments [4].
[1] Redpanda – A Kafka-compatible streaming platform for mission-critical workloads (120 comments):
https://news.ycombinator.com/item?id=25075739
[2] Redpanda website:
[3] Kafka vs. Redpanda performance – do the claims add up? (141 comments):
https://news.ycombinator.com/item?id=35949771
[4] What If We Could Rebuild Kafka from Scratch? (220 comments):
Re: Show HN: Walrus – a Kafka alternative written in Rust
#8Fun anecdote; a couple years ago I started writing a Kafka alternative in C++ with a friend. I got pretty far, but abandoned the project.
We called it `tuberculosis`, or `tube` for short; of course, that is what killed Kafka.
Re: Show HN: Walrus – a Kafka alternative written in Rust
#9As someone who myself worked on a hobby-level Rust based Kafka alternative that used Raft for metadata coordination for ~8 months: nice work! Wasn't immediately clear to me if the data-plane level replication also happens through Raft or something home-rolled? Getting consistency and reliability right with something home-rolled is challenging. Notes: - Would love to see it in an S3-backed mode, either entirely diskle…
Hi, the creator here, I think its a good idea to have S3 backed storage mode, its kinda tricky to do it for the 'active' block which we are currently writing to, but totally doable for historical data.
Also about the kafka API, I tried to implement that earlier, I had a sort of `translation` layer for that earlier, but it gets pretty complicated to maintain that because kafka is offset based, while walrus is message based.
Re: Show HN: Walrus – a Kafka alternative written in Rust
#10For Kafka alternative written in C++ there's Redpanda [1],[2]. Redpanda claim of better performance but benchmarks showed no clear winner [3]. It will be interesting to test them together on the performance benchmarks. I've got the feeling it's not due to programming language implementation of Scala/Java (Kafka), C++ (Redpanda) and Rust (Walrus). It's the very architecture of Kafka itself due to the notorious head of…
In the current benchmarks, I only have Kafka and rocksdb wal, will surely try to add redpanda there as well, curious how walrus would hold up against seastar based systems.