Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Walrus – a Kafka alternative written in Rust

github.com

Post reply on HN
Print view View original

11–20 of 57 posts

Re: Show HN: Walrus – a Kafka alternative written in Rust

#11

Post by EdwardDiego »

nubskr wrote: post #9
roncohen wrote: post #2

As someone who myself worked on a hobby-level Rust based Kafka alternative that used Raft for metadata coordination for ~8 months: nice work! Wasn't immediately clear to me if the data-plane level replication also happens through Raft or something home-rolled? Getting consistency and reliability right with something home-rolled is challenging. Notes: - Would love to see it in an S3-backed mode, either entirely diskle…

Hi, the creator here, I think its a good idea to have S3 backed storage mode, its kinda tricky to do it for the 'active' block which we are currently writing to, but totally doable for historical data. Also about the kafka API, I tried to implement that earlier, I had a sort of `translation` layer for that earlier, but it gets pretty complicated to maintain that because kafka is offset based, while walrus is message…

TBH I don't think anyone can utilise S3 for the active segment, I didn't dig into Warpstream too much, but I vaguely recall they only offloaded to S3 once the segment was rolled.

Re: Show HN: Walrus – a Kafka alternative written in Rust

#13

Post by EdwardDiego »

teleforce wrote: post #7

For Kafka alternative written in C++ there's Redpanda [1],[2]. Redpanda claim of better performance but benchmarks showed no clear winner [3]. It will be interesting to test them together on the performance benchmarks. I've got the feeling it's not due to programming language implementation of Scala/Java (Kafka), C++ (Redpanda) and Rust (Walrus). It's the very architecture of Kafka itself due to the notorious head of…

> It's the very architecture of Kafka itself due to the notorious head of line problem

Except a consumer can discard an unprocessable record? I'm not certain I understand how HOL applies to Kafka, but keen to learn more :)

Re: Show HN: Walrus – a Kafka alternative written in Rust

#14

Post by sgt »

lionkor wrote: post #8

Fun anecdote; a couple years ago I started writing a Kafka alternative in C++ with a friend. I got pretty far, but abandoned the project. We called it `tuberculosis`, or `tube` for short; of course, that is what killed Kafka.

Imagine talking to your clients about tech stacks and "we're running tuberculosis" comes up... while people are dying from it.

Re: Show HN: Walrus – a Kafka alternative written in Rust

#15

Post by EdwardDiego »

zbentley wrote: post #6
seanhunter wrote: post #4

Earlier quoted context omitted.

It says on the github page " It provides fault-tolerant streaming with automatic leadership rotation, segment-based partitioning, and Raft consensus for metadata coordination." So I guess that's a "yes" to raft?

GP asked about data plane consensus, not metadata/control plane.

They asked about data plane replication - e.g., leader -> followers. Unless I misunderstood them.

Re: Show HN: Walrus – a Kafka alternative written in Rust

#17

Post by lionkor »

sgt wrote: post #14
lionkor wrote: post #8

Fun anecdote; a couple years ago I started writing a Kafka alternative in C++ with a friend. I got pretty far, but abandoned the project. We called it `tuberculosis`, or `tube` for short; of course, that is what killed Kafka.

Imagine talking to your clients about tech stacks and "we're running tuberculosis" comes up... while people are dying from it.

You just say "well, the alternative was Kafka" and they'd surely get it. Or not. Either way we imagined it to be hilarious.

Re: Show HN: Walrus – a Kafka alternative written in Rust

#18

Post by chaotic-good »

nubskr wrote: post #10
teleforce wrote: post #7

For Kafka alternative written in C++ there's Redpanda [1],[2]. Redpanda claim of better performance but benchmarks showed no clear winner [3]. It will be interesting to test them together on the performance benchmarks. I've got the feeling it's not due to programming language implementation of Scala/Java (Kafka), C++ (Redpanda) and Rust (Walrus). It's the very architecture of Kafka itself due to the notorious head of…

In the current benchmarks, I only have Kafka and rocksdb wal, will surely try to add redpanda there as well, curious how walrus would hold up against seastar based systems.

I don't see any mentions of p99 latency in the benchmark results. Pushing gigabytes per second is not that difficult on modern hardware. Doing so with reasonable latency is what's challenging. Also, instead of using custom benchmarks it's better to just use the OMB (open-messaging benchmark).

Re: Show HN: Walrus – a Kafka alternative written in Rust

#19

Post by [deleted] »

sgt wrote: post #14
lionkor wrote: post #8

Fun anecdote; a couple years ago I started writing a Kafka alternative in C++ with a friend. I got pretty far, but abandoned the project. We called it `tuberculosis`, or `tube` for short; of course, that is what killed Kafka.

Imagine talking to your clients about tech stacks and "we're running tuberculosis" comes up... while people are dying from it.

[deleted]

Re: Show HN: Walrus – a Kafka alternative written in Rust

#20

Post by ramses0 »

lionkor wrote: post #17
sgt wrote: post #14

Earlier quoted context omitted.

Imagine talking to your clients about tech stacks and "we're running tuberculosis" comes up... while people are dying from it.

You just say "well, the alternative was Kafka" and they'd surely get it. Or not. Either way we imagined it to be hilarious.

t10s, pronounced "tíos" or a stuttering "t- tents" on your geo. :-D
Post reply on HN