Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Waltz: A Distributed Write-Ahead Log

wecode.wepay.com

Post reply on HN
Print view View original

31–40 of 46 posts

Re: Waltz: A Distributed Write-Ahead Log

#31

Post by dmix »

manigandham wrote: post #21
afrodc_ wrote: post #18

Earlier quoted context omitted.

I was looking into https://github.com/liftbridge-io/liftbridge . I believe he was a maintainer of NATS streaming and wanted to make something lightweight and Kafka-esque.

Liftbridge is an experiment and has no real production users. NATS Streaming is already fine for lightweight single-host usage. If you really need to scale to multiple servers then I recommend skipping NATS Streaming and going straight to Kafka or Apache Pulsar.

Have any more lighter-weight recommendations that fit the queuing and persistence category?

Re: Waltz: A Distributed Write-Ahead Log

#32

Post by hvidgaard »

georgewfraser wrote: post #26

This design seems to be an example of a deterministic database system. There's an excellent review of deterministic databases here: http://www.cs.umd.edu/~abadi/papers/abadi-cacm2018.pdf The core concept of all deterministic databases is simple: if your database is deterministic, multiple geographically-distributed replicas can execute the same transaction log independently, and they will all reach the same state. Th…

I have not gone into this at all, but my initial thought is that as long as every log entry also have a unique id of the last log entry at the time of it's writing, you can always piece together a deterministic log without any locking or voting. Just share everything you know.

Re: Waltz: A Distributed Write-Ahead Log

#33

Post by majidazimi »

georgewfraser wrote: post #26

This design seems to be an example of a deterministic database system. There's an excellent review of deterministic databases here: http://www.cs.umd.edu/~abadi/papers/abadi-cacm2018.pdf The core concept of all deterministic databases is simple: if your database is deterministic, multiple geographically-distributed replicas can execute the same transaction log independently, and they will all reach the same state. Th…

Essentially if each instance can keep track of validating transactions locally, the whole thing can be easily implemented with Bookkeeper and distributed log API [0]. Bookeeper guarantees single writer (the master node which is appending to log) for each ledger. All it needs to be done is to put concurrency control metadata (like version, timestamp, ...) inside the message so each node can validate stuff locally.

[0] https://bookkeeper.apache.org/docs/4.9.2/api/distributedlog-...

Re: Waltz: A Distributed Write-Ahead Log

#34

Post by james-mcelwain »

neuland wrote: post #12

Looks like this also uses Zookeeper. Does anyone know of a simple streaming log system / database? Like, SQLite3 for streaming? I'm using this for personal projects more and more, and the solutions I see in this space are always big, distributed and hard to setup and keep running. I've been using a simple file format that just writes each message out sequentially in a simple format like: [event-id][event-type][event-…

Kafka is moving towards eliminating the ZK dependency, while will make it easier to run as a single node if you don't care about HA.

Re: Waltz: A Distributed Write-Ahead Log

#35

Post by graetzer »

neuland wrote: post #12

Looks like this also uses Zookeeper. Does anyone know of a simple streaming log system / database? Like, SQLite3 for streaming? I'm using this for personal projects more and more, and the solutions I see in this space are always big, distributed and hard to setup and keep running. I've been using a simple file format that just writes each message out sequentially in a simple format like: [event-id][event-type][event-…

Maybe the pub-sub pattern of https://zeromq.org/ could work for that

Re: Waltz: A Distributed Write-Ahead Log

#36

Post by tveita »

docker_up wrote: post #7

Last I heard from some friends in YC when considering a position there, Wepay was handling less than 10^6 payments per day. Is that still the case and is something with such low requirements a good replacement for Kafka in the wild?

Your comment made me realize that I had confused WePay with WeChat Pay, which has a slightly different scale. The "We$VERB" field is getting a bit crowded.

Re: Waltz: A Distributed Write-Ahead Log

#37

Post by nemothekid »

FpUser wrote: post #25
nemothekid wrote: post #24

Earlier quoted context omitted.

I think Waltz is designed for availability, rather than performance. For a company like Wepay, maybe a couple minutes of downtime is incredibly expensive and having a single server just isn't going to cut it.

Maybe it is "incredibly expensive", can't argue with this. Have hot standby database servers then located elsewhere. Still way simpler

>Have hot standby database servers then located elsewhere

They explained why this wasn't ideal for them in the article.

Re: Waltz: A Distributed Write-Ahead Log

#38

Post by orthecreedence »

james-mcelwain wrote: post #34
neuland wrote: post #12

Looks like this also uses Zookeeper. Does anyone know of a simple streaming log system / database? Like, SQLite3 for streaming? I'm using this for personal projects more and more, and the solutions I see in this space are always big, distributed and hard to setup and keep running. I've been using a simple file format that just writes each message out sequentially in a simple format like: [event-id][event-type][event-…

Kafka is moving towards eliminating the ZK dependency, while will make it easier to run as a single node if you don't care about HA.

Hey, is there a tracker somewhere for this? I've always been into trying Kafka, but I heard horror stories about ZK so having it removed would be nice.

Re: Waltz: A Distributed Write-Ahead Log

#39

Post by manigandham »

atombender wrote: post #29
manigandham wrote: post #17

Earlier quoted context omitted.

If you're already using the Redis protocol then Redis v5 has Streams as a first-class data structure. It's fast, has consumer groups (like Kafka), and also supports individual message acknowledgement.

Is Redis Streams backed by disk? How does it perform when the size of the stream exceeds available RAM? Also, what's Redis' multi-node failover/high availability story these days (with streams)? Last I heard, it wasn't that great [1], but it's been a while. [1] https://aphyr.com/posts/283-jepsen-redis

Redis is an in-memory data store that has different options for persistence (snapshots + oplog) but it's not designed to persist every operation immediately. All data structures are covered including Streams.

Redis keeps the entire working set in RAM so it'll start dropping writes or freeze if you run out of memory. This is where the simplicity and speed comes from and is a fundamental limitation.

There's a simple replica system that works well but failover switching is the problem and requires a separate process or running the Redis Sentinel. There's also Redis Cluster but that just shards the keyspace and doesn't offer any scalability with a single key or stream, and is still hard to manage with the same failover issues.

The OP asked for a single-node option so I suggested Redis, if you need a serious messaging cluster then I recommend Kafka or Pulsar instead.

Re: Waltz: A Distributed Write-Ahead Log

#40

Post by atombender »

manigandham wrote: post #39
atombender wrote: post #29

Earlier quoted context omitted.

Is Redis Streams backed by disk? How does it perform when the size of the stream exceeds available RAM? Also, what's Redis' multi-node failover/high availability story these days (with streams)? Last I heard, it wasn't that great [1], but it's been a while. [1] https://aphyr.com/posts/283-jepsen-redis

Redis is an in-memory data store that has different options for persistence (snapshots + oplog) but it's not designed to persist every operation immediately. All data structures are covered including Streams. Redis keeps the entire working set in RAM so it'll start dropping writes or freeze if you run out of memory. This is where the simplicity and speed comes from and is a fundamental limitation. There's a simple re…

Thanks. That confirms pretty much the picture I had in my head of how Redis works these days.

It's frustrating that there's no obvious middle ground this and Kafka and Pulsar, both of which are memory-hungry JVM apps with multiple external dependencies. Both require ZooKeeper; Pulsar also requires BookKeeper. None of these components are operationally simple.

I'm a fan of NATS itself, but NATS Streaming's clustering design leaves a lot to be desired. In particular, it punts on failover/HA, asking you to instead run an SQL database or shared file system that provides this. (An obvious low-maintenance option here would be CockroachDB, but NATS doesn't support it.)

Post reply on HN