Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Waltz: A Distributed Write-Ahead Log

wecode.wepay.com

Post reply on HN
Print view View original

21–30 of 46 posts

Re: Waltz: A Distributed Write-Ahead Log

#21

Post by manigandham »

afrodc_ wrote: post #18
manigandham wrote: post #16

Earlier quoted context omitted.

Note that NATS on it's own is just a pub/sub system and doesn't have any queuing or persistence. There is no "log". You need the NATS Streaming server for that which implements persistence while communicating over NATS.

I was looking into https://github.com/liftbridge-io/liftbridge . I believe he was a maintainer of NATS streaming and wanted to make something lightweight and Kafka-esque.

Liftbridge is an experiment and has no real production users. NATS Streaming is already fine for lightweight single-host usage. If you really need to scale to multiple servers then I recommend skipping NATS Streaming and going straight to Kafka or Apache Pulsar.

Re: Waltz: A Distributed Write-Ahead Log

#22

Post by Xorlev »

neuland wrote: post #12

Looks like this also uses Zookeeper. Does anyone know of a simple streaming log system / database? Like, SQLite3 for streaming? I'm using this for personal projects more and more, and the solutions I see in this space are always big, distributed and hard to setup and keep running. I've been using a simple file format that just writes each message out sequentially in a simple format like: [event-id][event-type][event-…

https://github.com/xorlev/slogd

This is a project of mine I use for some side projects for exactly that reason: I want streaming logs & pubsub semantics. It's basically Kafka xtra-lite. It doesn't have a super simple TCP protocol (though I wouldn't say no to a PR adding one!), it's dual-available over gRPC and HTTP (JSON API).

Disclaimer: It's definitely one of my low-activity side projects and subject to change at any time.

Re: Waltz: A Distributed Write-Ahead Log

#23

Post by FpUser »

I have this nasty feeling that for the scale mentioned the whole logic can be safely run on a single high performance server with the decent database. Those little servers around can either run on the same server or placed on separate server/s around the main one. All nice and simple and no distributed transaction problems.

Re: Waltz: A Distributed Write-Ahead Log

#24

Post by nemothekid »

FpUser wrote: post #23

I have this nasty feeling that for the scale mentioned the whole logic can be safely run on a single high performance server with the decent database. Those little servers around can either run on the same server or placed on separate server/s around the main one. All nice and simple and no distributed transaction problems.

I think Waltz is designed for availability, rather than performance.

For a company like Wepay, maybe a couple minutes of downtime is incredibly expensive and having a single server just isn't going to cut it.

Re: Waltz: A Distributed Write-Ahead Log

#25

Post by FpUser »

nemothekid wrote: post #24
FpUser wrote: post #23

I have this nasty feeling that for the scale mentioned the whole logic can be safely run on a single high performance server with the decent database. Those little servers around can either run on the same server or placed on separate server/s around the main one. All nice and simple and no distributed transaction problems.

I think Waltz is designed for availability, rather than performance. For a company like Wepay, maybe a couple minutes of downtime is incredibly expensive and having a single server just isn't going to cut it.

Maybe it is "incredibly expensive", can't argue with this. Have hot standby database servers then located elsewhere. Still way simpler

Re: Waltz: A Distributed Write-Ahead Log

#26

Post by georgewfraser »

This design seems to be an example of a deterministic database system. There's an excellent review of deterministic databases here: http://www.cs.umd.edu/~abadi/papers/abadi-cacm2018.pdf

The core concept of all deterministic databases is simple: if your database is deterministic, multiple geographically-distributed replicas can execute the same transaction log independently, and they will all reach the same state. The problem of implementing a distributed database is reduced to implementing a distributed log.

However, there is one oddity of the Waltz design: the central locking mechanism. Other deterministic databases don't have anything like this, because it's not necessary. You can keep track of locks locally on each replica, and you can rely on every replica to reach the same conclusions about which transactions succeeded and failed, because they are deterministic.

Can anyone clarify why they are managing centralized locks?

Re: Waltz: A Distributed Write-Ahead Log

#28

Post by dmix »

georgewfraser wrote: post #26

This design seems to be an example of a deterministic database system. There's an excellent review of deterministic databases here: http://www.cs.umd.edu/~abadi/papers/abadi-cacm2018.pdf The core concept of all deterministic databases is simple: if your database is deterministic, multiple geographically-distributed replicas can execute the same transaction log independently, and they will all reach the same state. Th…

Daniel Abadi also has an excellent paper on column databases: http://db.csail.mit.edu/pubs/abadi-column-stores.pdf He's a great person at giving high level overviews of database systems before digging into the implementations.

Re: Waltz: A Distributed Write-Ahead Log

#29

Post by atombender »

manigandham wrote: post #17
neuland wrote: post #12

Looks like this also uses Zookeeper. Does anyone know of a simple streaming log system / database? Like, SQLite3 for streaming? I'm using this for personal projects more and more, and the solutions I see in this space are always big, distributed and hard to setup and keep running. I've been using a simple file format that just writes each message out sequentially in a simple format like: [event-id][event-type][event-…

If you're already using the Redis protocol then Redis v5 has Streams as a first-class data structure. It's fast, has consumer groups (like Kafka), and also supports individual message acknowledgement.

Is Redis Streams backed by disk? How does it perform when the size of the stream exceeds available RAM?

Also, what's Redis' multi-node failover/high availability story these days (with streams)? Last I heard, it wasn't that great [1], but it's been a while.

[1] https://aphyr.com/posts/283-jepsen-redis

Post reply on HN