Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Waltz: A Distributed Write-Ahead Log

wecode.wepay.com

Post reply on HN
Print view View original

11–20 of 46 posts

Re: Waltz: A Distributed Write-Ahead Log

#11

Post by refset »

elvinyung wrote: post #9

Here's a dumb question about log-structured systems like this: does this system work nicely with backfills? Suppose you start logging events with Waltz and you want to migrate an existing system's data into the same log. Or something goes wrong and oncall needs to manually insert old events. Does Waltz have capabilities to backfill events into the historical log or reassign transaction IDs? This might not be needed i…

These are good reasons to use a bitemporal database [1].

A log gives you "transaction time" but you need to create an efficient representation of "valid time" for backfilling and corrections.

[1] https://en.wikipedia.org/wiki/Temporal_database

Disclosure: I work on a database for Kafka that provides point-in-time bitemporal Datalog queries https://github.com/juxt/crux

Re: Waltz: A Distributed Write-Ahead Log

#12

Post by neuland »

Looks like this also uses Zookeeper. Does anyone know of a simple streaming log system / database? Like, SQLite3 for streaming? I'm using this for personal projects more and more, and the solutions I see in this space are always big, distributed and hard to setup and keep running.

I've been using a simple file format that just writes each message out sequentially in a simple format like: [event-id][event-type][event-size][event-bytes]. And there's a small TCP server that speaks Redis protocol to support remote access. But it's not really production code, rather something I hacked together over a couple weeks in the evening.

Re: Waltz: A Distributed Write-Ahead Log

#13

Post by elvinyung »

refset wrote: post #11
elvinyung wrote: post #9

Here's a dumb question about log-structured systems like this: does this system work nicely with backfills? Suppose you start logging events with Waltz and you want to migrate an existing system's data into the same log. Or something goes wrong and oncall needs to manually insert old events. Does Waltz have capabilities to backfill events into the historical log or reassign transaction IDs? This might not be needed i…

These are good reasons to use a bitemporal database [1]. A log gives you "transaction time" but you need to create an efficient representation of "valid time" for backfilling and corrections. [1] https://en.wikipedia.org/wiki/Temporal_database Disclosure: I work on a database for Kafka that provides point-in-time bitemporal Datalog queries https://github.com/juxt/crux

Yeah, I'm currently getting bitten by various workloads that query by `created_at` instead of a canonical timestamp :) Thanks, I'll take a look!

For the "valid time" primitive I was thinking of implementing something like a hybrid logical clock that CockroachDB has (but with looser guarantees, mostly just need uniqueness and monotonicity). A sequential ID would provide a slightly nicer interface for pagination but has all the problems that I previously mentioned.

Re: Waltz: A Distributed Write-Ahead Log

#14

Post by zcw100 »

neuland wrote: post #12

Looks like this also uses Zookeeper. Does anyone know of a simple streaming log system / database? Like, SQLite3 for streaming? I'm using this for personal projects more and more, and the solutions I see in this space are always big, distributed and hard to setup and keep running. I've been using a simple file format that just writes each message out sequentially in a simple format like: [event-id][event-type][event-…

https://nats.io/

Re: Waltz: A Distributed Write-Ahead Log

#15

Post by elvinyung »

neuland wrote: post #12

Looks like this also uses Zookeeper. Does anyone know of a simple streaming log system / database? Like, SQLite3 for streaming? I'm using this for personal projects more and more, and the solutions I see in this space are always big, distributed and hard to setup and keep running. I've been using a simple file format that just writes each message out sequentially in a simple format like: [event-id][event-type][event-…

Just curious, have you looked into Redis streams?

Re: Waltz: A Distributed Write-Ahead Log

#16

Post by manigandham »

zcw100 wrote: post #14
neuland wrote: post #12

Looks like this also uses Zookeeper. Does anyone know of a simple streaming log system / database? Like, SQLite3 for streaming? I'm using this for personal projects more and more, and the solutions I see in this space are always big, distributed and hard to setup and keep running. I've been using a simple file format that just writes each message out sequentially in a simple format like: [event-id][event-type][event-…

https://nats.io/

Note that NATS on it's own is just a pub/sub system and doesn't have any queuing or persistence. There is no "log". You need the NATS Streaming server for that which implements persistence while communicating over NATS.

Re: Waltz: A Distributed Write-Ahead Log

#17

Post by manigandham »

neuland wrote: post #12

Looks like this also uses Zookeeper. Does anyone know of a simple streaming log system / database? Like, SQLite3 for streaming? I'm using this for personal projects more and more, and the solutions I see in this space are always big, distributed and hard to setup and keep running. I've been using a simple file format that just writes each message out sequentially in a simple format like: [event-id][event-type][event-…

If you're already using the Redis protocol then Redis v5 has Streams as a first-class data structure.

It's fast, has consumer groups (like Kafka), and also supports individual message acknowledgement.

Re: Waltz: A Distributed Write-Ahead Log

#18

Post by afrodc_ »

manigandham wrote: post #16
zcw100 wrote: post #14

Earlier quoted context omitted.

https://nats.io/

Note that NATS on it's own is just a pub/sub system and doesn't have any queuing or persistence. There is no "log". You need the NATS Streaming server for that which implements persistence while communicating over NATS.

I was looking into https://github.com/liftbridge-io/liftbridge. I believe he was a maintainer of NATS streaming and wanted to make something lightweight and Kafka-esque.

Re: Waltz: A Distributed Write-Ahead Log

#19

Post by corford »

afrodc_ wrote: post #18
manigandham wrote: post #16

Earlier quoted context omitted.

Note that NATS on it's own is just a pub/sub system and doesn't have any queuing or persistence. There is no "log". You need the NATS Streaming server for that which implements persistence while communicating over NATS.

I was looking into https://github.com/liftbridge-io/liftbridge . I believe he was a maintainer of NATS streaming and wanted to make something lightweight and Kafka-esque.

Liftbridge looks cool but right now you need use their protobuf defs and cook your own client lib if you want bindings outside of go.

Re: Waltz: A Distributed Write-Ahead Log

#20

Post by corford »

neuland wrote: post #12

Looks like this also uses Zookeeper. Does anyone know of a simple streaming log system / database? Like, SQLite3 for streaming? I'm using this for personal projects more and more, and the solutions I see in this space are always big, distributed and hard to setup and keep running. I've been using a simple file format that just writes each message out sequentially in a simple format like: [event-id][event-type][event-…

Redis streams?
Post reply on HN