Show HN: WAL Implementation in Golang

#1

Post by stym06 » Mon, Jul 08, 2024, 6:50 PM UTC

I wrote this simple WAL library in Golang that I use to write data that my kafka producer fails due to errors like Broker going down or some other issue. Took inspiration from etcd/wal

Show HN: WAL Implementation in Golang
github.com

Re: Show HN: WAL Implementation in Golang

#2

Post by eatonphil » Mon, Jul 08, 2024, 7:19 PM UTC

Did I miss it or is there no call to os.File.Sync(), i.e. fsync, anywhere?

Since you mention etcd/wal:

https://github.com/etcd-io/etcd/blob/v3.3.27/wal/wal.go#L671

https://github.com/etcd-io/etcd/blob/v3.3.27/pkg/fileutil/sy...

Re: Show HN: WAL Implementation in Golang

#3

Post by stym06 » Mon, Jul 08, 2024, 7:24 PM UTC

eatonphil wrote: post #2

Did I miss it or is there no call to os.File.Sync(), i.e. fsync, anywhere? Since you mention etcd/wal: https://github.com/etcd-io/etcd/blob/v3.3.27/wal/wal.go#L671 https://github.com/etcd-io/etcd/blob/v3.3.27/pkg/fileutil/sy...

I used the bufio flush mechanism https://pkg.go.dev/bufio#Writer.Flush

Thanks for your comment, I'll definitely check it out. It was my first attempt at this. How can I make it better?

Re: Show HN: WAL Implementation in Golang

#4

Post by sakras » Mon, Jul 08, 2024, 7:54 PM UTC

stym06 wrote: post #3

eatonphil wrote: post #2

Did I miss it or is there no call to os.File.Sync(), i.e. fsync, anywhere? Since you mention etcd/wal: https://github.com/etcd-io/etcd/blob/v3.3.27/wal/wal.go#L671 https://github.com/etcd-io/etcd/blob/v3.3.27/pkg/fileutil/sy...

I used the bufio flush mechanism https://pkg.go.dev/bufio#Writer.Flush Thanks for your comment, I'll definitely check it out. It was my first attempt at this. How can I make it better?

Essentially, unless you `fsync`, there's no guarantee that your data will be durably written to disk. This is because the operating system keeps data buffered in in-memory caches, so if the machine crashes you may lose some data. The `fsync` system call forces the data to be flushed from the in-memory OS cache to the disk. As far as I could tell, the Flush you use does not `fsync`.

Re: Show HN: WAL Implementation in Golang

#5

Post by tjungblut » Mon, Jul 08, 2024, 7:57 PM UTC

Besides what Phil mentioned below, I can't write more than one record to the WAL. You're closing the file after every write, the second time you write the error `seek data/rebuf.tmp: file already closed` is returned.

I also think your rotation will delete the wrong segment when you have more than ten segments - imagine you're writing rebuf-1 to rebuf-10 - what's the "oldest file" to delete now? Besides, should you really delete those files?

Re: Show HN: WAL Implementation in Golang

#6

Post by stym06 » Mon, Jul 08, 2024, 8:06 PM UTC

tjungblut wrote: post #5

Besides what Phil mentioned below, I can't write more than one record to the WAL. You're closing the file after every write, the second time you write the error `seek data/rebuf.tmp: file already closed` is returned. I also think your rotation will delete the wrong segment when you have more than ten segments - imagine you're writing rebuf-1 to rebuf-10 - what's the "oldest file" to delete now? Besides, should you re…

Yes there are a lot of bugs since I just wrote this in one sitting today. Will be fixing all of this. For log rotation, I'll sort by the last_modified_at ts and then purge those

Re: Show HN: WAL Implementation in Golang

#7

Post by stym06 » Mon, Jul 08, 2024, 8:07 PM UTC

sakras wrote: post #4

stym06 wrote: post #3

Earlier quoted context omitted.

I used the bufio flush mechanism https://pkg.go.dev/bufio#Writer.Flush Thanks for your comment, I'll definitely check it out. It was my first attempt at this. How can I make it better?

Essentially, unless you `fsync`, there's no guarantee that your data will be durably written to disk. This is because the operating system keeps data buffered in in-memory caches, so if the machine crashes you may lose some data. The `fsync` system call forces the data to be flushed from the in-memory OS cache to the disk. As far as I could tell, the Flush you use does not `fsync`.

Thanks for your input @sakras. I'll fix this

Re: Show HN: WAL Implementation in Golang

#8

Post by stym06 » Mon, Jul 08, 2024, 8:09 PM UTC

OP here! Pls feel free to raise any bugs you encounter! I'll be doing the following immmediate fixes:

1. Use fsync for durable writes in case of system crashes

2. Fix log-rotation-purging logic

3. Fix `file already closed` bug on consecutive writes

4. Add CRC checksum

Re: Show HN: WAL Implementation in Golang

#9

Post by neonsunset » Mon, Jul 08, 2024, 8:24 PM UTC

This reminds me of ZoneTree which is persistent LSM tree project based on top of WAL, written in C#: https://github.com/koculu/ZoneTree

Similar to RocksDB.

Re: Show HN: WAL Implementation in Golang

#10

Post by tjungblut » Mon, Jul 08, 2024, 8:38 PM UTC

stym06 wrote: post #6

tjungblut wrote: post #5

Besides what Phil mentioned below, I can't write more than one record to the WAL. You're closing the file after every write, the second time you write the error `seek data/rebuf.tmp: file already closed` is returned. I also think your rotation will delete the wrong segment when you have more than ten segments - imagine you're writing rebuf-1 to rebuf-10 - what's the "oldest file" to delete now? Besides, should you re…

Yes there are a lot of bugs since I just wrote this in one sitting today. Will be fixing all of this. For log rotation, I'll sort by the last_modified_at ts and then purge those

Your generational approach to segment numbering is fine, if you prepend enough zeros to format the files properly then you're also able to sort them correctly. etcd uses the same trick.