in line 200 of rebuf.go, did you mean to return err (instead of returning nil even when an error occurs)?
I guess so! Will be fixed
Show HN: WAL Implementation in Golang
21–30 of 30 posts
Re: Show HN: WAL Implementation in Golang
#22This reminds me of ZoneTree which is persistent LSM tree project based on top of WAL, written in C#: https://github.com/koculu/ZoneTree Similar to RocksDB.
Absolutely loved your project ZoneTree. Gotta dig deep on this
Re: Show HN: WAL Implementation in Golang
#23Cool library. Two small generic Go library issues: 1. The rebuf.Init function panics. I almost never want a library to call panic, and when it does, I want the library function to denote that. The convention I’ve seen most often is to start the function name with Must, so MustInit instead of Init. In this case though, I think it’d be safe to be a little more lenient in what you accept as input and trim the trailing s…
1. You're right! Will fix it to handle this as well as the support for relative directories 2. Yes, Will integrate a logging library instead of fmt ( https://github.com/uber-go/zap )
What if I don't use zap?
Ideally you want to add an option via a function or interface:
func New(optOne bool, log func(string, ...any)) {
if log == nil {
log = func(m string, a ...any) { fmt.Printf(m, a...) }
}
log("starting; optOne=%v", optOne)
}
Re: Show HN: WAL Implementation in Golang
#24Cool library. Two small generic Go library issues: 1. The rebuf.Init function panics. I almost never want a library to call panic, and when it does, I want the library function to denote that. The convention I’ve seen most often is to start the function name with Must, so MustInit instead of Init. In this case though, I think it’d be safe to be a little more lenient in what you accept as input and trim the trailing s…
1. You're right! Will fix it to handle this as well as the support for relative directories 2. Yes, Will integrate a logging library instead of fmt ( https://github.com/uber-go/zap )
FYI, Go already has a structured logging package called [slog](https://go.dev/blog/slog) in the standard library since V1.21.
Re: Show HN: WAL Implementation in Golang
#25Earlier quoted context omitted.
1. You're right! Will fix it to handle this as well as the support for relative directories 2. Yes, Will integrate a logging library instead of fmt ( https://github.com/uber-go/zap )
What if I don't use zap? Ideally you want to add an option via a function or interface: func New(optOne bool, log func(string, ...any)) { if log == nil { log = func(m string, a ...any) { fmt.Printf(m, a...) } } log("starting; optOne=%v", optOne) } Or something along those lines.
This is what log/slog is for!
Re: Show HN: WAL Implementation in Golang
#26Another simple WAL: https://pkg.go.dev/modernc.org/file#WAL
Re: Show HN: WAL Implementation in Golang
#27Having written one of these, a few optimizations will go a long way: 1. syscall.Iovec allows you to build up multiple batches semi independently and then write them all in a single syscall and sync the file with the next one. It is a good basis for allowing multiple pending writes to proceed in independent go routines and have another one have all the responsibility for flushing data. 2. It is better to use larger pr…
Thanks! Could you please point me to a reference for (1) etcd/wal actually does do preallocations ( https://github.com/etcd-io/etcd/blob/24e05998c68f481af2bd567... ) Yet to implement max buffer age! Any references for this would be bomb! Is mmap() really needed here? Came across a similar project that does this? Really gotta dig deep here! https://github.com/jhunters/bigqueue
Can't share my references with you directly, the implementation I wrote is closed-source and is heavily intermingled with other internal bits. But I can provide examples:
1. syscall.Iovec is a struct that the writev() systemcall uses. You build it up something like this:
func b2iov(bs [][]byte) []syscall.Iovec {
res := []syscall.Iovec{}
for i := range bs {
res = append(res, syscall.Iovec{Base: &bs[i][0], Len: uint64(len(bs[i])}
}
return res
}
func write(fi *os.File, iov []syscall.Iovec, at int64) (written int64, err error) {
if _, err = fi.Seek(at, io.SeekStart); err != nil {
return
}
wr, _, errno := syscall.Syscall(syscall.SYS_WRITEV, fi.Fd(), uintptr(unsafe.Pointer(&iov[0])), uintptr(len(iov)))
if errno != 0 {
err = errno
return
}
written = int64(wr)
err = fi.Sync()
return
}
2. Yeah, I just zero-filled a new file using the fallocate as well.
3. I handled max buffer age by feeding writes to the WAL using a channel, then the main reader loop for that channel select on both the main channel and a time.Timer.C channel. Get clever with the Reset() method on that timer and you can implement whatever timeout scheme you like.
4. No, it is not needed, but my WAL implementation boiled down to a bunch of byte buffers protected by a rolling CRC64, and for me just mmap'ing the whole file into a big slice and sanity-checking the rolling crcs along with other metadata was easier and faster that way.
Re: Show HN: WAL Implementation in Golang
#28Earlier quoted context omitted.
What if I don't use zap? Ideally you want to add an option via a function or interface: func New(optOne bool, log func(string, ...any)) { if log == nil { log = func(m string, a ...any) { fmt.Printf(m, a...) } } log("starting; optOne=%v", optOne) } Or something along those lines.
This is what log/slog is for!
What if I don't use slog?
Re: Show HN: WAL Implementation in Golang
#29Re: Show HN: WAL Implementation in Golang
#30Cool library. Two small generic Go library issues: 1. The rebuf.Init function panics. I almost never want a library to call panic, and when it does, I want the library function to denote that. The convention I’ve seen most often is to start the function name with Must, so MustInit instead of Init. In this case though, I think it’d be safe to be a little more lenient in what you accept as input and trim the trailing s…
1. You're right! Will fix it to handle this as well as the support for relative directories 2. Yes, Will integrate a logging library instead of fmt ( https://github.com/uber-go/zap )
Based on the child thread about zap vs slog I think I might not have been clear in my phrasing. The issue isn’t the specific functions used to print to the screen, it’s that library code is doing it at all. As the user of a library, I don’t want that library printing things to the screen if I don’t explicitly tell it to; decisions on logging/printing text to the screen are the responsibility of the person writing the end-user application code, not the library author. If the library author feels really strongly about printing stuff on the screen, they should make that behavior opt in, either with a configuration option or by providing some other mechanism that gives the user as much control over that behavior as possible (hence my example of throwing printing behavior into a user-supplied io.Writer)