21–30 of 68 posts
Go is perhaps not the best tool to use for this as calling sqlite via Cgo will incur a penalty. Might be significant in a very hot loop.
Go is perhaps not the best tool to use for this as calling sqlite via Cgo will incur a penalty. Might be significant in a very hot loop.
Go’s the tool I’m using to build a side project, so I was really only curious about Go + SQLite vs Go + file system.
- https://pkg.go.dev/modernc.org/ql
- https://pkg.go.dev/modernc.org/sqlite
It’s hard to find much information about them, but the first one seems to be a database similar to sqlite, and the latter seems to be some kind of automated translation of sqlite to Go. I’m not sure either of these will outperform sqlite on CGo, but both of them should eliminate the CGo overhead and might give you some more room.
P.S.: The latter is a bit unorthodox in that it contains separate code for each architecture supported by Go, most likely a reflection of the method of the automated translation.
Earlier quoted context omitted.
If you only care about storing data and not doing any complex operations or query against it why are you using something like sqlite as well? It’s not a comparison as being in sqlite makes the ability to access this data significantly easier. This is comparing apples and dogs and i don’t see the merits.
In one application I use sqlite to store JPEG thumbnails. That's it; there's nothing else in there. It's super handy; I specifically needed to reduce the number of files that I open and close because that's slow on Windows/NTFS. SQLite made this trivial. I could have managed a binary pack format on my own but I didn't have to.
Earlier quoted context omitted.
Go’s the tool I’m using to build a side project, so I was really only curious about Go + SQLite vs Go + file system.
You might find these projects interesting: - https://pkg.go.dev/modernc.org/ql - https://pkg.go.dev/modernc.org/sqlite It’s hard to find much information about them, but the first one seems to be a database similar to sqlite, and the latter seems to be some kind of automated translation of sqlite to Go. I’m not sure either of these will outperform sqlite on CGo, but both of them should eliminate the CGo overhead and…
How about a fast filesystem? This is by far the slowest.
Edit: just ran it on EXT4 on Linode.
SQLite: 8.7s
EXT4: 18.3s
A quick google shows there are few FUSE SQLite implementations. Then you can use grep, ls, etc
Actually, having a sqlite filesystem is intriguing because you could in theory add any kind of metadata or filesystem feature you wanted to (such as a forward-error-correcting checksum field, auto-compressed/decompressed data, dedup, encryption, etc.) It would make it nearly trivial for anyone to experiment with new filesystems.
Earlier quoted context omitted.
In one application I use sqlite to store JPEG thumbnails. That's it; there's nothing else in there. It's super handy; I specifically needed to reduce the number of files that I open and close because that's slow on Windows/NTFS. SQLite made this trivial. I could have managed a binary pack format on my own but I didn't have to.
You could have easily done that with a single file with a tail header for look ups.
SQLite advertises itself as an fopen replacement. Sounds like a perfect match for parent’s use case.
Earlier quoted context omitted.
If you only care about storing data and not doing any complex operations or query against it why are you using something like sqlite as well? It’s not a comparison as being in sqlite makes the ability to access this data significantly easier. This is comparing apples and dogs and i don’t see the merits.
In one application I use sqlite to store JPEG thumbnails. That's it; there's nothing else in there. It's super handy; I specifically needed to reduce the number of files that I open and close because that's slow on Windows/NTFS. SQLite made this trivial. I could have managed a binary pack format on my own but I didn't have to.
But sqlite is still better, it is more reliable, a bad write on that end of zip index destroys the whole zip archive and sqlite also gives you a lot more potential flexibility later on if you need to add metadata or something else. It is better in terms of inserts and deletes, although you will still need to vacuum.
Earlier quoted context omitted.
In one application I use sqlite to store JPEG thumbnails. That's it; there's nothing else in there. It's super handy; I specifically needed to reduce the number of files that I open and close because that's slow on Windows/NTFS. SQLite made this trivial. I could have managed a binary pack format on my own but I didn't have to.
Another alternative that I've seen used is a zip or tar with no compression if you are just appending files and reading but only rarely updating or deleting. But sqlite is still better, it is more reliable, a bad write on that end of zip index destroys the whole zip archive and sqlite also gives you a lot more potential flexibility later on if you need to add metadata or something else. It is better in terms of inser…
For this situation it does matter, but it is recoverable.