An unscientific benchmark of SQLite vs. the file system (btrfs)
1–10 of 68 posts
Re: An unscientific benchmark of SQLite vs. the file system (btrfs)
#2I’m the author. Out of curiosity, how did you find and post this before I did?
Re: An unscientific benchmark of SQLite vs. the file system (btrfs)
#3I couldn't see any fsync (or similar) in your filesystem code (sorry if I missed them) -- it doesn't seem like a completely fair comparison, as sqlite has various promises about commits being completely made, etc, which you won't get from just chucking files on a filesystem.
Re: An unscientific benchmark of SQLite vs. the file system (btrfs)
#4I couldn't see any fsync (or similar) in your filesystem code (sorry if I missed them) -- it doesn't seem like a completely fair comparison, as sqlite has various promises about commits being completely made, etc, which you won't get from just chucking files on a filesystem.
If you only care about storing data and not doing any complex operations or query against it why are you using something like sqlite as well?
It’s not a comparison as being in sqlite makes the ability to access this data significantly easier. This is comparing apples and dogs and i don’t see the merits.
Re: An unscientific benchmark of SQLite vs. the file system (btrfs)
#5If you're using the same file, it would make sense if the db 'warms up' by growing enough to comfortably hold all your data.
Re: An unscientific benchmark of SQLite vs. the file system (btrfs)
#6I couldn't see any fsync (or similar) in your filesystem code (sorry if I missed them) -- it doesn't seem like a completely fair comparison, as sqlite has various promises about commits being completely made, etc, which you won't get from just chucking files on a filesystem.
If you only care about storing data and not doing any complex operations or query against it why are you using something like sqlite as well? It’s not a comparison as being in sqlite makes the ability to access this data significantly easier. This is comparing apples and dogs and i don’t see the merits.
It's not uncommon to see people advocate for using sqlite as an alternative to flat files or json blobs. In particular, transactions provide nice properties vs a web of separate flat files, and the stronger schema can be a good alternative to json. There's a lot of great existing sqlite tooling, too.
Re: An unscientific benchmark of SQLite vs. the file system (btrfs)
#7Earlier quoted context omitted.
If you only care about storing data and not doing any complex operations or query against it why are you using something like sqlite as well? It’s not a comparison as being in sqlite makes the ability to access this data significantly easier. This is comparing apples and dogs and i don’t see the merits.
It's not uncommon to see people advocate for using sqlite as an alternative to flat files or json blobs. In particular, transactions provide nice properties vs a web of separate flat files, and the stronger schema can be a good alternative to json. There's a lot of great existing sqlite tooling, too.
Not only other people, the creators also advocate for it as a file format:
Re: An unscientific benchmark of SQLite vs. the file system (btrfs)
#8I couldn't see any fsync (or similar) in your filesystem code (sorry if I missed them) -- it doesn't seem like a completely fair comparison, as sqlite has various promises about commits being completely made, etc, which you won't get from just chucking files on a filesystem.
If you only care about storing data and not doing any complex operations or query against it why are you using something like sqlite as well? It’s not a comparison as being in sqlite makes the ability to access this data significantly easier. This is comparing apples and dogs and i don’t see the merits.
In one application I use sqlite to store JPEG thumbnails. That's it; there's nothing else in there. It's super handy; I specifically needed to reduce the number of files that I open and close because that's slow on Windows/NTFS. SQLite made this trivial. I could have managed a binary pack format on my own but I didn't have to.
Re: An unscientific benchmark of SQLite vs. the file system (btrfs)
#9I couldn't see any fsync (or similar) in your filesystem code (sorry if I missed them) -- it doesn't seem like a completely fair comparison, as sqlite has various promises about commits being completely made, etc, which you won't get from just chucking files on a filesystem.
Yeah. I mentioned that in the final test where I write to a temporary file, then rename. This is much slower, probably due to an implicit fsync.
Re: An unscientific benchmark of SQLite vs. the file system (btrfs)
#10I couldn't see any fsync (or similar) in your filesystem code (sorry if I missed them) -- it doesn't seem like a completely fair comparison, as sqlite has various promises about commits being completely made, etc, which you won't get from just chucking files on a filesystem.
Yeah. I mentioned that in the final test where I write to a temporary file, then rename. This is much slower, probably due to an implicit fsync.
You can disable the fsync calls in sqlite if you wanted to do a little better with this benchmark. You're also explicitly choosing the WAL in your go-sqlite3 configuration which is not at all replicated by your filesystem test. I think, honestly, that you're just going to mislead and confuse people who don't know any better with this writeup. I can write faster to /dev/null, too, but that isn't a very interesting comparison.