41–50 of 68 posts
How about a fast filesystem? This is by far the slowest.
That’s fair. It’s the file system I have easy access to. I should spin up a VPS and use ext4 or xfs. I may do that when I have some spare time. It looks like I may get double the perf by making that switch. SQLite also seems to get a bump on different file systems. Edit: just ran it on EXT4 on Linode. SQLite: 8.7s EXT4: 18.3s
Earlier quoted context omitted.
Yeah. I mentioned that in the final test where I write to a temporary file, then rename. This is much slower, probably due to an implicit fsync.
You can disable the fsync calls in sqlite if you wanted to do a little better with this benchmark. You're also explicitly choosing the WAL in your go-sqlite3 configuration which is not at all replicated by your filesystem test. I think, honestly, that you're just going to mislead and confuse people who don't know any better with this writeup. I can write faster to /dev/null, too, but that isn't a very interesting com…
Earlier quoted context omitted.
In one application I use sqlite to store JPEG thumbnails. That's it; there's nothing else in there. It's super handy; I specifically needed to reduce the number of files that I open and close because that's slow on Windows/NTFS. SQLite made this trivial. I could have managed a binary pack format on my own but I didn't have to.
You could have easily done that with a single file with a tail header for look ups.
Earlier quoted context omitted.
Easier than SQLite? With less bugs and better tests? SQLite advertises itself as an fopen replacement. Sounds like a perfect match for parent’s use case.
Depends on the indexing needs, right? Something like IFF should be fine if you need a sequence of images to store that you don’t need to actually query for individually.
I couldn't see any fsync (or similar) in your filesystem code (sorry if I missed them) -- it doesn't seem like a completely fair comparison, as sqlite has various promises about commits being completely made, etc, which you won't get from just chucking files on a filesystem.
If you only care about storing data and not doing any complex operations or query against it why are you using something like sqlite as well? It’s not a comparison as being in sqlite makes the ability to access this data significantly easier. This is comparing apples and dogs and i don’t see the merits.
Earlier quoted context omitted.
You could have easily done that with a single file with a tail header for look ups.
Easier than SQLite? With less bugs and better tests? SQLite advertises itself as an fopen replacement. Sounds like a perfect match for parent’s use case.
Earlier quoted context omitted.
You can disable the fsync calls in sqlite if you wanted to do a little better with this benchmark. You're also explicitly choosing the WAL in your go-sqlite3 configuration which is not at all replicated by your filesystem test. I think, honestly, that you're just going to mislead and confuse people who don't know any better with this writeup. I can write faster to /dev/null, too, but that isn't a very interesting com…
Does /dev/null support sharding?
Earlier quoted context omitted.
Go’s the tool I’m using to build a side project, so I was really only curious about Go + SQLite vs Go + file system.
depending on your data, there's also Keyvalue stores like boltdb & badger.
I couldn't see any fsync (or similar) in your filesystem code (sorry if I missed them) -- it doesn't seem like a completely fair comparison, as sqlite has various promises about commits being completely made, etc, which you won't get from just chucking files on a filesystem.
Yeah. I mentioned that in the final test where I write to a temporary file, then rename. This is much slower, probably due to an implicit fsync.
I couldn't see any fsync (or similar) in your filesystem code (sorry if I missed them) -- it doesn't seem like a completely fair comparison, as sqlite has various promises about commits being completely made, etc, which you won't get from just chucking files on a filesystem.
Yeah. I mentioned that in the final test where I write to a temporary file, then rename. This is much slower, probably due to an implicit fsync.