"To blob or not to blob" - a paper by Microsoft Research from 2006. https://www.microsoft.com/en-us/research/publication/to-blob...
SQLite small blob storage: 35% Faster Than the Filesystem
51–60 of 208 posts
Re: SQLite small blob storage: 35% Faster Than the Filesystem
#52Earlier quoted context omitted.
Of course it matters. Different file systems have different performance characteristics. Make sqlite into a file system, and I'm pretty sure you can't beat it by putting another sqlite on top.
When the author has tested 3 or 4 different filesystems against SQLite (and at least 2 of them you can be pretty sure as to what they are simply based on the OS, with a good guess at the other 2 as well), and SQLite beats them all, then it really doesn't matter. As for turning SQLite into a filesystem, that's not really going to work. SQL isn't designed to support things like cheap "file" appends or reading only port…
As for benchmarking multiple file systems with consistent result, point taken. To a degree. Some file systems optimize for directory lookups, and some don't.
As for turning SQLite into a filesystem, we could make it into a file system that would be fast in this particular benchmark. Right?
Re: SQLite small blob storage: 35% Faster Than the Filesystem
#53Earlier quoted context omitted.
> Writing your own methods to retrieve arbitrary chunks of data from a monolithic file would be a lot of work. Oh goodness have we come a long way if interacting with a file is harder than interacting with a SQL database. (it really depends on what language and/or library(ies) you use and for what purpose)
As far as I know, querying a database has always been easier than querying a custom file format. It's the whole reason databases exist in the first place. And using one off the shelf is clearly easier than writing your own.
Not easier, but vastly more flexible.
Sure, it's completely trivial to read a file containing an array of C structs of metadata and offsets into a blob of concatenated PNGs: just mmap the thing, and your main data structure is sitting right there in memory.
But problems start arising when you need to update the structure of that metadata. Add some more fields. Some more structured, multi-value fields, referring to multiple entities in your increasingly complex data structure. Entities which must be guaranteed to exist. And perform modifications concurrently, from multiple processes. And have the file not get corrupted when something kills a writer process in the middle of writing...
Databases are slow and overkill for the simple case, but they certainly make adding unexpected requested features much easier. And it can be expected that you will get unexpected feature requests.
Re: SQLite small blob storage: 35% Faster Than the Filesystem
#5435% faster than the same blobs can be read from or written to individual files on disk using fread() or fwrite().
How about reading efficiently with unbuffered I/O directly on the file descriptor?
Re: SQLite small blob storage: 35% Faster Than the Filesystem
#55This is weird benchmarketing. They are comparing reading/writing 100,000 individual files vs writing 100,000 entries into a single file(sqlite database). For comparison one could concatenate the same data into a single big file even faster than into sqlite. They then do not offer logical analysis as to why things are faster. My understanding is that reads are probably faster due to operating system readahead being ab…
Why stop there? One could use the raw device to get even greater performance!
A database on a raw device is a thing. E.g. Oracle has this option for decades; MySQL, for some time, too.
Re: SQLite small blob storage: 35% Faster Than the Filesystem
#56During my first week at American Express I did not have the credentials to install any applications. My manager wanted me to build a prototype and I couldn't install a database. Firebase and other third party was impossible because I had to work on an internal server. I thought SQLite would save my day, but I remembered it being really hard to install and set up. So I had to write my own DB that I called 'stupid-db'. It literally just reads/writes data into a file: https://github.com/songz/stupid-db
The demo went well.
Re: SQLite small blob storage: 35% Faster Than the Filesystem
#57Earlier quoted context omitted.
> You’d end up rewriting your own version of a database. This is the hell we just crawled out of in one of our codebases.. it's not so much the access for a single process, but when you want to share the data in the file, that's when it seriously starts to get messy and breaking through otherwise clean abstraction layers and/or forcing recompile on compatibility breaking changes.
If you don't mind answering, out of sheer curiosity how was the data stored in the individual file/s?
Straight array of structs. The complication was that there were several of them, they were stored in bus-attached MRAM which was accessed directly (e.g. mmap of /dev/ram) by several independent processes.
The idea came from previous generations of the product which generally ran on microcontrollers coupled with custom FPGA designs, so, not entirely unwarranted.. but completely incorrect for a newer product with much greater computing power than their previous devices.
Re: SQLite small blob storage: 35% Faster Than the Filesystem
#58Earlier quoted context omitted.
Writing your own methods to retrieve arbitrary chunks of data from a monolithic file would be a lot of work. Using map tiles as the example, how do you easily retrieve just the tiles in a specific region? Ok, how about all tiles that have the “hasLand” attribute set? Or the “containsCoastline” attribute? You’d end up rewriting your own version of a database.
> Writing your own methods to retrieve arbitrary chunks of data from a monolithic file would be a lot of work. Oh goodness have we come a long way if interacting with a file is harder than interacting with a SQL database. (it really depends on what language and/or library(ies) you use and for what purpose)
> Oh goodness have we come a long way if interacting with a file is harder than interacting with a SQL database.
It's not interacting with a file, it's interacting with a file that contains other files of varying sizes that need to be accessed randomly with good performance.
Re: SQLite small blob storage: 35% Faster Than the Filesystem
#59Can we draw a side conclusion from these measurements that open()/close() overhead on Win10 is significantly higher than on Win7? This seems doubtful.
Re: SQLite small blob storage: 35% Faster Than the Filesystem
#60I mean, is SQLite not also using the filesystem? Maybe I'm confused here. Seems like saying using C is 15% faster than ASM. But that includes user error, right?