SQLite is 35% Faster Than The Filesystem (2017)
1–10 of 137 posts
Re: SQLite is 35% Faster Than The Filesystem (2017)
#2Locality strikes again!
Re: SQLite is 35% Faster Than The Filesystem (2017)
#3I'd be shocked if this weren't the case.
Re: SQLite is 35% Faster Than The Filesystem (2017)
#4(2017)
Re: SQLite is 35% Faster Than The Filesystem (2017)
#5I wondered the other day how things would behave in node if dependencies were in node_modules.sqlite3 (with posibility to eject to edit if needed).
Re: SQLite is 35% Faster Than The Filesystem (2017)
#6Locality strikes again!
> The performance difference arises (we believe) because when working from an SQLite database, the open() and close() system calls are invoked only once, whereas open() and close() are invoked once for each blob when using blobs stored in individual files. It appears that the overhead of calling open() and close() is greater than the overhead of using the database. The size reduction arises from the fact that individual files are padded out to the next multiple of the filesystem block size, whereas the blobs are packed more tightly into an SQLite database.
Re: SQLite is 35% Faster Than The Filesystem (2017)
#7I wondered the other day how things would behave in node if dependencies were in node_modules.sqlite3 (with posibility to eject to edit if needed).
Shouldn't it be possible to make an sql structure and mount it like an FS to find out?
Re: SQLite is 35% Faster Than The Filesystem (2017)
#8I would love for directories of small files to be stored this way under ZFS; like infrequently updated tar or zip (no compression) files. (Though the filesystem layer of compression might operate on the whole file, or maybe as two streams, one for the file and one for the index.)
Re: SQLite is 35% Faster Than The Filesystem (2017)
#9Is it a known issue that the filesystem on Windows 10 is so slow? Being 5 times slower than macOS was roughly my experience but I thought there was just something wrong with my Windows laptop. I can't find any benchmark or explanation about this.
Re: SQLite is 35% Faster Than The Filesystem (2017)
#10I wondered the other day how things would behave in node if dependencies were in node_modules.sqlite3 (with posibility to eject to edit if needed).
Well pnpm centralizes it so that you’re only referencing a single location via symlinks and that is a major speed up, I think moving dependencies out of the file system altogether would be nice. I’ve explored the possibility of this, but I think the way snowpack does streaming imports[1] in version 3.0 may be the best solution overall. I haven’t spent much time testing it but it appears to be a major process improvement.
[1] https://michalkuncio.com/how-to-use-snowpack-without-node-mo...