I wrote an ORM to use SQLite for serializing/persisting objects to disk (i.e. using SQLite DB as a file format).
This was one of the reasons why SQLite was an easy choice.
SQLite is 35% Faster Than The Filesystem (2017)
21–30 of 137 posts
Re: SQLite is 35% Faster Than The Filesystem (2017)
#22then making a small cdn using nginx and sqlite can be a thing?
but for a CDN, you'd probably have caching. that would eliminate the need to take advantage of this kind of optimization
Re: SQLite is 35% Faster Than The Filesystem (2017)
#23Is it a known issue that the filesystem on Windows 10 is so slow? Being 5 times slower than macOS was roughly my experience but I thought there was just something wrong with my Windows laptop. I can't find any benchmark or explanation about this.
Likely it is Windows Defender. Someone recently post the issue tracker for this particular issue and some are pointing at Defender as the cause of the I/O performance.
As a daily windows user, I'm not surprised if it is Defender. I have some files and folders in exclusion list because Defender was interfering with those files that the software is trying to use.
Re: SQLite is 35% Faster Than The Filesystem (2017)
#24Is it a known issue that the filesystem on Windows 10 is so slow? Being 5 times slower than macOS was roughly my experience but I thought there was just something wrong with my Windows laptop. I can't find any benchmark or explanation about this.
If you're doing a bunch of I/O, it's often worth it to go into Windows Defender settings and turn off "real time protection" temporarily. It can make a big difference.
Re: SQLite is 35% Faster Than The Filesystem (2017)
#25Is it a known issue that the filesystem on Windows 10 is so slow? Being 5 times slower than macOS was roughly my experience but I thought there was just something wrong with my Windows laptop. I can't find any benchmark or explanation about this.
Windows' slow performance when dealing with millions of tiny files per hour (e.g. Simply moving from files to a DBMS wasn't an option, since the whole point of this edge system was to allow third parties to FTP/SFTP/FTPS/etc in and out, to put/get files. We would have had to re-implement multiple transfer technologies/protocols to have a file-less database system (which didn't make cost-effective sense).
Re: SQLite is 35% Faster Than The Filesystem (2017)
#26I wondered the other day how things would behave in node if dependencies were in node_modules.sqlite3 (with posibility to eject to edit if needed).
Shouldn't it be possible to make an sql structure and mount it like an FS to find out?
I was thinking more about using ESM loader hooks [0] directly.
Re: SQLite is 35% Faster Than The Filesystem (2017)
#27Is it a known issue that the filesystem on Windows 10 is so slow? Being 5 times slower than macOS was roughly my experience but I thought there was just something wrong with my Windows laptop. I can't find any benchmark or explanation about this.
It isn't just Windows 10, as far as I know every version of windows has always wanted to spend like a minute (or multiple minutes!) slowly counting the files before it proceeds to slowly delete them one by one... until it finds one that's open in a background process and nixes the whole operation in a half complete state. 5 times slower sounds conservative. It's awful. Hey, maybe they could put rounded corners on the…
it does that for the UI
For power users, you're much better off using the command line tools
Re: SQLite is 35% Faster Than The Filesystem (2017)
#28Is it a known issue that the filesystem on Windows 10 is so slow? Being 5 times slower than macOS was roughly my experience but I thought there was just something wrong with my Windows laptop. I can't find any benchmark or explanation about this.
My theory is that the OS likes to do a deep inspection of your files in order to feed microsoft's machine learning models.
Re: SQLite is 35% Faster Than The Filesystem (2017)
#29Is it a known issue that the filesystem on Windows 10 is so slow? Being 5 times slower than macOS was roughly my experience but I thought there was just something wrong with my Windows laptop. I can't find any benchmark or explanation about this.
It isn't just Windows 10, as far as I know every version of windows has always wanted to spend like a minute (or multiple minutes!) slowly counting the files before it proceeds to slowly delete them one by one... until it finds one that's open in a background process and nixes the whole operation in a half complete state. 5 times slower sounds conservative. It's awful. Hey, maybe they could put rounded corners on the…
I think that's warts in the file explorer more so than warts in the system itself.
Re: SQLite is 35% Faster Than The Filesystem (2017)
#30I wondered the other day how things would behave in node if dependencies were in node_modules.sqlite3 (with posibility to eject to edit if needed).
Well pnpm centralizes it so that you’re only referencing a single location via symlinks and that is a major speed up, I think moving dependencies out of the file system altogether would be nice. I’ve explored the possibility of this, but I think the way snowpack does streaming imports[1] in version 3.0 may be the best solution overall. I haven’t spent much time testing it but it appears to be a major process improvem…
Snowpack is great for f/e, what I was thinking more about is millions of files after npm i everywhere on b/e - on your local dev machine, on ci, in docker image etc.