then making a small cdn using nginx and sqlite can be a thing?
CDN is effectively read only and served from memory for the most of the part.
SQLite is 35% Faster Than The Filesystem (2017)
41–50 of 137 posts
Re: SQLite is 35% Faster Than The Filesystem (2017)
#42Earlier quoted context omitted.
Likely it is Windows Defender. Someone recently post the issue tracker for this particular issue and some are pointing at Defender as the cause of the I/O performance. As a daily windows user, I'm not surprised if it is Defender. I have some files and folders in exclusion list because Defender was interfering with those files that the software is trying to use.
That’s probably part of it, but NTFS has just been slow forever and ever. Anything involving lots of small files is just pain. Perhaps it seems “normal” for anyone used to Explorer progress dialogs, but it’s not. And Microsoft just doesn’t seem to care :-(.
They care, as much as a group of people without a mind of its own can care, but it's a use case that was not common among windows software when NTFS was designed. Other design considerations were more important than high performance while writing or reading 1 million small files per minute.
Re: SQLite is 35% Faster Than The Filesystem (2017)
#43Is it a known issue that the filesystem on Windows 10 is so slow? Being 5 times slower than macOS was roughly my experience but I thought there was just something wrong with my Windows laptop. I can't find any benchmark or explanation about this.
My theory is that the OS likes to do a deep inspection of your files in order to feed microsoft's machine learning models.
your theory is based on ignorance and suspicion instead of anything real.
Re: SQLite is 35% Faster Than The Filesystem (2017)
#44Somewhat related, I wrote a fuse-based file system in Rust recently that used SQLite as the backing store for file records, though not the file contents. I imagine I could use it for file content as well, so it's good to know more about its performance. https://amoffat.github.io/supertag/
SQLite, iirc, has a limit of 1GB per row, and that might too severely limit the utility of your file system if you don't end up splitting files into multiple fragments (rows in the database).
Re: SQLite is 35% Faster Than The Filesystem (2017)
#45Did Microsoft not try embed SQL server as the backing store for files in Windows "Chicago" to make search a fundamental part of the OS ???.
Close, it was Cairo/NT5 for initial incarnation. Then it kept getting pushed back, finally shipped as a beta, then just didn't ship at all:
Re: SQLite is 35% Faster Than The Filesystem (2017)
#46Did Microsoft not try embed SQL server as the backing store for files in Windows "Chicago" to make search a fundamental part of the OS ???.
I believe you're thinking of WinFS[1] which was destined for Windows "Longhorn"[2]
[1]: https://en.wikipedia.org/wiki/WinFS
[2]: https://en.wikipedia.org/wiki/Development_of_Windows_Vista
Re: SQLite is 35% Faster Than The Filesystem (2017)
#47No idea what's with the sqlite articles making the front page every other day. This one is pretty old as well. The measurements in this article were made during the week of 2017-06-05 using a version of SQLite in between 3.19.2 and 3.20.0.
Either it's a concerted effort by some number of people to promote it, or it's that most people didn't see the original news and now find it interesting.
Re: SQLite is 35% Faster Than The Filesystem (2017)
#48I'd be shocked if this weren't the case.
I guess you must be a low-level developer then, because to application developer it would seem that sqlite write speed is actually bound by the file system performance (which it depends on).
Re: SQLite is 35% Faster Than The Filesystem (2017)
#49[deleted]
Re: SQLite is 35% Faster Than The Filesystem (2017)
#50Is it a known issue that the filesystem on Windows 10 is so slow? Being 5 times slower than macOS was roughly my experience but I thought there was just something wrong with my Windows laptop. I can't find any benchmark or explanation about this.
I think this was known for a long time. E.g. Git takes forever, compared to the equivalent operations in Linux