Did Microsoft not try embed SQL server as the backing store for files in Windows "Chicago" to make search a fundamental part of the OS ???.
SQLite is 35% Faster Than The Filesystem (2017)
31–40 of 137 posts
Re: SQLite is 35% Faster Than The Filesystem (2017)
#32Is it a known issue that the filesystem on Windows 10 is so slow? Being 5 times slower than macOS was roughly my experience but I thought there was just something wrong with my Windows laptop. I can't find any benchmark or explanation about this.
Yes. NTFS filesystem performance when accessing many small files rapidly is slow. It's an NTFS limitation, not any specific version of Windows.
I believe it is caused by some per-file overhead that NTFS incurs, though I'm not 100% sure.
Re: SQLite is 35% Faster Than The Filesystem (2017)
#33Is it a known issue that the filesystem on Windows 10 is so slow? Being 5 times slower than macOS was roughly my experience but I thought there was just something wrong with my Windows laptop. I can't find any benchmark or explanation about this.
It isn't just Windows 10, as far as I know every version of windows has always wanted to spend like a minute (or multiple minutes!) slowly counting the files before it proceeds to slowly delete them one by one... until it finds one that's open in a background process and nixes the whole operation in a half complete state. 5 times slower sounds conservative. It's awful. Hey, maybe they could put rounded corners on the…
The only reason I'm posting on HN right now is because I dared to right-click a folder on a network share. MyFolderName (not responding), whelp, guess I'll go fuck off for 60 seconds while every item in that shortcut menu opens a network connection, navigates to...or whatever the hell it's doing. Yeah, I know, use the CLI. OTOH, I argue: fix your broken UI.
I worked at MSFT for eight years as a full-time, Flavorade-guzzling employee, and fifteen years later rarely a day goes by that I don't question how Microsoft manages to get anyone to buy that steaming pile they call an operating system.
Re: SQLite is 35% Faster Than The Filesystem (2017)
#34No idea what's with the sqlite articles making the front page every other day. This one is pretty old as well.
The measurements in this article were made during the week of 2017-06-05 using a version of SQLite in between 3.19.2 and 3.20.0.Re: SQLite is 35% Faster Than The Filesystem (2017)
#35Is it a known issue that the filesystem on Windows 10 is so slow? Being 5 times slower than macOS was roughly my experience but I thought there was just something wrong with my Windows laptop. I can't find any benchmark or explanation about this.
Likely it is Windows Defender. Someone recently post the issue tracker for this particular issue and some are pointing at Defender as the cause of the I/O performance. As a daily windows user, I'm not surprised if it is Defender. I have some files and folders in exclusion list because Defender was interfering with those files that the software is trying to use.
That’s probably part of it, but NTFS has just been slow forever and ever. Anything involving lots of small files is just pain. Perhaps it seems “normal” for anyone used to Explorer progress dialogs, but it’s not.
And Microsoft just doesn’t seem to care :-(.
Re: SQLite is 35% Faster Than The Filesystem (2017)
#36Is it a known issue that the filesystem on Windows 10 is so slow? Being 5 times slower than macOS was roughly my experience but I thought there was just something wrong with my Windows laptop. I can't find any benchmark or explanation about this.
Likely it is Windows Defender. Someone recently post the issue tracker for this particular issue and some are pointing at Defender as the cause of the I/O performance. As a daily windows user, I'm not surprised if it is Defender. I have some files and folders in exclusion list because Defender was interfering with those files that the software is trying to use.
It's not Windows Defender. Not entirely, anyway. NTFS is slow when reading or writing lots of small files. It just is, and it always has been, and it's because of how NTFS was designed, if I am remembering correctly.
Re: SQLite is 35% Faster Than The Filesystem (2017)
#37Earlier quoted context omitted.
It isn't just Windows 10, as far as I know every version of windows has always wanted to spend like a minute (or multiple minutes!) slowly counting the files before it proceeds to slowly delete them one by one... until it finds one that's open in a background process and nixes the whole operation in a half complete state. 5 times slower sounds conservative. It's awful. Hey, maybe they could put rounded corners on the…
The only reason I'm posting on HN right now is because I dared to right-click a folder on a network share. MyFolderName (not responding) , whelp, guess I'll go fuck off for 60 seconds while every item in that shortcut menu opens a network connection, navigates to...or whatever the hell it's doing. Yeah, I know, use the CLI. OTOH, I argue: fix your broken UI. I worked at MSFT for eight years as a full-time, Flavorade-…
> rarely a day goes by that I don't question how Microsoft manages to get anyone to buy that steaming pile they call an operating system.
"But muh games!"
Re: SQLite is 35% Faster Than The Filesystem (2017)
#38Is it a known issue that the filesystem on Windows 10 is so slow? Being 5 times slower than macOS was roughly my experience but I thought there was just something wrong with my Windows laptop. I can't find any benchmark or explanation about this.
It isn't just Windows 10, as far as I know every version of windows has always wanted to spend like a minute (or multiple minutes!) slowly counting the files before it proceeds to slowly delete them one by one... until it finds one that's open in a background process and nixes the whole operation in a half complete state. 5 times slower sounds conservative. It's awful. Hey, maybe they could put rounded corners on the…
This has very little to do with Windows, itself. The issue is explorer that tries to estimate sizes and put the files in the recycled bin. Also likely it checks all the permissions while counting.
Try the command line: "rmdir /s" and it's quick.
Re: SQLite is 35% Faster Than The Filesystem (2017)
#39Locality strikes again!
> The performance difference arises (we believe) because when working from an SQLite database, the open() and close() system calls are invoked only once, whereas open() and close() are invoked once for each blob when using blobs stored in individual files. It appears that the overhead of calling open() and close() is greater than the overhead of using the database. The size reduction arises from the fact that individ…
One trick I used to use is to put a small read/write cache in front of file operations. I would usually pick something like the size of a sector or cluster. Windows is pretty bad at small writes, and mearly 'ok' at reads because you usually can just get away with the windows file store cache in that case.
I came across the write cache bit accidently when trying to minimize wear on an embedded flash device. It was borderline I thought I had done something wrong.
Minimizing open/close and keeping your reads/writes close to sector/cluster size on many filesystems can produce some very nice results. As you can minimize the context switches from user space to kernel. In the case of adding in a 'db' layer packing probably helps as well as slack on small files is huge percentage wise of the total file. So you would be more likely to hit the file cache as well as any in built ones for your stack.
Re: SQLite is 35% Faster Than The Filesystem (2017)
#40Did Microsoft not try embed SQL server as the backing store for files in Windows "Chicago" to make search a fundamental part of the OS ???.
It was a Memphis (NT 4.0) goal, canceled. Later, a Longhorn (Vista, "NT 6.0") goal, also canceled.
The Windows file system is still as dumb as it was in NT 3.1 -- hardly any changes since then.