Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

SQLite is 35% Faster Than The Filesystem (2017)

sqlite.org

Post reply on HN
Print view View original

71–80 of 137 posts

Re: SQLite is 35% Faster Than The Filesystem (2017)

#71

Post by williamvds »

mikestew wrote: post #33
jjoonathan wrote: post #20

Earlier quoted context omitted.

It isn't just Windows 10, as far as I know every version of windows has always wanted to spend like a minute (or multiple minutes!) slowly counting the files before it proceeds to slowly delete them one by one... until it finds one that's open in a background process and nixes the whole operation in a half complete state. 5 times slower sounds conservative. It's awful. Hey, maybe they could put rounded corners on the…

The only reason I'm posting on HN right now is because I dared to right-click a folder on a network share. MyFolderName (not responding) , whelp, guess I'll go fuck off for 60 seconds while every item in that shortcut menu opens a network connection, navigates to...or whatever the hell it's doing. Yeah, I know, use the CLI. OTOH, I argue: fix your broken UI. I worked at MSFT for eight years as a full-time, Flavorade-…

Most people don't buy Windows - it just comes with their computer, thanks to Microsoft's deals with OEMs.

Kids get accustomed to Windows and Microsoft Office in schools thanks to Microsoft's lobbying, so when they grow up they keep using Windows and pay for Office.

Everyone uses Windows and Microsoft Office, so most companies and organizations have to pay for licenses for each of their employees.

People know Windows, so they develop on Windows, so they get locked into yet more Microsoft products and libraries and have to run Windows on servers. That's yet more licenses for a Windows and for Visual Studio.

Their business strategy would almost be impressive if it weren't terrifyingly harmful.

Re: SQLite is 35% Faster Than The Filesystem (2017)

#72

Post by FractalHQ »

daenz wrote: post #18

Somewhat related, I wrote a fuse-based file system in Rust recently that used SQLite as the backing store for file records, though not the file contents. I imagine I could use it for file content as well, so it's good to know more about its performance. https://amoffat.github.io/supertag/

I’ve wanted to do something like this for ages- definitely going to check this out! The limitations imposed by the hierarchical structure of traditional file systems bothers me rather frequently.

Are there any notable trade-offs or limitations you’ve encountered with this approach?

Re: SQLite is 35% Faster Than The Filesystem (2017)

#73

Post by naikrovek »

masklinn wrote: post #59
naikrovek wrote: post #32

Earlier quoted context omitted.

Yes. NTFS filesystem performance when accessing many small files rapidly is slow. It's an NTFS limitation, not any specific version of Windows. I believe it is caused by some per-file overhead that NTFS incurs, though I'm not 100% sure.

> It's an NTFS limitation, not any specific version of Windows. I would guess their surprise is because the charts specifically show Win10 as being 100% slower than Win7 in the first chart, and still a good 30% slower in the best case (third chart). That's not NTFS v other, that's Win7 v Win10, both on NTFS.

NTFS comes in different versions, you know.

I don't care what version of Windows you're on, or what version of NTFS you're on; NTFS is dog slow on small files.

I am not sure that NTFS has versioned between Windows 7 and Windows 10, so let's just assume that it hasn't.

If the limitations of the filesystem do not change between operating systems, then what must change is how the OS deals with the filesystem limitation, and/or attempts to work around them.

Windows 7 does better than Windows 10. That doesn't mean NTFS isn't slow.

Re: SQLite is 35% Faster Than The Filesystem (2017)

#75

Post by nivenhuh »

shawnz wrote: post #14
laurent123456 wrote: post #9

Is it a known issue that the filesystem on Windows 10 is so slow? Being 5 times slower than macOS was roughly my experience but I thought there was just something wrong with my Windows laptop. I can't find any benchmark or explanation about this.

See here for some notes from the WSL team why certain filesystem operations that would be fast on Linux are slow on Windows: https://github.com/microsoft/WSL/issues/873#issuecomment-391... https://github.com/microsoft/WSL/issues/873#issuecomment-425...

Thanks for the links -- one pro-tip that stood out to me was to use the D: drive (because it's likely to have less filter drivers attached).

"Windows's IO stack is extensible, allowing filter drivers to attach to volumes and intercept IO requests before the file system sees them. This is used for numerous things, including virus scanning, compression, encryption, file virtualization, things like OneDrive's files on demand feature, gathering pre-fetching data to speed up app startup, and much more. Even a clean install of Windows will have a number of filters present, particularly on the system volume (so if you have a D: drive or partition, I recommend using that instead, since it likely has fewer filters attached). Filters are involved in many IO operations, most notably creating/opening files."

Re: SQLite is 35% Faster Than The Filesystem (2017)

#76

Post by edoceo »

logifail wrote: post #70
tyingq wrote: post #29

Earlier quoted context omitted.

I think that's warts in the file explorer more so than warts in the system itself.

> warts in the file explorer more so than warts in the system itself I'm struggling to understand why one shouldn't treat the behaviour of explorer.exe as simply part of the Windows system. Are there third-party explorer.exe alternatives available? If so, during Windows install, where do I click to _not_ install the MSFT explorer.exe and instead select one of the better alternatives?

The last time I really used Windows (Win2k) there were alternative WinShell options.

You'd set a regkey to your program. There used to be a 3.1 style Program Manager - I miss that one

Re: SQLite is 35% Faster Than The Filesystem (2017)

#77

Post by masklinn »

naikrovek wrote: post #73
masklinn wrote: post #59

Earlier quoted context omitted.

> It's an NTFS limitation, not any specific version of Windows. I would guess their surprise is because the charts specifically show Win10 as being 100% slower than Win7 in the first chart, and still a good 30% slower in the best case (third chart). That's not NTFS v other, that's Win7 v Win10, both on NTFS.

NTFS comes in different versions, you know. I don't care what version of Windows you're on, or what version of NTFS you're on; NTFS is dog slow on small files. I am not sure that NTFS has versioned between Windows 7 and Windows 10, so let's just assume that it hasn't. If the limitations of the filesystem do not change between operating systems, then what must change is how the OS deals with the filesystem limitation,…

> NTFS comes in different versions, you know.

A completely pointless remark when versions of NTFS and Windows are correlated. And AFAIK there's been no updated to NTFS itself since the addition of symlinks in NTFS 3.1, released with Windows XP. All "filesystem features" since were added at the level OS, using existing NTFS features.

> If the limitations of the filesystem do not change between operating systems, then what must change is how the OS deals with the filesystem limitation, and/or attempts to work around them.

Yeees? Which would make the issue vary based on the specific version of Windows maybe?

> Windows 7 does better than Windows 10. That doesn't mean NTFS isn't slow.

No, but I never claimed such a thing. I'm just pointing out that your assertion that

> It's […] not any specific version of Windows.

is bullshit.

Re: SQLite is 35% Faster Than The Filesystem (2017)

#78

Post by hughrr »

shawnz wrote: post #14
laurent123456 wrote: post #9

Is it a known issue that the filesystem on Windows 10 is so slow? Being 5 times slower than macOS was roughly my experience but I thought there was just something wrong with my Windows laptop. I can't find any benchmark or explanation about this.

See here for some notes from the WSL team why certain filesystem operations that would be fast on Linux are slow on Windows: https://github.com/microsoft/WSL/issues/873#issuecomment-391... https://github.com/microsoft/WSL/issues/873#issuecomment-425...

I'm going to have to complain about this because in real world use these are far less of a problem than MFT contention on sub-900 byte files generated by typical unix environments. Particularly things that are lockfile heavy and VCSs etc are the most painful. The WSL thing reads like an excuse here. If you actually go and look at what's happening it's small files which are the damage multiplier.

I think the real issue is that you shouldn't mix the two operating system paradigms and it's far better to just run Linux in a VM and benefit from the near native performance at the cost of a tiny bit of inconvenience. It's not a bad option when you consider the remote IDE capabilities that VScode gives you, which is the one product they're doing 100% right.

Re: SQLite is 35% Faster Than The Filesystem (2017)

#79

Post by xtracto »

simcop2387 wrote: post #16
mjevans wrote: post #8

I would love for directories of small files to be stored this way under ZFS; like infrequently updated tar or zip (no compression) files. (Though the filesystem layer of compression might operate on the whole file, or maybe as two streams, one for the file and one for the index.)

I think this is kind of what reiserfs did back in the day for small files. Keep them in the tree and share pages with lots of small files. It worked rather well, it's just that the file system had plenty of other issues with reliability and recovery at the time. It was vitally impotant that you didn't store any plain text disk images that contained a reiserfs partition on a reiserfs mount, fsck could decide to merge…

A killer feature if you allow me to say it.
Post reply on HN