Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

SQLite is 35% Faster Than The Filesystem (2017)

sqlite.org

Post reply on HN
Print view View original

91–100 of 137 posts

Re: SQLite is 35% Faster Than The Filesystem (2017)

#91

Post by imhoguy »

chungy wrote: post #44
daenz wrote: post #18

Somewhat related, I wrote a fuse-based file system in Rust recently that used SQLite as the backing store for file records, though not the file contents. I imagine I could use it for file content as well, so it's good to know more about its performance. https://amoffat.github.io/supertag/

SQLite, iirc, has a limit of 1GB per row, and that might too severely limit the utility of your file system if you don't end up splitting files into multiple fragments (rows in the database).

You are right, by default 1 billion bytes per row or blob. Can be raised up to 2GB with compilation parameter[0]. Storing video streams may be not a good idea, but e.g. photos, thumbnails or some JSON documents may be absolutely fine.

[0] https://sqlite.org/limits.html

Re: SQLite is 35% Faster Than The Filesystem (2017)

#93

Post by atatatat »

gavinray wrote: post #81
mikestew wrote: post #33

Earlier quoted context omitted.

The only reason I'm posting on HN right now is because I dared to right-click a folder on a network share. MyFolderName (not responding) , whelp, guess I'll go fuck off for 60 seconds while every item in that shortcut menu opens a network connection, navigates to...or whatever the hell it's doing. Yeah, I know, use the CLI. OTOH, I argue: fix your broken UI. I worked at MSFT for eight years as a full-time, Flavorade-…

Windows as an OS is... not the best. I miss Ubuntu/Pop!_OS. Switched due to MIDI controller driver compatibility issues that seemed immune to any WINE hackery =/ I actually really like the rest of what Microsoft does though. VS Code, TypeScript, creating the Language Server Protocol, the .NET Foundation/.NET in general and C#, the list goes on and on. They're an impactful organization which (depending on what kind of…

> that's a(n unlikely) risk

Oh, honey.

Re: SQLite is 35% Faster Than The Filesystem (2017)

#94

Post by tyingq »

logifail wrote: post #70
tyingq wrote: post #29

Earlier quoted context omitted.

I think that's warts in the file explorer more so than warts in the system itself.

> warts in the file explorer more so than warts in the system itself I'm struggling to understand why one shouldn't treat the behaviour of explorer.exe as simply part of the Windows system. Are there third-party explorer.exe alternatives available? If so, during Windows install, where do I click to _not_ install the MSFT explorer.exe and instead select one of the better alternatives?

I'm not excusing it, I'm just saying where the performance issue is. Filesystem ops are reasonably fast if you use the command line, do it programmatically, etc. With the caveat that Windows Defender can slow things down.

Re: SQLite is 35% Faster Than The Filesystem (2017)

#95

Post by jjoonathan »

metalliqaz wrote: post #27
jjoonathan wrote: post #20

Earlier quoted context omitted.

It isn't just Windows 10, as far as I know every version of windows has always wanted to spend like a minute (or multiple minutes!) slowly counting the files before it proceeds to slowly delete them one by one... until it finds one that's open in a background process and nixes the whole operation in a half complete state. 5 times slower sounds conservative. It's awful. Hey, maybe they could put rounded corners on the…

it does that for the UI For power users, you're much better off using the command line tools

I have a different power user workaround: I don't use Windows if I can at all avoid it.

EDIT: Does the CLI override the "open in background" check? Or do you still need Mark Russinovich sysinternals to reliably perform elementary file system administration tasks in Windows?

Re: SQLite is 35% Faster Than The Filesystem (2017)

#97

Post by qwertox »

laurent123456 wrote: post #9

Is it a known issue that the filesystem on Windows 10 is so slow? Being 5 times slower than macOS was roughly my experience but I thought there was just something wrong with my Windows laptop. I can't find any benchmark or explanation about this.

Alone the fact that it is twice as slow as Windows 7 is puzzling.

One would assume that it gets better with every upgrade, but a factor of 2x worse seems to indicate that they added some kind of logging or something.

Like the logging/access system which Android has started to integrate with Android 9 where every access needs to go through the StorageManager.

Re: SQLite is 35% Faster Than The Filesystem (2017)

#98

Post by qwertox »

jjoonathan wrote: post #20
laurent123456 wrote: post #9

Is it a known issue that the filesystem on Windows 10 is so slow? Being 5 times slower than macOS was roughly my experience but I thought there was just something wrong with my Windows laptop. I can't find any benchmark or explanation about this.

It isn't just Windows 10, as far as I know every version of windows has always wanted to spend like a minute (or multiple minutes!) slowly counting the files before it proceeds to slowly delete them one by one... until it finds one that's open in a background process and nixes the whole operation in a half complete state. 5 times slower sounds conservative. It's awful. Hey, maybe they could put rounded corners on the…

Oh man... Deleting a folder with a million files, or even only 100.000 is painful in Windows. You need to use the command line in order to sanely delete such folders.

Re: SQLite is 35% Faster Than The Filesystem (2017)

#99

Post by qwertox »

GordonS wrote: post #87
xxs wrote: post #38

Earlier quoted context omitted.

This has very little to do with Windows, itself. The issue is explorer that tries to estimate sizes and put the files in the recycled bin. Also likely it checks all the permissions while counting. Try the command line: "rmdir /s" and it's quick.

This is true, but I don't understand why Explorer has been plagued with performance issues (and some of those intermittent; I presume based on network status) since forever. I'm actually a big fan of Windows on the desktop, and as long as I avoid Explorer, I have no issues at all with file system performance while I'm developing, working from the command line, VS Code and Rider, and working with containers and WSL -…

Sometimes it's some .dll extensions which get loaded into explorer.exe which cause these problems.

I once had a PDF file on the desktop and for some reason this file was causing explorer.exe to freeze under certain circumstances. It's all a big mistery.

Re: SQLite is 35% Faster Than The Filesystem (2017)

#100

Post by GauntletWizard »

daenz wrote: post #18

Somewhat related, I wrote a fuse-based file system in Rust recently that used SQLite as the backing store for file records, though not the file contents. I imagine I could use it for file content as well, so it's good to know more about its performance. https://amoffat.github.io/supertag/

I wanted to learn about Fuse and SQLite in Golang, so I built a filesystem that was entirely stored in SQLite. It's incomplete (read-only for directories at the moment, among other nits), but was surprisingly easy and performed with decent performance on some tests I did; Nearly half throughput of a raw filesystem.
Post reply on HN