Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

SQLite is 35% Faster Than The Filesystem (2017)

sqlite.org

Post reply on HN
Print view View original

111–120 of 137 posts

Re: SQLite is 35% Faster Than The Filesystem (2017)

#111

Post by naikrovek »

masklinn wrote: post #77
naikrovek wrote: post #73

Earlier quoted context omitted.

NTFS comes in different versions, you know. I don't care what version of Windows you're on, or what version of NTFS you're on; NTFS is dog slow on small files. I am not sure that NTFS has versioned between Windows 7 and Windows 10, so let's just assume that it hasn't. If the limitations of the filesystem do not change between operating systems, then what must change is how the OS deals with the filesystem limitation,…

> NTFS comes in different versions, you know. A completely pointless remark when versions of NTFS and Windows are correlated. And AFAIK there's been no updated to NTFS itself since the addition of symlinks in NTFS 3.1, released with Windows XP. All "filesystem features" since were added at the level OS, using existing NTFS features. > If the limitations of the filesystem do not change between operating systems, then…

>> It's […] not any specific version of Windows.

> is bullshit.

No, it's not. NTFS is slow, even when it's running under Linux. NTFS has a very pronounced weakness, here.

Performance changing between Windows 7 and Windows 10 means that ON TOP OF NTFS there are other performance problems in Windows 10. It does not mean that it isn't an NTFS problem in the case of Windows 10 -- it means that it isn't ONLY an NTFS problem in the case of Windows 10. It's an NTFS problem in addition to whatever other things Windows 10 is screwing up, when compared to Windows 7.

The difference between Windows 7 and Windows 10 alone? Yes, that's entirely because of Windows 10 and not because of NTFS. There. I never disagreed with that.

Re: SQLite is 35% Faster Than The Filesystem (2017)

#112

Post by blooalien »

poisonborz wrote: post #107
jjoonathan wrote: post #105

Earlier quoted context omitted.

What about refusing to delete files that some process has open in a dangling descriptor? Is that just a GUI thing? I'm pretty sure it isn't. Also, is there a better way to hunt down the background process than sysinternals? It seems pretty ridiculous that a low level debugging tool is required to do something as modest as reliably move and delete files, but I never heard of a better workaround.

LockHunter is a nice right-click utility for this

You just outed yourself as a "power user". ;)

Re: SQLite is 35% Faster Than The Filesystem (2017)

#113

Post by blooalien »

qwertox wrote: post #99
GordonS wrote: post #87

Earlier quoted context omitted.

This is true, but I don't understand why Explorer has been plagued with performance issues (and some of those intermittent; I presume based on network status) since forever. I'm actually a big fan of Windows on the desktop, and as long as I avoid Explorer, I have no issues at all with file system performance while I'm developing, working from the command line, VS Code and Rider, and working with containers and WSL -…

Sometimes it's some .dll extensions which get loaded into explorer.exe which cause these problems. I once had a PDF file on the desktop and for some reason this file was causing explorer.exe to freeze under certain circumstances. It's all a big mistery.

> It's all a big mistery.

Such is the nature of closed source black box software.

Re: SQLite is 35% Faster Than The Filesystem (2017)

#114

Post by motogpjimbo »

puzzlingcaptcha wrote: post #82
chungy wrote: post #37

Earlier quoted context omitted.

> rarely a day goes by that I don't question how Microsoft manages to get anyone to buy that steaming pile they call an operating system. "But muh games!"

Yeah, 'cause the incredible fragmentation of Linux GUI toolkits and windowing systems is so fun to deal with. Year of Linux on the Desktop when.

There are two windowing systems on Linux - the old one and the new one.

GUI toolkits are irrelevant to games.

Re: SQLite is 35% Faster Than The Filesystem (2017)

#115

Post by mickotron »

anigbrowl wrote: post #86
jjoonathan wrote: post #20

Earlier quoted context omitted.

It isn't just Windows 10, as far as I know every version of windows has always wanted to spend like a minute (or multiple minutes!) slowly counting the files before it proceeds to slowly delete them one by one... until it finds one that's open in a background process and nixes the whole operation in a half complete state. 5 times slower sounds conservative. It's awful. Hey, maybe they could put rounded corners on the…

I think it's file manager in particular. When I need to do something muscular I open a DOS box and just do it at the command line. It feels plenty zippy though I haven't bothered to do any benchmarks; I have to confess I've never learned Powershell because it's so easy to get most things done with command.com. Although Windows remains my favorite environment I share the annoyance at how MS really polishes some things…

Yep, "dir /s /b | findstr x" is way quicker than searching for files in explorer.

Re: SQLite is 35% Faster Than The Filesystem (2017)

#116

Post by notriddle »

chungy wrote: post #37
mikestew wrote: post #33

Earlier quoted context omitted.

The only reason I'm posting on HN right now is because I dared to right-click a folder on a network share. MyFolderName (not responding) , whelp, guess I'll go fuck off for 60 seconds while every item in that shortcut menu opens a network connection, navigates to...or whatever the hell it's doing. Yeah, I know, use the CLI. OTOH, I argue: fix your broken UI. I worked at MSFT for eight years as a full-time, Flavorade-…

> rarely a day goes by that I don't question how Microsoft manages to get anyone to buy that steaming pile they call an operating system. "But muh games!"

Yeah, this but unironically.

The Windows filesystem has dismal performance. The Windows graphics stack is arguably better than Linux’s. And it’s not just the drivers, so you can’t just blame it on GPU vendors, though poor support from a certain popular GPU vendor definitely doesn’t help.

Some of it is just the little things, like doing mouse movement at very high priority in kernel space [1] [2]. Other parts are bigger and more impressive, like allowing the graphics driver to crash/upgrade/restart without you also losing all of your open windows.

[1]: https://reactos.org/wiki/Techwiki:Win32k/desktops#win32k.21P...

[2]: https://devblogs.microsoft.com/oldnewthing/20061117-03/?p=28...

Re: SQLite is 35% Faster Than The Filesystem (2017)

#117

Post by tyingq »

jjoonathan wrote: post #103
tyingq wrote: post #29

Earlier quoted context omitted.

I think that's warts in the file explorer more so than warts in the system itself.

Is "you can't delete a file if anything has it open" an explorer thing? I thought that was an OS thing.

That's an OS thing, but "wait forever while it grinds, finds one of those, and bails half finished" isn't.

Re: SQLite is 35% Faster Than The Filesystem (2017)

#118

Post by SXX »

xtracto wrote: post #79
simcop2387 wrote: post #16

Earlier quoted context omitted.

I think this is kind of what reiserfs did back in the day for small files. Keep them in the tree and share pages with lots of small files. It worked rather well, it's just that the file system had plenty of other issues with reliability and recovery at the time. It was vitally impotant that you didn't store any plain text disk images that contained a reiserfs partition on a reiserfs mount, fsck could decide to merge…

A killer feature if you allow me to say it.

This joke will never grow old...

Re: SQLite is 35% Faster Than The Filesystem (2017)

#119

Post by [deleted] »

hughrr wrote: post #78
shawnz wrote: post #14

Earlier quoted context omitted.

See here for some notes from the WSL team why certain filesystem operations that would be fast on Linux are slow on Windows: https://github.com/microsoft/WSL/issues/873#issuecomment-391... https://github.com/microsoft/WSL/issues/873#issuecomment-425...

I'm going to have to complain about this because in real world use these are far less of a problem than MFT contention on sub-900 byte files generated by typical unix environments. Particularly things that are lockfile heavy and VCSs etc are the most painful. The WSL thing reads like an excuse here. If you actually go and look at what's happening it's small files which are the damage multiplier. I think the real issu…

[deleted]

Re: SQLite is 35% Faster Than The Filesystem (2017)

#120

Post by zmj »

jamal-kumar wrote: post #60

This gets posted a lot here and I really wonder if anyone's bothered to try and replicate these results or test them with different sizes of data on different filesystems - You can see that the greatest disparity is with NTFS/windows while linux is pretty close in performance, but they don't bother to mention if they've got ubuntu formatted for ext4 or whatever. I can't really seem to find anything looking around, an…

I tried this recently (~2 years ago). SQLite is faster, but returning unused space to the OS is a pain. If you don't need that to be prompt it's a good solution.
Post reply on HN