Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

SQLite is 35% Faster Than The Filesystem (2017)

sqlite.org

Post reply on HN
Print view View original

81–90 of 137 posts

Re: SQLite is 35% Faster Than The Filesystem (2017)

#81

Post by gavinray »

mikestew wrote: post #33
jjoonathan wrote: post #20

Earlier quoted context omitted.

It isn't just Windows 10, as far as I know every version of windows has always wanted to spend like a minute (or multiple minutes!) slowly counting the files before it proceeds to slowly delete them one by one... until it finds one that's open in a background process and nixes the whole operation in a half complete state. 5 times slower sounds conservative. It's awful. Hey, maybe they could put rounded corners on the…

The only reason I'm posting on HN right now is because I dared to right-click a folder on a network share. MyFolderName (not responding) , whelp, guess I'll go fuck off for 60 seconds while every item in that shortcut menu opens a network connection, navigates to...or whatever the hell it's doing. Yeah, I know, use the CLI. OTOH, I argue: fix your broken UI. I worked at MSFT for eight years as a full-time, Flavorade-…

Windows as an OS is... not the best. I miss Ubuntu/Pop!_OS. Switched due to MIDI controller driver compatibility issues that seemed immune to any WINE hackery =/

I actually really like the rest of what Microsoft does though.

VS Code, TypeScript, creating the Language Server Protocol, the .NET Foundation/.NET in general and C#, the list goes on and on. They're an impactful organization which (depending on what kind of software you build/what you write) probably has a significant impact on your day-to-day experience as a developer.

The benefits of vertical integration can have when you (for lack of a better word) "own" these things can be beneficial for the end-user/developer, ignoring whatever kind of anti-competitive arguments there are.

It's the same argument I hear from Apple users -- Apple stewards their digital experience and gives them continuity end-to-end.

Microsoft -- by nature of owning my OS, IDE, the specification and dev team driving my primary language, a cloud provider, and having their fingers in all of these parts of my digital life -- has the ability to be really awesome for me.

They could also do really sinister things if they wanted and I'd be real fucked I suppose, but that's a(n unlikely) risk I accept.

Re: SQLite is 35% Faster Than The Filesystem (2017)

#82

Post by puzzlingcaptcha »

chungy wrote: post #37
mikestew wrote: post #33

Earlier quoted context omitted.

The only reason I'm posting on HN right now is because I dared to right-click a folder on a network share. MyFolderName (not responding) , whelp, guess I'll go fuck off for 60 seconds while every item in that shortcut menu opens a network connection, navigates to...or whatever the hell it's doing. Yeah, I know, use the CLI. OTOH, I argue: fix your broken UI. I worked at MSFT for eight years as a full-time, Flavorade-…

> rarely a day goes by that I don't question how Microsoft manages to get anyone to buy that steaming pile they call an operating system. "But muh games!"

Yeah, 'cause the incredible fragmentation of Linux GUI toolkits and windowing systems is so fun to deal with. Year of Linux on the Desktop when.

Re: SQLite is 35% Faster Than The Filesystem (2017)

#83

Post by naikrovek »

tantalor wrote: post #54
bsaul wrote: post #48

Earlier quoted context omitted.

I guess you must be a low-level developer then, because to application developer it would seem that sqlite write speed is actually bound by the file system performance (which it depends on).

The title is a joke, right? It's not actually "faster than the filesystem" unless it's not using the filesystem; it's faster than something else that also uses the filesystem because they use different system calls (context switching).

you can be faster than the filesystem, while using the filesystem, if you emulate something the filesystem does poorly in a more performant way.

That's what's happening here. storing files in SQLite removes any per-file overhead in the filesystem. The filesystem now only has one file to deal with, instead of however many are stored inside the SQLite database.

This is a very real phenomenon, and definitely not a joke.

Re: SQLite is 35% Faster Than The Filesystem (2017)

#84

Post by xxs »

mackwell wrote: post #62
xxs wrote: post #38

Earlier quoted context omitted.

This has very little to do with Windows, itself. The issue is explorer that tries to estimate sizes and put the files in the recycled bin. Also likely it checks all the permissions while counting. Try the command line: "rmdir /s" and it's quick.

I think for the purpose of this discussion, explorer IS Windows. It is the UI that we all wish didn’t have these issues. I like using the command line / powershell when I’m on Windows for large file operations, but sometimes I still go to drag files around out of habit and quickly remember I have made a huge mistake as everything grinds to a halt / takes forever. On Linux and MacOS GUI file operations most of the tim…

My habit is shift+del for windows; but most of the time I use cygwin anyways.

>I think for the purpose of this discussion, explorer IS Windows.

The article is about the file system, not the UI.

Re: SQLite is 35% Faster Than The Filesystem (2017)

#86

Post by anigbrowl »

jjoonathan wrote: post #20
laurent123456 wrote: post #9

Is it a known issue that the filesystem on Windows 10 is so slow? Being 5 times slower than macOS was roughly my experience but I thought there was just something wrong with my Windows laptop. I can't find any benchmark or explanation about this.

It isn't just Windows 10, as far as I know every version of windows has always wanted to spend like a minute (or multiple minutes!) slowly counting the files before it proceeds to slowly delete them one by one... until it finds one that's open in a background process and nixes the whole operation in a half complete state. 5 times slower sounds conservative. It's awful. Hey, maybe they could put rounded corners on the…

I think it's file manager in particular. When I need to do something muscular I open a DOS box and just do it at the command line. It feels plenty zippy though I haven't bothered to do any benchmarks; I have to confess I've never learned Powershell because it's so easy to get most things done with command.com.

Although Windows remains my favorite environment I share the annoyance at how MS really polishes some things while failing on basic functionality, when they definitely have the resources and skills to fix that. Every time my wifi or WAN hiccups I wonder why Windows still lacks decent network history/QOS tools out of the box.

Re: SQLite is 35% Faster Than The Filesystem (2017)

#87

Post by GordonS »

xxs wrote: post #38
jjoonathan wrote: post #20

Earlier quoted context omitted.

It isn't just Windows 10, as far as I know every version of windows has always wanted to spend like a minute (or multiple minutes!) slowly counting the files before it proceeds to slowly delete them one by one... until it finds one that's open in a background process and nixes the whole operation in a half complete state. 5 times slower sounds conservative. It's awful. Hey, maybe they could put rounded corners on the…

This has very little to do with Windows, itself. The issue is explorer that tries to estimate sizes and put the files in the recycled bin. Also likely it checks all the permissions while counting. Try the command line: "rmdir /s" and it's quick.

This is true, but I don't understand why Explorer has been plagued with performance issues (and some of those intermittent; I presume based on network status) since forever.

I'm actually a big fan of Windows on the desktop, and as long as I avoid Explorer, I have no issues at all with file system performance while I'm developing, working from the command line, VS Code and Rider, and working with containers and WSL - all works great. But Explorer is such a central component of Windows that it affects pretty all users. I genuinely don't understand how it can be so rubbish ¯\_(ツ)_/¯

Re: SQLite is 35% Faster Than The Filesystem (2017)

#88

Post by opheliate »

daenz wrote: post #18

Somewhat related, I wrote a fuse-based file system in Rust recently that used SQLite as the backing store for file records, though not the file contents. I imagine I could use it for file content as well, so it's good to know more about its performance. https://amoffat.github.io/supertag/

Very cool! I've been using TMSU for this purpose for a while, but I'll have to check out supertag :)

Re: SQLite is 35% Faster Than The Filesystem (2017)

#89

Post by vagrantJin »

mikestew wrote: post #33
jjoonathan wrote: post #20

Earlier quoted context omitted.

It isn't just Windows 10, as far as I know every version of windows has always wanted to spend like a minute (or multiple minutes!) slowly counting the files before it proceeds to slowly delete them one by one... until it finds one that's open in a background process and nixes the whole operation in a half complete state. 5 times slower sounds conservative. It's awful. Hey, maybe they could put rounded corners on the…

The only reason I'm posting on HN right now is because I dared to right-click a folder on a network share. MyFolderName (not responding) , whelp, guess I'll go fuck off for 60 seconds while every item in that shortcut menu opens a network connection, navigates to...or whatever the hell it's doing. Yeah, I know, use the CLI. OTOH, I argue: fix your broken UI. I worked at MSFT for eight years as a full-time, Flavorade-…

> rarely a day goes by that I don't question how Microsoft manages to get anyone to buy that steaming pile they call an operating system.

Haha. Maybe because it works? I do miss XP though. It felt like the only OS that can work as well on a PC and a Tank.

Re: SQLite is 35% Faster Than The Filesystem (2017)

#90

Post by yashap »

kissgyorgy wrote: post #55
xxs wrote: post #34

No idea what's with the sqlite articles making the front page every other day. This one is pretty old as well. The measurements in this article were made during the week of 2017-06-05 using a version of SQLite in between 3.19.2 and 3.20.0.

I think it's just simply a lot of folks on HN (including me) just like SQLite very much and instantly voting up an article about SQLite (even when we already saw it :D).

Yeah, IMO there’s certain tech that HN, in aggregate, likes a lot, and readily upvotes positive articles about - SQLite is in that category, along with Postgres, CockroachDB, Go, Rust, etc.

There’s also certain tech HN, in aggregate, strongly dislikes, and readily upvotes negative articles - Mongo, anything “modern JS ecosystem”, systemd, etc.

Post reply on HN