Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

SQLite is 35% Faster Than The Filesystem (2017)

sqlite.org

Post reply on HN
Print view View original

61–70 of 137 posts

Re: SQLite is 35% Faster Than The Filesystem (2017)

#61

Post by naikrovek »

nightfly wrote: post #17
Already__Taken wrote: post #7

Earlier quoted context omitted.

Shouldn't it be possible to make an sql structure and mount it like an FS to find out?

That would be the worst of both worlds: The overhead of storing your data in SQL, plus the overhead of filesystem search and access. The better way to do this would be to have node access the modules in SQLite directly.

filesystem search and access is different when you're talking a single larger file versus multiple smaller files. All filesystems have a per-file overhead that would effectively be eliminated if you could pack all of your files into a database structure. Indices would also speed up access to individual rows of the table significantly.

There is overhead in small file storage anyway, if the files are not the exact size, or a multiple, of the sector size. storing 1kb files on a disk where the sector size is 16kb is far more of an overhead expense than storing those files in an SQL database.

Re: SQLite is 35% Faster Than The Filesystem (2017)

#62

Post by mackwell »

xxs wrote: post #38
jjoonathan wrote: post #20

Earlier quoted context omitted.

It isn't just Windows 10, as far as I know every version of windows has always wanted to spend like a minute (or multiple minutes!) slowly counting the files before it proceeds to slowly delete them one by one... until it finds one that's open in a background process and nixes the whole operation in a half complete state. 5 times slower sounds conservative. It's awful. Hey, maybe they could put rounded corners on the…

This has very little to do with Windows, itself. The issue is explorer that tries to estimate sizes and put the files in the recycled bin. Also likely it checks all the permissions while counting. Try the command line: "rmdir /s" and it's quick.

I think for the purpose of this discussion, explorer IS Windows. It is the UI that we all wish didn’t have these issues. I like using the command line / powershell when I’m on Windows for large file operations, but sometimes I still go to drag files around out of habit and quickly remember I have made a huge mistake as everything grinds to a halt / takes forever. On Linux and MacOS GUI file operations most of the time seem like they are just a visual way to call the same operations you’d be using on the command line.

Re: SQLite is 35% Faster Than The Filesystem (2017)

#64

Post by TrackerFF »

xxs wrote: post #34

No idea what's with the sqlite articles making the front page every other day. This one is pretty old as well. The measurements in this article were made during the week of 2017-06-05 using a version of SQLite in between 3.19.2 and 3.20.0.

I think that whenever some topic sticks on the front page, and has a couple of hundred upvotes, you'll see a trailing/decaying trend of similar posts the days after.

Some people see the main post, do some more research on the topic, find something cool / interesting, and post them here. Then this induces similar reactions, until the interest low enough to make them disappear.

But then again, some recurring topics seem to stick around forever. It seems like there's a new Rust article here almost daily.

Re: SQLite is 35% Faster Than The Filesystem (2017)

#65

Post by bombela »

chungy wrote: post #44
daenz wrote: post #18

Somewhat related, I wrote a fuse-based file system in Rust recently that used SQLite as the backing store for file records, though not the file contents. I imagine I could use it for file content as well, so it's good to know more about its performance. https://amoffat.github.io/supertag/

SQLite, iirc, has a limit of 1GB per row, and that might too severely limit the utility of your file system if you don't end up splitting files into multiple fragments (rows in the database).

You read too fast:

> SQLite as the backing store for file records, though not the file contents

Re: SQLite is 35% Faster Than The Filesystem (2017)

#66

Post by jacobwg »

bombela wrote: post #65
chungy wrote: post #44

Earlier quoted context omitted.

SQLite, iirc, has a limit of 1GB per row, and that might too severely limit the utility of your file system if you don't end up splitting files into multiple fragments (rows in the database).

You read too fast: > SQLite as the backing store for file records, though not the file contents

I believe they are referring to:

> I imagine I could use it for file content as well

Re: SQLite is 35% Faster Than The Filesystem (2017)

#67

Post by amarshall »

bombela wrote: post #65
chungy wrote: post #44

Earlier quoted context omitted.

SQLite, iirc, has a limit of 1GB per row, and that might too severely limit the utility of your file system if you don't end up splitting files into multiple fragments (rows in the database).

You read too fast: > SQLite as the backing store for file records, though not the file contents

So did you:

> I imagine I could use [SQLite] for file content as well

Re: SQLite is 35% Faster Than The Filesystem (2017)

#68

Post by [deleted] »

bombela wrote: post #65
chungy wrote: post #44

Earlier quoted context omitted.

SQLite, iirc, has a limit of 1GB per row, and that might too severely limit the utility of your file system if you don't end up splitting files into multiple fragments (rows in the database).

You read too fast: > SQLite as the backing store for file records, though not the file contents

[deleted]

Re: SQLite is 35% Faster Than The Filesystem (2017)

#69

Post by jokoon »

laurent123456 wrote: post #9

Is it a known issue that the filesystem on Windows 10 is so slow? Being 5 times slower than macOS was roughly my experience but I thought there was just something wrong with my Windows laptop. I can't find any benchmark or explanation about this.

One thing I know, is that mechanical disk drives have suffered a very big drop in performance in windows 10.

I've asked this question several times, "Why would an OS need an SSD to work normally and be responsive when it was already fast without SSD some years ago", and I often get responses like I don't know how computers work, that win10 takes advantages of SSD and that it cannot work well on old techs. Even when you have a high amount of RAM and use a HDD, it's still slow.

I'm still looking for a decent filesystem or kernel developer to answer that question with a good enough explanation, because to me, I'm either stupid, or there is negligence or some attempt to increase sales of SSD. Maybe windows now uses SSD as some sort of "secondary RAM", and thus it will never be fast with a HDD.

I mean, linux desktops seem to still be fast enough with mechanical HDD.

Re: SQLite is 35% Faster Than The Filesystem (2017)

#70

Post by logifail »

tyingq wrote: post #29
jjoonathan wrote: post #20

Earlier quoted context omitted.

It isn't just Windows 10, as far as I know every version of windows has always wanted to spend like a minute (or multiple minutes!) slowly counting the files before it proceeds to slowly delete them one by one... until it finds one that's open in a background process and nixes the whole operation in a half complete state. 5 times slower sounds conservative. It's awful. Hey, maybe they could put rounded corners on the…

I think that's warts in the file explorer more so than warts in the system itself.

> warts in the file explorer more so than warts in the system itself

I'm struggling to understand why one shouldn't treat the behaviour of explorer.exe as simply part of the Windows system.

Are there third-party explorer.exe alternatives available?

If so, during Windows install, where do I click to _not_ install the MSFT explorer.exe and instead select one of the better alternatives?

Post reply on HN