Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

SQLite is 35% Faster Than The Filesystem (2017)

sqlite.org

Post reply on HN
Print view View original

131–137 of 137 posts

Re: SQLite is 35% Faster Than The Filesystem (2017)

#131

Post by blooalien »

gnubison wrote: post #128
blooalien wrote: post #110

Earlier quoted context omitted.

> Year of Linux on the Desktop when. Couple decades ago for me. > Yeah, 'cause the incredible fragmentation of Linux GUI toolkits and windowing systems is so fun to deal with. Sounds like probably "never" for you. You seem hostile toward Linux, so you're probably better off sticking with whatever you use currently.

I don’t think calling GP “hostile to Linux” after they critiqued one part of the ecosystem is very fair. All systems have their flaws, and if judging part of a system negatively makes you hostile to that system, then I guess Linus Torvalds is hostile to Linux too.

I no longer care. None of it matters one little bit anyhow.

Re: SQLite is 35% Faster Than The Filesystem (2017)

#132

Post by hughrr »

rossy wrote: post #126
hughrr wrote: post #78

Earlier quoted context omitted.

I'm going to have to complain about this because in real world use these are far less of a problem than MFT contention on sub-900 byte files generated by typical unix environments. Particularly things that are lockfile heavy and VCSs etc are the most painful. The WSL thing reads like an excuse here. If you actually go and look at what's happening it's small files which are the damage multiplier. I think the real issu…

I'm sorry, but I find that hard to believe when the linked posts are from a Microsoft employee, who says repeatedly that the problem isn't a simple as "NTFS is slow," that they spent a release optimising NTFS, they've gotten rid of all the low hanging fruit and so on. I don't see why they'd be making excuses for NTFS when the underlying problems seem to be much more fundamental to Windows, which is much worse. You sa…

No it’s still MFT contention. Someone at my company independently discovered the same problem recently with git on WSL.

MSFT don’t listen to or care about their customers findings from experience. That’s why canned our partner status.

In this case they haven’t touched NTFS at all - that’s out right rubbish.

I pointed out the paradigm mismatch already. Run unix loads on unix not NT.

Re: SQLite is 35% Faster Than The Filesystem (2017)

#133

Post by Seattle3503 »

SQLite also performs well when there is a large number of "files". A simulation I wrote used a large number of files (10k+). Eventually I had to transition away from using the file system as a key-value store because open, reads, writes, and directory listings were slow when you have that many files in a single directory and that many open file descriptors.

Re: SQLite is 35% Faster Than The Filesystem (2017)

#134

Post by Seattle3503 »

xxs wrote: post #34

No idea what's with the sqlite articles making the front page every other day. This one is pretty old as well. The measurements in this article were made during the week of 2017-06-05 using a version of SQLite in between 3.19.2 and 3.20.0.

Because SQLite is the real deal. We see a lot of hype about technologies that have come and gone, but SQLite has stood the test of time AND is a pleasure to use.

In some way it is the Platonic Ideal of what a SQL database should be (from a a developer's perspective). There isn't any mucking about with dev-ops configuration, because the DB is file based. Just type "sqlite3 out.db" and you have a database to write queries against. At the same time it still manages to be quite performant.

Re: SQLite is 35% Faster Than The Filesystem (2017)

#135

Post by wscott »

phepranto wrote: post #102
laurent123456 wrote: post #9

Is it a known issue that the filesystem on Windows 10 is so slow? Being 5 times slower than macOS was roughly my experience but I thought there was just something wrong with my Windows laptop. I can't find any benchmark or explanation about this.

At the end of the article it says: "SQLite is much faster than direct writes to disk on Windows when anti-virus protection is turned on. Since anti-virus software is and should be on by default in Windows, that means that SQLite is generally much faster than direct disk writes on Windows. "

This is a key point. Windows is starting up the anti-virus program and scanning every tiny file as it is written. This is less about NTFS being slow and more about anti-virus being considered a required overhead on Windows.

Re: SQLite is 35% Faster Than The Filesystem (2017)

#136

Post by Tostino »

naikrovek wrote: post #111
masklinn wrote: post #77

Earlier quoted context omitted.

> NTFS comes in different versions, you know. A completely pointless remark when versions of NTFS and Windows are correlated. And AFAIK there's been no updated to NTFS itself since the addition of symlinks in NTFS 3.1, released with Windows XP. All "filesystem features" since were added at the level OS, using existing NTFS features. > If the limitations of the filesystem do not change between operating systems, then…

>> It's […] not any specific version of Windows. > is bullshit. No, it's not. NTFS is slow, even when it's running under Linux. NTFS has a very pronounced weakness, here. Performance changing between Windows 7 and Windows 10 means that ON TOP OF NTFS there are other performance problems in Windows 10. It does not mean that it isn't an NTFS problem in the case of Windows 10 -- it means that it isn't ONLY an NTFS probl…

You are making a point that truly doesn't matter.

Re: SQLite is 35% Faster Than The Filesystem (2017)

#137

Post by naikrovek »

Tostino wrote: post #136
naikrovek wrote: post #111

Earlier quoted context omitted.

>> It's […] not any specific version of Windows. > is bullshit. No, it's not. NTFS is slow, even when it's running under Linux. NTFS has a very pronounced weakness, here. Performance changing between Windows 7 and Windows 10 means that ON TOP OF NTFS there are other performance problems in Windows 10. It does not mean that it isn't an NTFS problem in the case of Windows 10 -- it means that it isn't ONLY an NTFS probl…

You are making a point that truly doesn't matter.

everyone on HN does that 100% of the time they comment. it's not the "things that matter" section of the site, it's the "comment" section.
Post reply on HN