35% Faster Than The Filesystem (2017) • Hacker News Board

Re: 35% Faster Than The Filesystem (2017)

#111

Post by majewsky » Sun, Aug 18, 2019, 6:21 PM UTC

GordonS wrote: post #101

wayneftw wrote: post #50

Wow, so maybe instead of a node_modules folder, npm should use a node_modules.db?

That would be amazing! To work though, you'd need some kind of interface to actually access the contents. I guess that's doable, but not ideal. Someone else posted libsqlfs, which is FUSE Sqlite file system, and should do the job...not exactly plug-and-play though, and presumably wouldn't work on Windows (maybe on WSL tho?)

Not sure if /s :)

With a FUSE filesystem, you're still incurring the syscall overhead that's causing this performance disparity in the first case, plus now the kernel has to context-switch into the FUSEing process. So what would be 2 context switches for each syscall (program -> kernel -> program) now is 3 or 4 context switches (program -> kernel -> FUSE -> (kernel ->) program).

FUSE has some amazing usecases, but performance is none of them.

Re: 35% Faster Than The Filesystem (2017)

#112

Post by derefr » Sun, Aug 18, 2019, 6:27 PM UTC

notyourday wrote: post #102

derefr wrote: post #100

Earlier quoted context omitted.

Show me a filesystem that can efficiently hold onto all the inode information for a blockchain represented as files and directories. (Hint: not even LevelDB can hold onto all the trie information efficiently; solutions are being sought that pack things tighter than LevelDB.)

You don't need to do it efficiently . You are optimizing for non-existent problem. Here's what you actually need to optimize for: you have a hundred million files. The are all somehow reachable via http://origin/someuniqueurl . You have multiple copies because you aren't an idiot and you know that users actually hate either losing or getting corrupted files back. You have fingerprints associated with every copy. Some…

Er, yes, you do need to do it efficiently. A blockchain needs to read about 1000 of these “files” per transaction in order to verify the next block (and there are ~10000 transactions per block, and the accesses have no special locality because they’re a mixture of requests submitted by unaffiliated parties.) It’s an at-scale OLTP system (ingesting thousands of TPS) that is required to execute on consumer hardware.

This is a real problem. The current implementation of Ethereum is IOPS-bound precisely because of this overhead, with ~80% of the time spent verifying a block being spent on the filesystem and index-access overhead of getting the relevant data into memory, rather than being IO-bound by the actual bandwidth of data required. This is an on-disk datastructures problem.

Re: 35% Faster Than The Filesystem (2017)

#113

Post by SQLite » Sun, Aug 18, 2019, 6:31 PM UTC

muststopmyths wrote: post #104

stat to get filesize is a pathlogically bad pattern on Windows. If I remove the call to fileSize() in kvtest and instead replace it with 16k (since the max size in the post is 12k), I can significantly improve the time measured on Windows 10. Before: C:\temp>kvtest run test1.dir --count 100k --blob-api --count 100000 --max-id 1000 --asc Total elapsed time: 9.495 Microseconds per BLOB read: 94.950 Content read rate: 1…

Thanks for this. I did not realize that fileSize() was a performance issue on Windows. I'll attempt to update kvtest.c to deal with that, rerun the tests, and update the page, as I have opportunity.

Re: 35% Faster Than The Filesystem (2017)

#114

Post by GordonS » Sun, Aug 18, 2019, 6:35 PM UTC

majewsky wrote: post #111

GordonS wrote: post #101

Earlier quoted context omitted.

That would be amazing! To work though, you'd need some kind of interface to actually access the contents. I guess that's doable, but not ideal. Someone else posted libsqlfs, which is FUSE Sqlite file system, and should do the job...not exactly plug-and-play though, and presumably wouldn't work on Windows (maybe on WSL tho?)

Not sure if /s :) With a FUSE filesystem, you're still incurring the syscall overhead that's causing this performance disparity in the first case, plus now the kernel has to context-switch into the FUSEing process. So what would be 2 context switches for each syscall (program -> kernel -> program) now is 3 or 4 context switches (program -> kernel -> FUSE -> (kernel ->) program). FUSE has some amazing usecases, but pe…

Erm, no, no sarcasm, but in truth I know very little about FUSE, beyond it being an API for userland file systems.

I'm also approaching this as primarily a Windows user - a lot of the overhead on Windows is access checks, so my thinking is these would be implemented as noops.

Re: 35% Faster Than The Filesystem (2017)

#115

Post by notyourday » Sun, Aug 18, 2019, 6:36 PM UTC

derefr wrote: post #112

notyourday wrote: post #102

Earlier quoted context omitted.

You don't need to do it efficiently . You are optimizing for non-existent problem. Here's what you actually need to optimize for: you have a hundred million files. The are all somehow reachable via http://origin/someuniqueurl . You have multiple copies because you aren't an idiot and you know that users actually hate either losing or getting corrupted files back. You have fingerprints associated with every copy. Some…

Er, yes, you do need to do it efficiently. A blockchain needs to read about 1000 of these “files” per transaction in order to verify the next block (and there are ~10000 transactions per block, and the accesses have no special locality because they’re a mixture of requests submitted by unaffiliated parties.) It’s an at-scale OLTP system (ingesting thousands of TPS) that is required to execute on consumer hardware. Th…

Oh, this is the block chain discussion i.e. another solution in search of a problem. Never mind.

Re: 35% Faster Than The Filesystem (2017)

#116

Post by burtonator » Sun, Aug 18, 2019, 7:08 PM UTC

zaroth wrote: post #6

For small- to mid-sized projects, I’ve always realized huge gains in simplicity by haves “Files” tables to store various assets. It means instances in a web-farm can pull the files down when they initialize easily, it means files are automatically versioned, it provides an obvious place to put the files when they are being uploaded on the Admin panel. It means all the files are getting backed up as part of the databa…

For the last 6-8 months I've been working on a hybrid electron + web app that uses files on local disk for storage.

I have a Datastore implementation that uses disk when locally and firebase when remote.

https://getpolarized.io/2019/03/22/portable-datastores-and-p...

You can read about it there.

I thought it would be a great idea to use just simple flat files as you can commit them to git, diff them, and there's kind of already an API for working with them.

In retrospect I regret it and wish we had something more powerful.

First. It's twice as much code to maintain. Which introduces bugs.

Seconds if you're keeping your PDFs online vs on disk they're duplicated. It would be nice to build in a higher level abstraction that was both network AND disk aware and just pulled things down when you needed it.

Third, the annotations on your PDF aren't really indexable without a 3rd party system.

Getting the right balance of remote-first vs web-first is tough.

Now, after building a complex webapp with electron support I think it's better to go web-first and then try to use the browsers local caching to get offline-first support.

Re: 35% Faster Than The Filesystem (2017)

#117

Post by vr46 » Sun, Aug 18, 2019, 7:25 PM UTC

Billions of tech-years ago, BeOS made a filesystem that had database-like properties. Now, mainly because I have no idea what I’m talking about in this area, I don’t know if anything sprung from that into mainstream computing, so features like this from SQLite make the news. (It’s very cool, to be sure)

Re: 35% Faster Than The Filesystem (2017)

#118

Post by SiVal » Sun, Aug 18, 2019, 7:31 PM UTC

c-smile wrote: post #94

Slightly off the topic. In one of web applications (CRM alike) I was experimenting with SQLite-per-user approach. In that application user related data was clustered on per-user basis. So instead of single-db-for-all-users I had one-db-per-user. Performance gain was significant - around 25% for most of requests. I suspect that this is due the fact that index trees were significantly shorter for each particular user,…

This is exactly the topic that made me read this thread. I have been wondering whether one sqlite file per customer was a good option for an online service where each customer needed only his own, private data but wanted to work on it from his desktop, laptop, or phone, with the option of occasionally downloading the file for additional backup. I just don't know enough about multi-tenant database approaches to know if this is a good idea or absurd, and I haven't found anyone talking about it.

Any pointers to further insights would be greatly appreciated. Even a forum where (utterly unlike StackOverflow) I could ask such a question without already almost knowing the answer.

Re: 35% Faster Than The Filesystem (2017)

#119

Post by zzzcpan » Sun, Aug 18, 2019, 7:33 PM UTC

ahachete wrote: post #106

zzzcpan wrote: post #78

Earlier quoted context omitted.

Backups are fundamentally limited to eventual consistent, there is no need for databases to be synchronously replicated for backups. I mean splitting database has no effect on backup consistency, although a more decent way of dealing with it is not splitting database, but simply running an async replica to do backups from.

Backups on databases like PostgreSQL are strongly consistent, regardless of where you take it from (master or replica). Postgres replication is strictly sequential wrt transaction commit order. As such, splitting the database may incur in significant consistency issues that a backup doesn't incur into. I believe this splitting technique is not a good one except for potentially narrow use cases.

Strong consistency is not what you think then. You can only do stale reads from backups.

Re: 35% Faster Than The Filesystem (2017)

#120

Post by gigatexal » Sun, Aug 18, 2019, 7:40 PM UTC

SQLite really is the best of both worlds when it comes to files and databases: it is a file, one that can usually live entirely in memory; and it’s got all the awesome utility of SQL (even window functions too) that can be used very simply to do powerful things.