101–110 of 166 posts
Wow, so maybe instead of a node_modules folder, npm should use a node_modules.db?
To work though, you'd need some kind of interface to actually access the contents. I guess that's doable, but not ideal.
Someone else posted libsqlfs, which is FUSE Sqlite file system, and should do the job...not exactly plug-and-play though, and presumably wouldn't work on Windows (maybe on WSL tho?)
Earlier quoted context omitted.
> By putting all data in the db you have less moving parts, transactions, referential integrity, well defined behaviours, etc If you are doing it for 100k files and below you are using a Ferrari to pick up eggs in a corner store. If you are using it at 100k files and above, you are using a Ferrari to move a pile of paving stones one by one.
Show me a filesystem that can efficiently hold onto all the inode information for a blockchain represented as files and directories. (Hint: not even LevelDB can hold onto all the trie information efficiently; solutions are being sought that pack things tighter than LevelDB.)
Here's what you actually need to optimize for: you have a hundred million files. The are all somehow reachable via http://origin/someuniqueurl. You have multiple copies because you aren't an idiot and you know that users actually hate either losing or getting corrupted files back. You have fingerprints associated with every copy. Something happens and you can't reach part of the tree or you are getting incorrect hashes ( which you know because your origin computes the hashes every time someone requests and one and if those hashes don't match it triggers a re-request to a backup copy ). And now you need to
(a) ensure you still have the needed protection factor ( you you went from 3x to 2x as one copy is dead ) for all affected files
(b) minimize the time needed to remove the "thing" ( probably a disk or a node ) that caused the failure.
(c) minimize the cost of ensuring (a) and (b)
People spent lots of time figuring out very clever ways of doing it via DB. It neither scaled nor worked well when real life (i.e. disk crashes/nodes going away/bad data was returned) happened.
The only thing that you really need to know is
(a) where the data and its copies are stored.
(b) what is the hash of the data that is stored there
(c) how fast you can recover the information about where that data is stored in event of the issue with whatever the system that is used to keep track between (a) and (b).
(d) preferably you also want to know what other data objects ( files ) could be affected when some of the other objects are misbehaving ( if you know a file at nodeA/volumeB had a fingerprint Ykntr8H8pL9PyAtCwdw/CB5tToXTPf55+hKSZb0uhV0 when it was written and now for some reason it says its fingerprint is sw0SI4jkU5VVk6CH4oPYtwx+bK2hIlrw8hVM7i9zmNk while the rest of the copies are saying their fingerprint is still Ykntr8H8pL9PyAtCwdw/CB5tToXTPf55+hKSZb0uhV0 you could decide that you want to trash the nodeX/volumeB because you no longer trust it.
Yes, open() can introduce significant overhead, especially when doing it sequentially on a remote filesystem, e.g. NFS.
But that's using the filesystems in the most inefficient way possible, there are better ways to do it. E.g. you can pre-touch the files or even pre-open them in parallel and then point to /proc/self/fd/ instead, which turns an open() into a dup(). I was able to make some properietary software run 5x faster that way simply by feeding it already open file descriptors.
Before:
C:\temp>kvtest run test1.dir --count 100k --blob-api
--count 100000 --max-id 1000 --asc
Total elapsed time: 9.495
Microseconds per BLOB read: 94.950
Content read rate: 104.0 MB/s
C:\temp>kvtest run test1.dir --count 100k --blob-api
--count 100000 --max-id 1000 --asc
Total elapsed time: 5.218
Microseconds per BLOB read: 52.180
Content read rate: 313.3 MB/s
Windows is a completely different beast than Unix and you need to know the beast to tame it.
You might say that stat+open+read+close should be compared/measured together. In that case my answer would be that you would not read all files in a directory in this way on Windows anyway. You'd use overlapped I/O with buffers of FS cache granularity (which is 64Kb, unless it has changed in recent versions of Windows). It wouldn't matter for smaller files anyway (since the OS aggressively reads ahead when you open a file) and would be better for larger files.
It is a bugbear of mine when I see complaints about Windows from people who never actually tried to optimize for the OS.
reading blobs in Sqlite from db is still ~3x faster on my Surface Pro 2017 (don't call it Surface Pro 5!), but that's to be expected.
For small- to mid-sized projects, I’ve always realized huge gains in simplicity by haves “Files” tables to store various assets. It means instances in a web-farm can pull the files down when they initialize easily, it means files are automatically versioned, it provides an obvious place to put the files when they are being uploaded on the Admin panel. It means all the files are getting backed up as part of the databa…
I though, one of the advantage here is, you are able to store/read metadata along the file content easy without worrying the metadata and file may desync in some way or corrupted. This makes serve something like Image (photo dimension) or video(length) metadata way easier. Some filesystem (like ntfs) do have place to save metadata along files, but most filesystem don't have the way to done that properly
> In Linux, the ext2, ext3, ext4, JFS, Squashfs, Yaffs2, ReiserFS, Reiser4, XFS, Btrfs, OrangeFS, Lustre, OCFS2 1.6, ZFS, and F2FS[9] filesystems support extended attributes (abbreviated xattr) when enabled in the kernel configuration.
> The Linux kernel allows extended attribute to have names of up to 255 bytes and values of up to 64KiB,[13] as do XFS and ReiserFS, but ext2/3/4 and btrfs impose much smaller limits, requiring all the attributes (names and values) of one file to fit in one "filesystem block" (usually 4 KiB).
Applications that make extensive use of xattrs typically recommend one of the FSs that allow full-size xattrs. For example, OpenStack Swift, an object storage service, recommends XFS as a backing storage. (Source: I operate Swift clusters at $dayjob.) But if your metadata is not larger than 4 KiB, any production-quality Linux FS will do.
Earlier quoted context omitted.
Doesn't that introduce the desync problem again? Is there something that can enforce consistency across the two databases?
Backups are fundamentally limited to eventual consistent, there is no need for databases to be synchronously replicated for backups. I mean splitting database has no effect on backup consistency, although a more decent way of dealing with it is not splitting database, but simply running an async replica to do backups from.
As such, splitting the database may incur in significant consistency issues that a backup doesn't incur into.
I believe this splitting technique is not a good one except for potentially narrow use cases.
Earlier quoted context omitted.
This sounds like a fantastic idea. You would go from 4000 directories and 50000 files to a single file with b-tree indices, hashes, etc. You could probably get a 100x speedup over the current approach.
Here you go: https://github.com/guardianproject/libsqlfs
[1] Calling it Nix confuses with NixOS...
Wow, so maybe instead of a node_modules folder, npm should use a node_modules.db?
That would be amazing! To work though, you'd need some kind of interface to actually access the contents. I guess that's doable, but not ideal. Someone else posted libsqlfs, which is FUSE Sqlite file system, and should do the job...not exactly plug-and-play though, and presumably wouldn't work on Windows (maybe on WSL tho?)
Perhaps Dokan? Dokan has a FUSE wrapper [1]
For small- to mid-sized projects, I’ve always realized huge gains in simplicity by haves “Files” tables to store various assets. It means instances in a web-farm can pull the files down when they initialize easily, it means files are automatically versioned, it provides an obvious place to put the files when they are being uploaded on the Admin panel. It means all the files are getting backed up as part of the databa…
My (naive?) view is that Nginx caches static files by default after the first hit so the inconvenience isn't worth it in my web apps. Would I really see an improvement switching to a SQLite-esque file system?
That's likely the OS's filesystem cache, not anything in nginx. If you're on Linux, you can confirm this by clearing the OS filesystem cache with
$ sudo -i
# echo 3 > /proc/sys/vm/drop_caches
For small- to mid-sized projects, I’ve always realized huge gains in simplicity by haves “Files” tables to store various assets. It means instances in a web-farm can pull the files down when they initialize easily, it means files are automatically versioned, it provides an obvious place to put the files when they are being uploaded on the Admin panel. It means all the files are getting backed up as part of the databa…
The files were the storage for some third-party content targeting tool, and were edited using an admin UI that tool, then deployed to the servers where the live part of the tool would read them. You couldn't bring the system up in a complete state without them.
Apart from those files, we could restore the whole system using the latest code (really, the artifact built from it), and the latest database dump. So, before taking a database dump, we'd run a script to load those files into the database. We also had a script for undumping them, which we tested regularly in our staging environment.
We didn't use the database for deployment, because the third-party tool did that. Perhaps we should have - it would have been a bit of a subversion of the tool's usual mode of operation, but it might have had some advantages.