Earlier quoted context omitted.
I'm only ever switching to a new filesystem if its correctness has been formally verified.
For which filesystems is that case?
None. But the tools are there, so it should be possible.
DwarFS: A fast high compression read-only file system
101–110 of 112 posts
Re: DwarFS: A fast high compression read-only file system
#102Earlier quoted context omitted.
My Python shits have pip requirements that easily dump 3-4 gigs in a venv folder. Do that once or twice a month when starting a new project for a couple of years and it gets messy...
I'd like to see a pip freeze of whatever you're doing to consistently need venvs of that size.
affine==2.3.0
attrs==20.3.0
certifi==2020.11.8
click==7.1.2
click-plugins==1.1.1
cligj==0.7.1
cycler==0.10.0
dataclasses==0.6
decorator==4.4.2
Fiona==1.8.17
future==0.18.2
geopandas==0.8.1
imageio==2.9.0
joblib==0.17.0
kiwisolver==1.3.1
llvmlite==0.34.0
matplotlib==3.3.0
munch==2.5.0
networkx==2.5
numba==0.51.2
numpy==1.19.2
pandas==1.1.3
Pillow==7.2.0
pkg-resources==0.0.0
psycopg2-binary==2.8.6
PyCRS==1.0.1
pyparsing==2.4.7
pyproj==3.0.0.post1
python-dateutil==2.8.1
pytz==2020.4
PyWavelets==1.1.1
rasterio==1.1.8
scikit-image==0.17.2
scikit-learn==0.23.2
scipy==1.5.2
Shapely==1.7.1
six==1.15.0
snuggs==1.4.7
threadpoolctl==2.1.0
tifffile==2020.11.18
torch==1.7.0
tqdm==4.48.2
typing-extensions==3.7.4.3
Re: DwarFS: A fast high compression read-only file system
#103So I tried it out on my 17BG of perl builds. (just on my laptop, not on my big machine). mkdwarfs crashed with recursive links (1-level, just pointing to itself) and when I removed dirs while running mkdwarfs, which were part of of the input path. Which is fair, I assume.
On success, mkdwarfs needed 1 hr, and reduced 219 dirs to a size of 970 MB. Not just source files, but also the build and install object files.
1 hr is a lot, but just think how long squashfs would have needed. Totally impractical. Thanks mhx
Re: DwarFS: A fast high compression read-only file system
#104Earlier quoted context omitted.
Most ZFS howto's now recommend against dedup on the prolongued memory cost consequences. Yes, you would get some block level compression outcome. But, you enter the cost/benefit hell of balancing CPU and memory at runtime.
Can't you periodically run the dedup out of band (for example whenever you scrub)? https://btrfs.wiki.kernel.org/index.php/Deduplication
I was just googling this myself and I think this is a feature that btrfs has over zfs. There’s no way to do native offline deduplication as far as I could find.
Re: DwarFS: A fast high compression read-only file system
#105It looks like the benefit is some kind of block or file deduplication. @OP: Can you please explain why you keep 50 gigs of perl around? :-) I use compressed read-only file systems all the time to save space on my travel laptop. I have one squashfs for firefox, one for the TeX base install, one for LLVM, one for qemu, one for my cross compiler collection. I suspect the gains over squashfs will be far less pronounced t…
Speculating here, but perl has a very rich test library and harnesses for running tests across multiple perls and platforms. If you upload a module to CPAN, you automatically get it tested against a huge matrix of configurations: http://matrix.cpantesters.org/?dist=Log-Any-Adapter-FileHand...
> If you upload a module to CPAN, you automatically get it tested against a huge matrix of configurations.
Very true, and it's definitely a great service!
However, the set of versions/configurations is still limited, and it can take an awful lot of time for the matrix to fill up. I've fixed a bug specific to perl-5.10.0 about a week ago and so far the module hasn't been picked up by that version again.
So while this is definitely good as a service for the general public, it doesn't get you very far if you're trying to build a thing that's supposed to ensure compatibility for other Perl modules across 20 years of Perl history. :)
Re: DwarFS: A fast high compression read-only file system
#106> You can pick either clang or g++, but at least recent clang versions will produce substantially faster code have you investigated why this might be the case?
> have you investigated why this might be the case?
Very briefly. It looks like clang has a different strategy breaking up the code (which is mostly C++ templates) into actual functions vs. inlining it, and the hot code ultimately performs fewer function calls with clang than it does with gcc. But this is nowhere near a proper analysis of what's going on. :)
Re: DwarFS: A fast high compression read-only file system
#107Circa 2 years ago, I was working on a side project and got so annoyed with SquashFS tooling, that I decided to fix it instead. After getting stuck with the spaghetti code behind mksquashfs, I decided to start from scratch, having learnt enough about SquashFS to roughly understand the on-disk format. Because squashfs-tools seemed pretty unmaintained in late 2018 (no activity on the official site & git tree for years a…
I just want to say thank you for squashfs-tools-ng. For my usecase I had to patch mksquashfs and your tool fits just right. I'm yet to switch however.
Re: DwarFS: A fast high compression read-only file system
#108Earlier quoted context omitted.
> How opposed would you be to this being reworked to being able to be mainline kernel support too? I don't see any way of getting this anywhere near the kernel without a full rewrite. It's C++ and it depends on libraries that aren't even shipped by a lot of distributions (folly & fbthrift). And, tbh, I don't see much benefit given that FUSE these days doesn't seem to be significantly worse in terms of performance. >…
Is it possible to rebuild a DwarFS fs to incorporate changes from an overlay fs without decompressing, then recompressing? It seems feasible that a second DwarFS fs could be built from an overlay/DwarFS, then delete the original overlay/DwarFS fs. That would require 2N storage as the new DwarFS is being built. Is it possible to patch an existing DwarFS? By overlay, are you referring to overlayfs [0]? https://wiki.arc…
Yes, and I've documented this now:
https://github.com/mhx/dwarfs/blob/main/doc/dwarfs.md#settin...
Re: DwarFS: A fast high compression read-only file system
#109Earlier quoted context omitted.
If you're talking about the kernel's filesystem cache, wouldn't that cache the compressed files? As far as I understand it userspace caching is necessary to cache uncompressed blocks, since the decompression is (presumably) done in userspace. I definitely could be wrong though, let me know if you're talking about a different kind of kernel caching. Actually, I guess if DwarFS is a kernel module and decompresses block…
It's using FUSE, and FUSE filesystems still participate in some parts of kernel caching.
Thanks, good to know!
Re: DwarFS: A fast high compression read-only file system
#110So I tried it out on my 17BG of perl builds. (just on my laptop, not on my big machine). mkdwarfs crashed with recursive links (1-level, just pointing to itself) and when I removed dirs while running mkdwarfs, which were part of of the input path. Which is fair, I assume.
> mkdwarfs crashed with recursive links (1-level, just pointing to itself)
That's odd, it shouldn't crash with links at all, as it doesn't actively follow links. Can you please file a bug if you can reproduce this?
> and when I removed dirs while running mkdwarfs, which were part of of the input path
I guess this is fair, but I'll try to take a look anyway. :-)
> On success, mkdwarfs needed 1 hr, and reduced 219 dirs to a size of 970 MB. Not just source files, but also the build and install object files.
My 500 MB image with the 1100+ perls is just installations, from which I've actually removed libperl.a as I've never needed it and it really bloats the image. I've got a separate image with debug information (everything built with -g in case I need to debug the binaries), so the binaries in the main image are essentially all stripped. If I need to debug, I'll just mount the debug image as well, which contains the source files and the stripped debug data.
> 1 hr is a lot, but just think how long squashfs would have needed.
It might be worth trying a lower compression level, especially if you find that mkdwarfs is CPU bound and not I/O bound.