51–60 of 112 posts
mksquashfs supports gzip, xz, lzo, lz4 and zstd too, you can also compile it to have any of those as a default instead of gzip. Does the performance benchmark show DwarFS versus single-threaded gzip compressed SquashFS?
> Parallel mksquashfs: Using 12 processors
It looks like the benefit is some kind of block or file deduplication. @OP: Can you please explain why you keep 50 gigs of perl around? :-) I use compressed read-only file systems all the time to save space on my travel laptop. I have one squashfs for firefox, one for the TeX base install, one for LLVM, one for qemu, one for my cross compiler collection. I suspect the gains over squashfs will be far less pronounced t…
Sure. I've been the maintainer of a perl portability module (Devel::PPPort) for a long time and every release was tested against basically every possible version (and several build flag permutations) of perl that was potentially out in the wild.
Earlier quoted context omitted.
This is a read-only file system, so it’s able to exploit certain properties of that—- locating similar files next to each other, for example.
I don’t see how read only helps at all. Btrfs can dedupe at the block level.
For example, when dedupping in block level it needs to know (right?) how many times a block is being used, so it can be collected when it runs out of uses.
ISO9660 can also express dedupped (hardlinked) files with the Rock Ridge extensions. I don't know but I'm wondering it could even do block-level dedupping if the generating program abused the format a bit..
Neat! I'd like to see benchmarks for more typical squashfs payloads-- embedded root filesystems totalling under 100MB. Small docker images like alpine would be a decent proxy. The given corpus of thousands of perl versions is more appropriate for comparison against git.
Author here :) I'll add more benchmarks, this is still WIP and so far I've mainly tried to satisfy my own needs. My intention with DwarFS wasn't to write "a better SquashFS", but to make it better in certain scenarios (huge, highly redundant data) than SquashFS. SquashFS still has the big advantage of being part of the kernel, which makes it a lot more attractive for things like root file systems.
Earlier quoted context omitted.
Author here :) I'm not sure low-spec hardware is necessarily the best use case for DwarFS. It doesn't necessarily use less resources than SquashFS, although it can create file systems that are smaller with much less CPU resources. However, it'll still need a reasonable amount of memory at run time to cache active, decompressed blocks.
Are you doing your own caching in userspace, or are you working with the kernel's caching? The latter would substantially reduce memory requirements.
Actually, I guess if DwarFS is a kernel module and decompresses blocks before they hit the kernel's filesystem cache, then the kernel cache would do it? I'm not sure how to tell from the README if DwarFS is a kernel module or not. So I guess I'm just confused and looking to learn -- what kind of kernel caching did you have in mind?
> I started working on DwarFS in 2013 and my main use case and major motivation was that I had several hundred different versions of Perl that were taking up something around 30 gigabytes of disk space, and I was unwilling to spend more than 10% of my hard drive keeping them around for when I happened to need them. It fills me with joy that someone has been coding a fs for 7 years due to perl installs taking too much…
Thanks to mhx I can move them now back to my fast disk. This is also perfect for testers.
I wish there was a semi-compressed transparent filesystem layer which slowly compresses the least recently used files in the background, and un-compresses files upon use. That way you could store much more mostly unused content than space on the disk, without sacrificing accessibility.
It is not what you describe but it can help.
Earlier quoted context omitted.
This is a read-only file system, so it’s able to exploit certain properties of that—- locating similar files next to each other, for example.
I don’t see how read only helps at all. Btrfs can dedupe at the block level.
Earlier quoted context omitted.
Author here :) I'm not sure low-spec hardware is necessarily the best use case for DwarFS. It doesn't necessarily use less resources than SquashFS, although it can create file systems that are smaller with much less CPU resources. However, it'll still need a reasonable amount of memory at run time to cache active, decompressed blocks.
Are you doing your own caching in userspace, or are you working with the kernel's caching? The latter would substantially reduce memory requirements.
$ dwarfs perl-install.dwarfs mnt -f
23:02:42.673390 dwarfs (0.2.1)
23:02:42.676663 file system initialized [1.94ms]
23:02:49.210158 blocks created: 226
23:02:49.210189 blocks evicted: 194
23:02:49.210216 request sets merged: 123
23:02:49.210241 total requests: 50056
23:02:49.210270 active hits (fast): 1515
23:02:49.210293 active hits (slow): 833
23:02:49.210318 cache hits (fast): 47482
23:02:49.210343 cache hits (slow): 0
23:02:49.210392 fast hit rate: 97.8844%
23:02:49.210417 slow hit rate: 1.66414%
23:02:49.210441 miss rate: 0.451494%