It would amazing to see this work on OpenWRT, I think it would fit perfectly using less resources than squashfs.
The other location would be on a Raspberry pi for scenarios where power can be cut at any time.
DwarFS: A fast high compression read-only file system
11–20 of 112 posts
Re: DwarFS: A fast high compression read-only file system
#12It would amazing to see this work on OpenWRT, I think it would fit perfectly using less resources than squashfs. The other location would be on a Raspberry pi for scenarios where power can be cut at any time.
I was thinking the same thing! I'm not sure what it would take to make /rom a FUSE based filesystem, to make it bootable. The current boot process involves the bare kernel mounting Squashfs to find it's init=/etc/preinit & booting from there[1].
Would love some theorycrafting on possible ways to work with DwarFS being a FUSE filesystem.
Re: DwarFS: A fast high compression read-only file system
#13It would amazing to see this work on OpenWRT, I think it would fit perfectly using less resources than squashfs. The other location would be on a Raspberry pi for scenarios where power can be cut at any time.
Author here :) I'm not sure low-spec hardware is necessarily the best use case for DwarFS. It doesn't necessarily use less resources than SquashFS, although it can create file systems that are smaller with much less CPU resources. However, it'll still need a reasonable amount of memory at run time to cache active, decompressed blocks.
Re: DwarFS: A fast high compression read-only file system
#14> I started working on DwarFS in 2013 and my main use case and major motivation was that I had several hundred different versions of Perl that were taking up something around 30 gigabytes of disk space, and I was unwilling to spend more than 10% of my hard drive keeping them around for when I happened to need them. It fills me with joy that someone has been coding a fs for 7 years due to perl installs taking too much…
Hahaha, I haven't actually been coding on this for that long, it's more that I coded for a few weeks back in 2013 and only found the motivation to resurrect the whole thing a few weeks back.
Funny thing about it is that I've got a similar problem powering https://perl.bot/ (and the associated irc bot). I don't have as many installs as you currently but It's not far off and I want to add more compile time settings to them. I'd need to setup a full build server/system though because I need to regularly update them with new modules.
How opposed would you be to this being reworked to being able to be mainline kernel support too?
Re: DwarFS: A fast high compression read-only file system
#15Why not use BTRFS with file deduplication and transparent compression (zstd specifically)?
Re: DwarFS: A fast high compression read-only file system
#16> I started working on DwarFS in 2013 and my main use case and major motivation was that I had several hundred different versions of Perl that were taking up something around 30 gigabytes of disk space, and I was unwilling to spend more than 10% of my hard drive keeping them around for when I happened to need them. It fills me with joy that someone has been coding a fs for 7 years due to perl installs taking too much…
Nowadays you can have the same problem with Python and Javascript too!
Re: DwarFS: A fast high compression read-only file system
#17It looks like the benefit is some kind of block or file deduplication.
@OP: Can you please explain why you keep 50 gigs of perl around? :-)
I use compressed read-only file systems all the time to save space on my travel laptop. I have one squashfs for firefox, one for the TeX base install, one for LLVM, one for qemu, one for my cross compiler collection. I suspect the gains over squashfs will be far less pronounced than for the pathological "400 perl version".
Re: DwarFS: A fast high compression read-only file system
#18Earlier quoted context omitted.
Hahaha, I haven't actually been coding on this for that long, it's more that I coded for a few weeks back in 2013 and only found the motivation to resurrect the whole thing a few weeks back.
Funny thing about it is that I've got a similar problem powering https://perl.bot/ (and the associated irc bot). I don't have as many installs as you currently but It's not far off and I want to add more compile time settings to them. I'd need to setup a full build server/system though because I need to regularly update them with new modules. How opposed would you be to this being reworked to being able to be mainlin…
> How opposed would you be to this being reworked to being able to be mainline kernel support too?
I don't see any way of getting this anywhere near the kernel without a full rewrite. It's C++ and it depends on libraries that aren't even shipped by a lot of distributions (folly & fbthrift). And, tbh, I don't see much benefit given that FUSE these days doesn't seem to be significantly worse in terms of performance.
> I'd need to setup a full build server/system though because I need to regularly update them with new modules.
Overlay the mounted read-only fs with a read-write fs. Then you can install modules as you like and if you want to start fresh, just throw away the read-write fs. That's what I've done in the past.
Re: DwarFS: A fast high compression read-only file system
#19I wish there was a semi-compressed transparent filesystem layer which slowly compresses the least recently used files in the background, and un-compresses files upon use. That way you could store much more mostly unused content than space on the disk, without sacrificing accessibility.
I believe NT file compression works like this, and before that MSDOS "DriveSpace" ...
Re: DwarFS: A fast high compression read-only file system
#20What are the use cases for a read only file system?
Booting. Arguably all containers, too.