What are the use cases for a read only file system?
squashfs is widely used in Linux install media.
DwarFS: A fast high compression read-only file system
41–50 of 112 posts
Re: DwarFS: A fast high compression read-only file system
#42Is this viable as a backup/archive format? Would it make sense to e.g. have an incremental backup as a DwarFS file, referring to the base backup in another DwarFS file?
I guess something like borgbackup would be better suited for this.
You could theoretically try to build this with dwarfs, by using overlayfs and then compressing the upper layer again with dwarfs, but that sounds pretty fragile and cumbersome.
Re: DwarFS: A fast high compression read-only file system
#43It looks like the benefit is some kind of block or file deduplication. @OP: Can you please explain why you keep 50 gigs of perl around? :-) I use compressed read-only file systems all the time to save space on my travel laptop. I have one squashfs for firefox, one for the TeX base install, one for LLVM, one for qemu, one for my cross compiler collection. I suspect the gains over squashfs will be far less pronounced t…
AppFS provides global file deduplication and also solves the distribution problem, and also you don't need to have all the resources locally
Re: DwarFS: A fast high compression read-only file system
#44Earlier quoted context omitted.
Funny thing about it is that I've got a similar problem powering https://perl.bot/ (and the associated irc bot). I don't have as many installs as you currently but It's not far off and I want to add more compile time settings to them. I'd need to setup a full build server/system though because I need to regularly update them with new modules. How opposed would you be to this being reworked to being able to be mainlin…
> How opposed would you be to this being reworked to being able to be mainline kernel support too? I don't see any way of getting this anywhere near the kernel without a full rewrite. It's C++ and it depends on libraries that aren't even shipped by a lot of distributions (folly & fbthrift). And, tbh, I don't see much benefit given that FUSE these days doesn't seem to be significantly worse in terms of performance. >…
> Overlay the mounted read-only fs with a read-write fs. Then you can install modules as you like and if you want to start fresh, just throw away the read-write fs. That's what I've done in the past.
It would be nice to be able to build a new read-only filesystem in incremental mode: given a compressed filesystem and some new uncompressed data, incorporate the uncompressed data without completely re-doing all the work.
Re: DwarFS: A fast high compression read-only file system
#45It would amazing to see this work on OpenWRT, I think it would fit perfectly using less resources than squashfs. The other location would be on a Raspberry pi for scenarios where power can be cut at any time.
Author here :) I'm not sure low-spec hardware is necessarily the best use case for DwarFS. It doesn't necessarily use less resources than SquashFS, although it can create file systems that are smaller with much less CPU resources. However, it'll still need a reasonable amount of memory at run time to cache active, decompressed blocks.
Are you doing your own caching in userspace, or are you working with the kernel's caching? The latter would substantially reduce memory requirements.
Re: DwarFS: A fast high compression read-only file system
#46Re: DwarFS: A fast high compression read-only file system
#47Earlier quoted context omitted.
For most use-cases, yes. But not if you plan on using raid5+ [0] [0]: https://btrfs.wiki.kernel.org/index.php/Gotchas#Parity_RAID My own experience indicates it's brittle to power failures and will corrupt in annoying ways in the event of a power failure (or hard reboot) as of ~1 year ago.
Year isnt as useful as kernel version.
I was running whatever Arch's Linux kernel was at the time. 5.4 I believe? Arch is pretty much bleeding edge, so year is relevant.
Re: DwarFS: A fast high compression read-only file system
#48mksquashfs supports gzip, xz, lzo, lz4 and zstd too, you can also compile it to have any of those as a default instead of gzip.
Does the performance benchmark show DwarFS versus single-threaded gzip compressed SquashFS?
Re: DwarFS: A fast high compression read-only file system
#49Re: DwarFS: A fast high compression read-only file system
#50> You can pick either clang or g++, but at least recent clang versions will produce substantially faster code
have you investigated why this might be the case?