DwarFS: A fast high compression read-only file system
1–10 of 112 posts
Re: DwarFS: A fast high compression read-only file system
#2Neat! I'd like to see benchmarks for more typical squashfs payloads-- embedded root filesystems totalling under 100MB. Small docker images like alpine would be a decent proxy. The given corpus of thousands of perl versions is more appropriate for comparison against git.
Re: DwarFS: A fast high compression read-only file system
#3Whew! It was easy to find out how you actually initialize this thing, if it's read-only:
Re: DwarFS: A fast high compression read-only file system
#4I wish there was a semi-compressed transparent filesystem layer which slowly compresses the least recently used files in the background, and un-compresses files upon use. That way you could store much more mostly unused content than space on the disk, without sacrificing accessibility.
Re: DwarFS: A fast high compression read-only file system
#5I wish there was a semi-compressed transparent filesystem layer which slowly compresses the least recently used files in the background, and un-compresses files upon use. That way you could store much more mostly unused content than space on the disk, without sacrificing accessibility.
You could probably build something easily in nbdkit to do this. (Note this is at the block layer). An advantage of nbdkit is you could write the whole thing in the high-level language of your choice, even a scripting language such as Python, which might make it easier to rapidly explore designs.
Having said that I did try to implement a deduplication layer for nbdkit, but what I found was that it wasn't very effective. It turns out that duplicate data in typical VM filesystems isn't common, and the other parts of the filesystem (block free lists etc) were not sufficiently similar to deduplicate given my somewhat naive approach.
Re: DwarFS: A fast high compression read-only file system
#6What are the use cases for a read only file system?
Re: DwarFS: A fast high compression read-only file system
#7Neat! I'd like to see benchmarks for more typical squashfs payloads-- embedded root filesystems totalling under 100MB. Small docker images like alpine would be a decent proxy. The given corpus of thousands of perl versions is more appropriate for comparison against git.
Author here :)
I'll add more benchmarks, this is still WIP and so far I've mainly tried to satisfy my own needs. My intention with DwarFS wasn't to write "a better SquashFS", but to make it better in certain scenarios (huge, highly redundant data) than SquashFS. SquashFS still has the big advantage of being part of the kernel, which makes it a lot more attractive for things like root file systems.
Re: DwarFS: A fast high compression read-only file system
#8What are the use cases for a read only file system?
Read-only media, for example. Or in general, stuff that doesn't really change. In my case: https://github.com/mhx/dwarfs#history
Re: DwarFS: A fast high compression read-only file system
#9> I started working on DwarFS in 2013 and my main use case and major motivation was that I had several hundred different versions of Perl that were taking up something around 30 gigabytes of disk space, and I was unwilling to spend more than 10% of my hard drive keeping them around for when I happened to need them.
It fills me with joy that someone has been coding a fs for 7 years due to perl installs taking too much space. Necessity is the mother of all invention.
Re: DwarFS: A fast high compression read-only file system
#10> I started working on DwarFS in 2013 and my main use case and major motivation was that I had several hundred different versions of Perl that were taking up something around 30 gigabytes of disk space, and I was unwilling to spend more than 10% of my hard drive keeping them around for when I happened to need them. It fills me with joy that someone has been coding a fs for 7 years due to perl installs taking too much…
Hahaha, I haven't actually been coding on this for that long, it's more that I coded for a few weeks back in 2013 and only found the motivation to resurrect the whole thing a few weeks back.