Compression - anyone using lrzip on production servers?
DwarFS: A fast high compression read-only file system
61–70 of 112 posts
Re: DwarFS: A fast high compression read-only file system
#62Neat! I'd like to see benchmarks for more typical squashfs payloads-- embedded root filesystems totalling under 100MB. Small docker images like alpine would be a decent proxy. The given corpus of thousands of perl versions is more appropriate for comparison against git.
Author here :) I'll add more benchmarks, this is still WIP and so far I've mainly tried to satisfy my own needs. My intention with DwarFS wasn't to write "a better SquashFS", but to make it better in certain scenarios (huge, highly redundant data) than SquashFS. SquashFS still has the big advantage of being part of the kernel, which makes it a lot more attractive for things like root file systems.
How much does the compression of the perl repo become when compressed with
lrzip -UL9 filetarball.tar
It would be a good data point for everyone.
Re: DwarFS: A fast high compression read-only file system
#63I noticed that enabling compression on zfs made a huge difference with the source size of some of my largely text file petitions. I never turned on deduplication because I don’t want to bother with the memory overhead, but I bet that would help even further.
Re: DwarFS: A fast high compression read-only file system
#64Earlier quoted context omitted.
Are you doing your own caching in userspace, or are you working with the kernel's caching? The latter would substantially reduce memory requirements.
If you're talking about the kernel's filesystem cache, wouldn't that cache the compressed files? As far as I understand it userspace caching is necessary to cache uncompressed blocks, since the decompression is (presumably) done in userspace. I definitely could be wrong though, let me know if you're talking about a different kind of kernel caching. Actually, I guess if DwarFS is a kernel module and decompresses block…
It's using FUSE, and FUSE filesystems still participate in some parts of kernel caching.
Re: DwarFS: A fast high compression read-only file system
#65Earlier quoted context omitted.
Are you doing your own caching in userspace, or are you working with the kernel's caching? The latter would substantially reduce memory requirements.
Files that you've accessed will be kept in the kernel's cache. The cache I was talking about is a cache for decompressed blocks. Single files can stretch across multiple blocks, so you need to be able to keep more than one in memory anyway. However, decompressed files are kept in the cache in the hope that further (or even concurrent) reads will access the same blocks. Taking the example from the README where over a…
Ah, I see. So this specifically saves the decompression time for data you've already decompressed, if another file references the same data?
Re: DwarFS: A fast high compression read-only file system
#66Earlier quoted context omitted.
This is a read-only file system, so it’s able to exploit certain properties of that—- locating similar files next to each other, for example.
I don’t see how read only helps at all. Btrfs can dedupe at the block level.
A compression benchmark of both filesystems would be of interest in this regard (lzo, zstd and zlib), both read speed and compression wise
Re: DwarFS: A fast high compression read-only file system
#67I have several highly-redundant NTFS backups that I'd like to compress into a read-only fs. Can DwarFS preserve all NTFS metadata?
I think it uses FUSE, which is linux specific.
Re: DwarFS: A fast high compression read-only file system
#68Perhaps not strictly on-topic, but is there any equivalent FS/program in Windows that will allow users to have read-only access to files that are deduplicated in some way?
My use case is the MAME console archives, which are now full of copies of games from different localisations with 99% identical content. 7Z will compress them together and deduplicate, but breaks once the archive exceeds a few gigs.
These archives are already compressed (CHD format, which is 7Z + FLAC for ISOs), but it's deduplication that needs to happen on top of these already compressed files that I'm struggling with.
Sorry for the off-topic ask!
Re: DwarFS: A fast high compression read-only file system
#69This could be awesome for compressing Docker image layers. After all, they can be huge (hundreds of MB) and, if the Dockerfile is well organized, each step should contain a fairly homogeneous set of files (like apt-get artifacts, for example).
Re: DwarFS: A fast high compression read-only file system
#70> I started working on DwarFS in 2013 and my main use case and major motivation was that I had several hundred different versions of Perl that were taking up something around 30 gigabytes of disk space, and I was unwilling to spend more than 10% of my hard drive keeping them around for when I happened to need them. It fills me with joy that someone has been coding a fs for 7 years due to perl installs taking too much…
I have about the very same problems as mhx, several hundreds of huge perl versions which are almost the same, taking up enourmous amounts of diskspace. E.g. I had to move most of them from my SSD to a spinning disk. I really want to move them back. Thanks to mhx I can move them now back to my fast disk. This is also perfect for testers.
If they're almost the same, could you use one git repo with different branches for each version? Or archive them with restic into a folder and restore which one you need each time. Either method should deduplicate data if they're mostly the same file structure and content.
Edit: You could even have several read only shadow copies of the repo for parallel working directory usage, if your hard link the .git directory except for the HEAD ref in each.