Does anyone remember back in the 90s when we'd install DoubleSpace to get on the fly compression? And then they built it into MSDOS 6 and that was a major game changer?
DwarFS: A fast high compression read-only file system
81–90 of 112 posts
Re: DwarFS: A fast high compression read-only file system
#82I noticed that enabling compression on zfs made a huge difference with the source size of some of my largely text file petitions. I never turned on deduplication because I don’t want to bother with the memory overhead, but I bet that would help even further.
Most ZFS howto's now recommend against dedup on the prolongued memory cost consequences. Yes, you would get some block level compression outcome. But, you enter the cost/benefit hell of balancing CPU and memory at runtime.
Re: DwarFS: A fast high compression read-only file system
#83Earlier quoted context omitted.
Hahaha, I haven't actually been coding on this for that long, it's more that I coded for a few weeks back in 2013 and only found the motivation to resurrect the whole thing a few weeks back.
> " taking up something around 30 gigabytes of disk space, and I was unwilling to spend more than 10% of my hard drive " I imagine these days you have more than 300GB hard disk space, making this all moot?
256GB SSDs are still everywhere.
Re: DwarFS: A fast high compression read-only file system
#84I wish there was a semi-compressed transparent filesystem layer which slowly compresses the least recently used files in the background, and un-compresses files upon use. That way you could store much more mostly unused content than space on the disk, without sacrificing accessibility.
I don't know about you guys, but most of the stuff that takes up space on my drives are: 1) Videos from my DSLR 2) RAW images from my DSLR 3) Various movies / TV series I downloaded 4) Game files (most of which are textures and 3D models) None of that stuff is really compressible.
The RAWs aren't compressible? Are they LZ encoded on the camera?
Re: DwarFS: A fast high compression read-only file system
#85Is this useful for long-term log storage? say, from a typical webapp (eg. Nginx logs, Rails logs, Postgres logs, etc)
Re: DwarFS: A fast high compression read-only file system
#86Does anyone remember back in the 90s when we'd install DoubleSpace to get on the fly compression? And then they built it into MSDOS 6 and that was a major game changer?
It was DoubleDrive until Microsoft licensed it and relabeled it as DoubleSpace. Stacker was the far more popular drive compression solution until MSDOS 6 was released.
Re: DwarFS: A fast high compression read-only file system
#87Earlier quoted context omitted.
I don't know about you guys, but most of the stuff that takes up space on my drives are: 1) Videos from my DSLR 2) RAW images from my DSLR 3) Various movies / TV series I downloaded 4) Game files (most of which are textures and 3D models) None of that stuff is really compressible.
The RAWs aren't compressible? Are they LZ encoded on the camera?
RAW imagery I have worked with is about 25-50% losslessly compressible on average. Most of your gains in image compression are from quantizing the gamut in clever, (usually) imperceptible ways.
Raw imagery contains a lot of entropy that usually doesn't affect the appearance perceptually, but still has sig figs that frustrate compression.
Re: DwarFS: A fast high compression read-only file system
#88It looks like the benefit is some kind of block or file deduplication. @OP: Can you please explain why you keep 50 gigs of perl around? :-) I use compressed read-only file systems all the time to save space on my travel laptop. I have one squashfs for firefox, one for the TeX base install, one for LLVM, one for qemu, one for my cross compiler collection. I suspect the gains over squashfs will be far less pronounced t…
Speculating here, but perl has a very rich test library and harnesses for running tests across multiple perls and platforms.
If you upload a module to CPAN, you automatically get it tested against a huge matrix of configurations:
http://matrix.cpantesters.org/?dist=Log-Any-Adapter-FileHand...
Re: DwarFS: A fast high compression read-only file system
#89Earlier quoted context omitted.
I don't know about you guys, but most of the stuff that takes up space on my drives are: 1) Videos from my DSLR 2) RAW images from my DSLR 3) Various movies / TV series I downloaded 4) Game files (most of which are textures and 3D models) None of that stuff is really compressible.
The RAWs aren't compressible? Are they LZ encoded on the camera?
I just tried compressing one with 7-zip ultra level compression. Saved maybe 5%. Wouldn't get even that with realtime compression.
Re: DwarFS: A fast high compression read-only file system
#90Earlier quoted context omitted.
I don't know about you guys, but most of the stuff that takes up space on my drives are: 1) Videos from my DSLR 2) RAW images from my DSLR 3) Various movies / TV series I downloaded 4) Game files (most of which are textures and 3D models) None of that stuff is really compressible.
The RAWs aren't compressible? Are they LZ encoded on the camera?
My Fuji has lossless RAW compression and as far as I know quite a few others too.