31–40 of 115 posts
Related, and impressive: https://github.com/elfshaker/manyclangs > manyclangs is a project enabling you to run any commit of clang within a few seconds, without having to build it. > It provides elfshaker pack files, each containing ~2000 builds of LLVM packed into ~100MiB. Running any particular build takes about 4s.
Thanks, seems like that could be good solution for storing of daily backups of DB. I didn't know I needed it but seems like I do.
Author here, this software is young, please don't use it for backups! But also, in general, it might not work well for your use case, and our use case is niche. Please give it a try before making assumptions about any suitability for use.
Earlier quoted context omitted.
An author here, I agree! The packfile format is heavily inspired by git, and git may also do quite well at this. We did some preliminary experiments with git a while back but found we were able to do the packing and extraction much faster and smaller than git was able to manage. However, we haven't had the time to repeat the experiments with our latest knowledge and the latest version of git. So it is entirely possib…
As far as I was able to learn (don't remember the details, sorry), git does not do well with large binary files. I believe it ends up with a lot of duplication. It is the major thing I am missing from git, currently we store assets (like big PSDs that change often) outside of version control and it is suboptimal.
Perforce is still used by game developers and other creatives, because it handles large binaries, quite well.
In fact, I'm not sure if they still do it, but one of the game engines (I think, maybe, Unreal) used to have a free tier that also included a free Perforce install.
Using `bup` (deduplicating backup tool using git packfile format) I deduplicated 4 Chromium builds into the size of 1. It could probably pack thousands into the size of a few.
Large download/storage requirements for updates are one of NixOS's few drawbacks, and I think deduplication could solve that pretty much completely.
Does it make a sense to turn it into fuse fs, with transparent deduplication?
Author here. Maybe, it's a fun idea. I have toyed with providing a fuse filesystem for access to a pack but my time for completing this is limited at the moment.
It might make sense to check how they do it.
I'd also be interested in how elfshaker compares to those (and `bupstash`, which is written in Rust but doesn't have a FUSE mount yet) in terms of compression and speed.
Did you know of their existence when making elfshaker?
Edit: Question also posted in your Q&A: https://github.com/elfshaker/elfshaker/discussions/58#discus...
I experimented with something similar with a Linux distribution's package binary cache. Using `bup` (deduplicating backup tool using git packfile format) I deduplicated 4 Chromium builds into the size of 1. It could probably pack thousands into the size of a few. Large download/storage requirements for updates are one of NixOS's few drawbacks, and I think deduplication could solve that pretty much completely. Details…
Does this work well with image files? (PNG, JPEG, etc)
Author here, it works particularly well for our presented use case because it has these properties: * There are many files, * Most of them don't change very often, * When they do change, the deltas of the binaries are not huge. So, if the image files aren't changing very much, then it might work well for you. If the images are changing, their binary deltas would be quite large, so you'd get a compression ratio somewh…
Thanks, seems like that could be good solution for storing of daily backups of DB. I didn't know I needed it but seems like I do.