Elfshaker: Version control system fine-tuned for binaries
1–10 of 115 posts
Re: Elfshaker: Version control system fine-tuned for binaries
#2Related, and impressive: https://github.com/elfshaker/manyclangs
> manyclangs is a project enabling you to run any commit of clang within a few seconds, without having to build it.
> It provides elfshaker pack files, each containing ~2000 builds of LLVM packed into ~100MiB. Running any particular build takes about 4s.
Re: Elfshaker: Version control system fine-tuned for binaries
#3I find the description a bit confusing, is there and example where we can see the usage?
Re: Elfshaker: Version control system fine-tuned for binaries
#4Huh, interesting, could you maybe use this as an in-repo alternative to something like git-lfs?
Re: Elfshaker: Version control system fine-tuned for binaries
#5This seems very much like the Git repository format, with loose objects being collected into compressed pack files - except I think Git has smarter heuristics about which files are likely to compress well together. It would be interesting to see a comparison between this tool and Git used to store the same collection of similar files.
Re: Elfshaker: Version control system fine-tuned for binaries
#6I find the description a bit confusing, is there and example where we can see the usage?
Same here. There is a usage guide, which helped a tiny bit: https://github.com/elfshaker/elfshaker/blob/main/docs/users/...
Honestly, I sort of looked at it for conventional backup strategy...as in, i wonder if it could work as a replacement for tar-zipping up a directory, etc. But, not sure if the use cases is appropriate.
Re: Elfshaker: Version control system fine-tuned for binaries
#7This should be integrated with Cargo to reduce the size of the target directories which are becoming ridiculously large.
Re: Elfshaker: Version control system fine-tuned for binaries
#8This seems very much like the Git repository format, with loose objects being collected into compressed pack files - except I think Git has smarter heuristics about which files are likely to compress well together. It would be interesting to see a comparison between this tool and Git used to store the same collection of similar files.
An author here, I agree! The packfile format is heavily inspired by git, and git may also do quite well at this.
We did some preliminary experiments with git a while back but found we were able to do the packing and extraction much faster and smaller than git was able to manage. However, we haven't had the time to repeat the experiments with our latest knowledge and the latest version of git. So it is entirely possible that git might be an even better answer here in the end. We just haven't done the best experiments yet. It's something to bear in mind. If someone wants, they could measure this fairly easily by unpacking our snapshots and storing them into git.
On our machines, forming a snapshot of one llvm+clang build takes hundreds of milliseconds. Forming a packfile for 2,000 clang builds with elfshaker can take seconds during the pack phase with a 'low' compression level (a minute or two for the best compression level, which gets it down to the ~50-100MiB/mo range), and extracting takes less than a second. Initial experiments with git showed it was going to be much slower.
Re: Elfshaker: Version control system fine-tuned for binaries
#9I find the description a bit confusing, is there and example where we can see the usage?
My top level being that it's a VCS (like Git) specialized for binaries; with commands baked in to prevent the slowdown that often comes with large git repositories.
Re: Elfshaker: Version control system fine-tuned for binaries
#10I find the description a bit confusing, is there and example where we can see the usage?
There is an associated presentation on manyclangs at LLVM dev meeting. I think they presented yesterday?
Unfortunately it won't be uploaded until later but it will show up on the llvm YouTube channel: