Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Elfshaker: Version control system fine-tuned for binaries

github.com

Post reply on HN
Print view View original

21–30 of 115 posts

Re: Elfshaker: Version control system fine-tuned for binaries

#21

Post by johnyzee »

peterwaller-arm wrote: post #8
thristian wrote: post #5

This seems very much like the Git repository format, with loose objects being collected into compressed pack files - except I think Git has smarter heuristics about which files are likely to compress well together. It would be interesting to see a comparison between this tool and Git used to store the same collection of similar files.

An author here, I agree! The packfile format is heavily inspired by git, and git may also do quite well at this. We did some preliminary experiments with git a while back but found we were able to do the packing and extraction much faster and smaller than git was able to manage. However, we haven't had the time to repeat the experiments with our latest knowledge and the latest version of git. So it is entirely possib…

As far as I was able to learn (don't remember the details, sorry), git does not do well with large binary files. I believe it ends up with a lot of duplication. It is the major thing I am missing from git, currently we store assets (like big PSDs that change often) outside of version control and it is suboptimal.

Re: Elfshaker: Version control system fine-tuned for binaries

#23

Post by peterwaller-arm »

henvic wrote: post #13

Interesting. I wonder if this can also be [ab]used to, say, deliver deltas of programs, so that you can have faster updates, but maybe it doesn't make sense. https://en.wikipedia.org/wiki/Binary_delta_compression

Author here, I don't think it would apply well to that scenario. elfshaker is good for manyclangs where we ship 2,000 revisions in one file (pack), so the cost of individual revision is amortized. If one build of llvm+clang costs you some ~400 MiB; a single elfshaker pack containing 2,000 builds has an amortized cost of around 40kiB/build. But this amazing win is only happening because you are shipping 2,000 builds at once. If you wanted to ship a single delta, you can't compress against all the other builds.

Re: Elfshaker: Version control system fine-tuned for binaries

#24

Post by dilap »

peterwaller-arm wrote: post #19
dilap wrote: post #4

Huh, interesting, could you maybe use this as an in-repo alternative to something like git-lfs?

Author here, I don't currently know how this compares to git-lfs. It it is possible git-lfs would perform quite well on the same inputs as elfshaker works on. If git-lfs does already work well for your use case I'd recommend using that rather than elfshaker, as it is more established.

Thanks for the response! I was more just curious about future possibilities vs immediate practicle use.

git-lfs just offloads the storage of the large binaries to a remote site, and then downloads on demand.

If you have a lot of binary assets like artwork or huge excel spreadsheets, it's very useful, because in those cases, without git-lfs, the git repo will get very large, git will get extremely slow, and github will get angry at you for having too large a repo.

But it's not all roses with git-lfs, since now you're reliant on the external network to do checkouts, vs having fetched everything at once w/ the initial clone, and also of course just switching between revisions can get slower since you're network-limited to fetch those large files. (And though I'm not sure, it doesn't seem like git-lfs is doing any local caching.)

So you could imagine where something like having elfshaker embedded in the repo and integrated as a checkout filter could potentially be a useful alternative. Basically an efficient way to store binaries directly in the repo.

(Maybe it would be too small a band of use cases to be practicle though? Obviously if you have lots of distinct art assets, that's just going to be big, no matter what...)

Re: Elfshaker: Version control system fine-tuned for binaries

#25

Post by xdfgh1112 »

mxuribe wrote: post #6
carlmr wrote: post #3

I find the description a bit confusing, is there and example where we can see the usage?

Same here. There is a usage guide, which helped a tiny bit: https://github.com/elfshaker/elfshaker/blob/main/docs/users/... Honestly, I sort of looked at it for conventional backup strategy...as in, i wonder if it could work as a replacement for tar-zipping up a directory, etc. But, not sure if the use cases is appropriate.

For backup you probably want something like Borg to handle deduplication of identical content between backups.

Re: Elfshaker: Version control system fine-tuned for binaries

#27

Post by peterwaller-arm »

johnyzee wrote: post #21
peterwaller-arm wrote: post #8

Earlier quoted context omitted.

An author here, I agree! The packfile format is heavily inspired by git, and git may also do quite well at this. We did some preliminary experiments with git a while back but found we were able to do the packing and extraction much faster and smaller than git was able to manage. However, we haven't had the time to repeat the experiments with our latest knowledge and the latest version of git. So it is entirely possib…

As far as I was able to learn (don't remember the details, sorry), git does not do well with large binary files. I believe it ends up with a lot of duplication. It is the major thing I am missing from git, currently we store assets (like big PSDs that change often) outside of version control and it is suboptimal.

Performing poorly with non-textual data happens for a a number of reasons. Binary data, when changed, often have a lot of 'non-local' changes in them. For example, a PSD file might well have a compression algorithm already applied to it. An insertion/deletion is going to result in a very different compressed representation for which there is no good way to have an efficient delta. elfshaker will suffer the same problem here.

Re: Elfshaker: Version control system fine-tuned for binaries

#28

Post by peterwaller-arm »

i_like_waiting wrote: post #26

Thanks, seems like that could be good solution for storing of daily backups of DB. I didn't know I needed it but seems like I do.

Author here, this software is young, please don't use it for backups!

But also, in general, it might not work well for your use case, and our use case is niche. Please give it a try before making assumptions about any suitability for use.

Re: Elfshaker: Version control system fine-tuned for binaries

#30

Post by peterwaller-arm »

xdfgh1112 wrote: post #25
mxuribe wrote: post #6

Earlier quoted context omitted.

Same here. There is a usage guide, which helped a tiny bit: https://github.com/elfshaker/elfshaker/blob/main/docs/users/... Honestly, I sort of looked at it for conventional backup strategy...as in, i wonder if it could work as a replacement for tar-zipping up a directory, etc. But, not sure if the use cases is appropriate.

For backup you probably want something like Borg to handle deduplication of identical content between backups.

Author here, I agree with xdfgh1112, please take care before using brand new software to store your backups!
Post reply on HN