Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Elfshaker: Version control system fine-tuned for binaries

github.com

Post reply on HN
Print view View original

41–50 of 115 posts

Re: Elfshaker: Version control system fine-tuned for binaries

#41

Post by peterwaller-arm »

mrich wrote: post #34

I'm guessing this does not yield that high compression for release builds, where code can be optimized across translation units? Likewise when a commit changes a header that is included in many cpps?

Author here. The executables shipped in manyclangs are release builds! The catch is that manyclangs stores object files pre-link. Executables are materialized by relinking after they are extracted with elfshaker.

The stored object files are compiled with -ffunction-sections and -fdata-sections, which ensures that insertions/deletions to the object file only have a local effect (they don't cause relative addresses to change across the whole binary).

As you observe, anything which causes significant non-local changes in the data you store is going to have a negative effect when it comes to compression ratio. This is why we don't store the original executables directly.

Re: Elfshaker: Version control system fine-tuned for binaries

#44

Post by nh2 »

peterwaller-arm wrote: post #37
nh2 wrote: post #35

I experimented with something similar with a Linux distribution's package binary cache. Using `bup` (deduplicating backup tool using git packfile format) I deduplicated 4 Chromium builds into the size of 1. It could probably pack thousands into the size of a few. Large download/storage requirements for updates are one of NixOS's few drawbacks, and I think deduplication could solve that pretty much completely. Details…

Author here. I've used bup, and elfshaker was partially inspired by it! It's great. However, during initial experiments on this project I found bup to be slow, taking quite a long time to snapshot and extract. I think this could in principle be fixed in bup one day, perhaps.

I also use bup for a long time, but found that for very large server backups I'm hitting performance problems (both in time and memory usage).

I'm currently evaluating `bupstash` (also written in Rust) as a replacment. It's faster and uses a lot less memory, but is younger and thus lacks some features.

Here is somebody's benchmark of bupstas (unfortunately not including `bup`): https://acha.ninja/blog/encrypted_backup_shootout/

The `bupstash` author is super responsive on Gitter/Matrix, it may make sense to join there to discuss approaches/findings together.

I would really like to eventually have deduplication-as-a-library, to make it easier to put into programs like nix, or also other programs, e.g. for versioned "Save" functionality in software like Blender or Meshlab that work with huge files and for which diff-based incremental saving is more difficult/fragile to implement than deduplcating snapshot based saving.

Re: Elfshaker: Version control system fine-tuned for binaries

#45

Post by peterwaller-arm »

tttsxhub wrote: post #38

Why does it depend on the CPU architecture?

(Disclosure: I work for Arm, opinions are my own)

Author here. elfshaker itself does not have a dependency on any architecture to our knowledge. We support the architectures we have use of. Contributions to add missing support are welcome.

manyclangs provides binary pack files for aarch64 because that's what we have immediate use of. If elfshaker and manyclangs proves useful to people, I would love to see resource invested to make it more widely useful.

You can still run the manyclangs binaries on other architectures using qemu [0], with some performance cost, which may be tolerable depending on your use case.

[0] https://github.com/elfshaker/manyclangs/tree/main/docker-qem...

Re: Elfshaker: Version control system fine-tuned for binaries

#46

Post by 3np »

peterwaller-arm wrote: post #8
thristian wrote: post #5

This seems very much like the Git repository format, with loose objects being collected into compressed pack files - except I think Git has smarter heuristics about which files are likely to compress well together. It would be interesting to see a comparison between this tool and Git used to store the same collection of similar files.

An author here, I agree! The packfile format is heavily inspired by git, and git may also do quite well at this. We did some preliminary experiments with git a while back but found we were able to do the packing and extraction much faster and smaller than git was able to manage. However, we haven't had the time to repeat the experiments with our latest knowledge and the latest version of git. So it is entirely possib…

Do you think it would be feasible to do a git-lfs replacement based on elfshaker?

Down the line maybe it would even be possible to have binaries as “first-class” (save for diff I guess)

Re: Elfshaker: Version control system fine-tuned for binaries

#47

Post by shp0ngle »

peterwaller-arm wrote: post #20
bogwog wrote: post #15

Does this work well with image files? (PNG, JPEG, etc)

Author here, it works particularly well for our presented use case because it has these properties: * There are many files, * Most of them don't change very often, * When they do change, the deltas of the binaries are not huge. So, if the image files aren't changing very much, then it might work well for you. If the images are changing, their binary deltas would be quite large, so you'd get a compression ratio somewh…

Ahhh that’s the key insight I have been missing, and that should be higher somewhere.

Thanks

Re: Elfshaker: Version control system fine-tuned for binaries

#48

Post by peterwaller-arm »

IceWreck wrote: post #39
peterwaller-arm wrote: post #20

Earlier quoted context omitted.

Author here, it works particularly well for our presented use case because it has these properties: * There are many files, * Most of them don't change very often, * When they do change, the deltas of the binaries are not huge. So, if the image files aren't changing very much, then it might work well for you. If the images are changing, their binary deltas would be quite large, so you'd get a compression ratio somewh…

Please add these points under a usecase heading in your README.

Done, hopefully this is clearer. Please let us know if you see a way to improve it further: https://github.com/elfshaker/elfshaker/pull/60

Re: Elfshaker: Version control system fine-tuned for binaries

#49

Post by peterwaller-arm »

0942v8653 wrote: post #43

Does it do any architecture-specific processing, i.e. BCJ filter? Or is there a generic version of this? The performance seems quite good.

Author here. No architecture specific processing currently. Most of the magic happens in zstandard (hat tip to this amazing project).

Please see our new applicability section which explains the result in a bit more detail:

https://github.com/elfshaker/elfshaker/blob/1bedd4eacd3ddd83...

In manyclangs (which uses elfshaker for storage) we arrange that the object code has stable addresses when you do insertions/deletions, which means you don't need such a filter. But today I learned about such filters, so thanks for sharing your question!

Re: Elfshaker: Version control system fine-tuned for binaries

#50

Post by JoshTriplett »

peterwaller-arm wrote: post #27
johnyzee wrote: post #21

Earlier quoted context omitted.

As far as I was able to learn (don't remember the details, sorry), git does not do well with large binary files. I believe it ends up with a lot of duplication. It is the major thing I am missing from git, currently we store assets (like big PSDs that change often) outside of version control and it is suboptimal.

Performing poorly with non-textual data happens for a a number of reasons. Binary data, when changed, often have a lot of 'non-local' changes in them. For example, a PSD file might well have a compression algorithm already applied to it. An insertion/deletion is going to result in a very different compressed representation for which there is no good way to have an efficient delta. elfshaker will suffer the same probl…

Can you talk a bit more about what ELF-specific heuristics elfshaker uses? What kind of preprocessing do you do before zstd? Do you handle offsets changing in instructions, like the BCJ/BCJ2 filter? Do you do anything to detect insertions/deletions?
Post reply on HN