Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Elfshaker: Version control system fine-tuned for binaries

github.com

Post reply on HN
Print view View original

101–110 of 115 posts

Re: Elfshaker: Version control system fine-tuned for binaries

#101

Post by pdimitar »

nh2 wrote: post #44
peterwaller-arm wrote: post #37

Earlier quoted context omitted.

Author here. I've used bup, and elfshaker was partially inspired by it! It's great. However, during initial experiments on this project I found bup to be slow, taking quite a long time to snapshot and extract. I think this could in principle be fixed in bup one day, perhaps.

I also use bup for a long time, but found that for very large server backups I'm hitting performance problems (both in time and memory usage). I'm currently evaluating `bupstash` (also written in Rust) as a replacment. It's faster and uses a lot less memory, but is younger and thus lacks some features. Here is somebody's benchmark of bupstas (unfortunately not including `bup`): https://acha.ninja/blog/encrypted_backu…

I used `bupstash` and evaluated it for a while. I am looking to do 5+ offsite backups of a small personal directory to services that offer 5GB of cloud space for free.

`bupstash` lacked good compression. I settled with `borg` because I could use `zstd` compression with it. Currently at 60 snapshots of the directory and the `borg` repo directory is at ~1.52GB out of 5GB quota. The source directory is ~12.19GB uncompressed. Very happy with `borg` + `zstd` and how they handle my scenario.

I liked `bupstash` a lot, and the author is responsive and friendly. But I won't be giving it another try until it implements much more aggressive compression compared to what it can do now. It's a shame, I really wanted to use it.

I do recognize that for many other scenarios `bupstash` is very solid though.

Re: Elfshaker: Version control system fine-tuned for binaries

#102

Post by pdimitar »

mhx77 wrote: post #75

Somewhat related (and definitely born out of a very similar use case): https://github.com/mhx/dwarfs I initially built this for having access to 1000+ Perl installations (spanning decades of Perl releases). The compression in this case is not quite as impressive (50 GiB to around 300 MiB), but access times are typically in the millisecond region.

That's super impressive, I will definitely give it a go. Thanks for sharing!

Re: Elfshaker: Version control system fine-tuned for binaries

#103

Post by tedunangst »

henvic wrote: post #13

Interesting. I wonder if this can also be [ab]used to, say, deliver deltas of programs, so that you can have faster updates, but maybe it doesn't make sense. https://en.wikipedia.org/wiki/Binary_delta_compression

This is how openbsd binary patches for the kernel work. The changed object files are shipped; the end system relinks a new kernel.

Re: Elfshaker: Version control system fine-tuned for binaries

#105

Post by lathiat »

pdimitar wrote: post #101
nh2 wrote: post #44

Earlier quoted context omitted.

I also use bup for a long time, but found that for very large server backups I'm hitting performance problems (both in time and memory usage). I'm currently evaluating `bupstash` (also written in Rust) as a replacment. It's faster and uses a lot less memory, but is younger and thus lacks some features. Here is somebody's benchmark of bupstas (unfortunately not including `bup`): https://acha.ninja/blog/encrypted_backu…

I used `bupstash` and evaluated it for a while. I am looking to do 5+ offsite backups of a small personal directory to services that offer 5GB of cloud space for free. `bupstash` lacked good compression. I settled with `borg` because I could use `zstd` compression with it. Currently at 60 snapshots of the directory and the `borg` repo directory is at ~1.52GB out of 5GB quota. The source directory is ~12.19GB uncompre…

Borg has been working great for me with zstd.

Re: Elfshaker: Version control system fine-tuned for binaries

#107

Post by glandium »

peterwaller-arm wrote: post #82
Tobu wrote: post #65

Earlier quoted context omitted.

The clever idea that makes manyclangs compress well is to store object files before they are linked, with each function and each variable in its own elf section so that changes are mostly local; addresses will indirect through sections and a change to one item won't cascade into moving every address. I'm not sure the linking step they provide is deterministic/hermetic, if it is that would prove a decent way to compre…

Author here, I'd like to see such a comparison too actually, but I'm not in the position to do the work at the moment. We did some preliminary experiments at the beginning, but a lot changed over the course of the project and I don't know how well elfshaker fares ultimately against all the options out there. Some basic tests against git found that git is quite a bit slower (10s vs 100ms) during 'git add' and git chec…

It would be interesting to compare to gitoxide tweaked to use zstd compression for packs.

Re: Elfshaker: Version control system fine-tuned for binaries

#108

Post by josephg »

derefr wrote: post #68
peterwaller-arm wrote: post #27

Earlier quoted context omitted.

Performing poorly with non-textual data happens for a a number of reasons. Binary data, when changed, often have a lot of 'non-local' changes in them. For example, a PSD file might well have a compression algorithm already applied to it. An insertion/deletion is going to result in a very different compressed representation for which there is no good way to have an efficient delta. elfshaker will suffer the same probl…

One could, in theory, write a git-clean filter (like the one used for git-lfs), that teaches git various heuristic approaches to "take apart" well-known binary container formats into trees of binary object leaf-nodes. Then, when you committed a large binary that git could understand, what git would really be committing in its place would be a directory tree — sort of like the "resource tree" you see if you edit an MK…

As I understand it, this is essentially what the Google chrome updater does. It disassembles the binary and recalculates jump labels. Then it generates a diff based on the assembly code. When it applies that diff on people’s computers, the your computer again pulls the chrome binary apart and reconstructs it. The code for this is complex, but it’s all opensource.

I remember reading about this technique years ago, thinking “cool when this catches on, software updates in all my software will be tiny”. But no, for some reason macos updates are still gigabytes in size. I have no idea why?

Re: Elfshaker: Version control system fine-tuned for binaries

#109

Post by hcs »

josephg wrote: post #108
derefr wrote: post #68

Earlier quoted context omitted.

One could, in theory, write a git-clean filter (like the one used for git-lfs), that teaches git various heuristic approaches to "take apart" well-known binary container formats into trees of binary object leaf-nodes. Then, when you committed a large binary that git could understand, what git would really be committing in its place would be a directory tree — sort of like the "resource tree" you see if you edit an MK…

As I understand it, this is essentially what the Google chrome updater does. It disassembles the binary and recalculates jump labels. Then it generates a diff based on the assembly code. When it applies that diff on people’s computers, the your computer again pulls the chrome binary apart and reconstructs it. The code for this is complex, but it’s all opensource. I remember reading about this technique years ago, thi…

I think you're referring to courgette, which does the disassemble + patch + reassemble thing. Fwiw, Chrome now uses a simpler but competetive format called zucchini, see my recent comment: https://news.ycombinator.com/item?id=29028534

Re: Elfshaker: Version control system fine-tuned for binaries

#110

Post by aabbcc1241 »

lxe wrote: post #90

> There are many files, > Most of them don't change very often so there are a lot of duplicate files, > When they do change, the deltas of the [binaries] are not huge. We need this but for node_modules

checkout pnpm, it stores each version of package only once, and setup your project's node_modules with symbolic to the exact cached version
Post reply on HN