Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Elfshaker: Version control system fine-tuned for binaries

github.com

Post reply on HN
Print view View original

81–90 of 115 posts

Re: Elfshaker: Version control system fine-tuned for binaries

#81

Post by mal10c »

This project reminded me of something I've been looking for for a while - although it's not exactly what I'm looking for...

I use SolidWorks PDM at work to control drawings, BOMs, test procedures, etc. In all honesty, PDM does an alright job when it works, but when I have problems with our local server, all hell breaks loose and worst case, the engineers can't move forward.

In that light, I'd love to switch to another option. Preferably something decentralized just to ensure we have more backups. Git almost gets us there but doesn't include things like "where used."

All that being said, am I overlooking some features of Elfshaker that would fit well into my hopes of finding an alternative to PDM?

I also see there's another HN thread that asks the question I'm asking - just not through the lens of Elfshaker: https://news.ycombinator.com/item?id=20644770

Re: Elfshaker: Version control system fine-tuned for binaries

#82

Post by peterwaller-arm »

Tobu wrote: post #65
wlll wrote: post #2

Related, and impressive: https://github.com/elfshaker/manyclangs > manyclangs is a project enabling you to run any commit of clang within a few seconds, without having to build it. > It provides elfshaker pack files, each containing ~2000 builds of LLVM packed into ~100MiB. Running any particular build takes about 4s.

The clever idea that makes manyclangs compress well is to store object files before they are linked, with each function and each variable in its own elf section so that changes are mostly local; addresses will indirect through sections and a change to one item won't cascade into moving every address. I'm not sure the linking step they provide is deterministic/hermetic, if it is that would prove a decent way to compre…

Author here, I'd like to see such a comparison too actually, but I'm not in the position to do the work at the moment. We did some preliminary experiments at the beginning, but a lot changed over the course of the project and I don't know how well elfshaker fares ultimately against all the options out there. Some basic tests against git found that git is quite a bit slower (10s vs 100ms) during 'git add' and git checkout. Maybe that can be fixed with some tuning or finding appropriate options.

Re: Elfshaker: Version control system fine-tuned for binaries

#83

Post by zeotroph »

peterwaller-arm wrote: post #61
zeotroph wrote: post #57

Earlier quoted context omitted.

Thank you for the explanation, so the pre-link storage is one of the magical ingredients, maybe mention this as well in the README? Is this the reason why manyclang (using llvms cmake based build system) can be provided easily, but it would be more difficult for gcc? Or is the object -> binary dependency automatically deduced?

> maybe mention this as well in the README? We've tweaked the readme, I hope it's clearer. It would be great to provide this for gcc too. The project is new and we've just started out. I know less about gcc's build system and how hard it will be to apply these techniques there. It seems as though it should be possible though and I'd love to see it happen. To infer the object->executable dependencies we currently read…

Ah, the compilation database is where more magic originates from :)

Re: Elfshaker: Version control system fine-tuned for binaries

#84

Post by peterwaller-arm »

zeotroph wrote: post #83
peterwaller-arm wrote: post #61

Earlier quoted context omitted.

> maybe mention this as well in the README? We've tweaked the readme, I hope it's clearer. It would be great to provide this for gcc too. The project is new and we've just started out. I know less about gcc's build system and how hard it will be to apply these techniques there. It seems as though it should be possible though and I'd love to see it happen. To infer the object->executable dependencies we currently read…

Ah, the compilation database is where more magic originates from :)

Yes, this is less great than I would like! :( :)

Re: Elfshaker: Version control system fine-tuned for binaries

#85

Post by kvnhn »

mal10c wrote: post #81

This project reminded me of something I've been looking for for a while - although it's not exactly what I'm looking for... I use SolidWorks PDM at work to control drawings, BOMs, test procedures, etc. In all honesty, PDM does an alright job when it works, but when I have problems with our local server, all hell breaks loose and worst case, the engineers can't move forward. In that light, I'd love to switch to anothe…

Maybe not precisely what you want, but I built a CLI tool[1] that's like a simplified and decoupled Git-LFS. It tracks large files in a content-addressed directory, and then you track the references to that store in source control. Data compression isn't a top priority for my tool; it uses immutable symlinks, not archives.

[1]: https://github.com/kevin-hanselman/dud

Re: Elfshaker: Version control system fine-tuned for binaries

#86

Post by ghoul2 »

If I already have, lets say a 100MB pack file containing (say) 200 builds of clang and then I import the 201st build into that pack file - is it possible to send across a small delta of this new, updated pack file to someone else who already had the older pack file (with 200 builds) such that they can apply the delta to the old pack and get the new pack containing 201 builds?

Re: Elfshaker: Version control system fine-tuned for binaries

#87

Post by JoshTriplett »

peterwaller-arm wrote: post #52
JoshTriplett wrote: post #50

Earlier quoted context omitted.

Can you talk a bit more about what ELF-specific heuristics elfshaker uses? What kind of preprocessing do you do before zstd? Do you handle offsets changing in instructions, like the BCJ/BCJ2 filter? Do you do anything to detect insertions/deletions?

We've just added an applicability section, which explains a bit more what we do. We don't have any ELF specific heuristics [0]. https://github.com/elfshaker/elfshaker#applicability In summary, for manyclangs, we compile with -ffunction-sections and -fdata-sections, and store the resulting object files. These are fairly robust to insertions and deletions, since the addresses are section relative, so the damage of any…

Ah, I see! Makes sense that you can do much better if you get to compile the programs with your choice of options.

Re: Elfshaker: Version control system fine-tuned for binaries

#88

Post by henvic »

peterwaller-arm wrote: post #23
henvic wrote: post #13

Interesting. I wonder if this can also be [ab]used to, say, deliver deltas of programs, so that you can have faster updates, but maybe it doesn't make sense. https://en.wikipedia.org/wiki/Binary_delta_compression

Author here, I don't think it would apply well to that scenario. elfshaker is good for manyclangs where we ship 2,000 revisions in one file (pack), so the cost of individual revision is amortized. If one build of llvm+clang costs you some ~400 MiB; a single elfshaker pack containing 2,000 builds has an amortized cost of around 40kiB/build. But this amazing win is only happening because you are shipping 2,000 builds a…

Thank you for the insight!
Post reply on HN