Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Elfshaker: Version control system fine-tuned for binaries

github.com

Post reply on HN
Print view View original

71–80 of 115 posts

Re: Elfshaker: Version control system fine-tuned for binaries

#71

Post by Daishiman »

derefr wrote: post #64
perth wrote: post #59

Earlier quoted context omitted.

Reminds me of how Microsoft packages the Windows installer actually. If you’ve ever unpacked Microsoft’s install.esd it’s interestingly insane how heavily it’s compressed. I assume it’s full of a lot of stuff that provides semi redundant binaries for compatibility to a lot of different systems, because the unpacked esd container goes from a few GiBs to I think around 40-50 iirc.

The emulation community also has "ROMsets" — collections of game ROM images, where the ROM images for a given game title are all grouped together into an archive. So you'd have one archive for e.g. "every release, dump, and ROMhack of Super Mario Bros 1." These ROM-set archives — especially when using more modern compression algorithms, like LZMA/7zip — end up about 1.1x the size of a single one of the contained game…

How does this work? Do all the game series use the same engine code and assets?

Re: Elfshaker: Version control system fine-tuned for binaries

#73

Post by kristjansson »

goodpoint wrote: post #58

I'm surprised nobody mentioned git-annex. It does the same using git for metadata. It's extremely efficient.

AFAIK, git-annex doesn't address address sub-file deduplication/compression at all, it just stores a new copy for each new hash it sees? I suppose that content-addressed storage, combined with the pre-link strategy discussed elsewhere for the related manyclangs project would produce similar, if less spectacular, results?

Re: Elfshaker: Version control system fine-tuned for binaries

#74

Post by bena »

Daishiman wrote: post #71
derefr wrote: post #64

Earlier quoted context omitted.

The emulation community also has "ROMsets" — collections of game ROM images, where the ROM images for a given game title are all grouped together into an archive. So you'd have one archive for e.g. "every release, dump, and ROMhack of Super Mario Bros 1." These ROM-set archives — especially when using more modern compression algorithms, like LZMA/7zip — end up about 1.1x the size of a single one of the contained game…

How does this work? Do all the game series use the same engine code and assets?

Sort of. ROMHacks are modified ROM images of a certain game.

If you knew where in the ROM image the level data was contained, you could modify it. As long as you didn't violate any constraints, the game would run fine.

You could also potentially influence game behavior as well.

The Game Genie and Gameshark were kind based on this concept. Except, being further along the chain, it could write values coming into and out of memory, so other effects were possible.

So, in the case of Super Mario Bros. ROMHacks, they all use Super Mario Bros. as a base ROM. Then from there, all you need to do is store the diff from the base.

Re: Elfshaker: Version control system fine-tuned for binaries

#75

Post by mhx77 »

Somewhat related (and definitely born out of a very similar use case): https://github.com/mhx/dwarfs

I initially built this for having access to 1000+ Perl installations (spanning decades of Perl releases). The compression in this case is not quite as impressive (50 GiB to around 300 MiB), but access times are typically in the millisecond region.

Re: Elfshaker: Version control system fine-tuned for binaries

#76

Post by notafraudster »

Daishiman wrote: post #71
derefr wrote: post #64

Earlier quoted context omitted.

The emulation community also has "ROMsets" — collections of game ROM images, where the ROM images for a given game title are all grouped together into an archive. So you'd have one archive for e.g. "every release, dump, and ROMhack of Super Mario Bros 1." These ROM-set archives — especially when using more modern compression algorithms, like LZMA/7zip — end up about 1.1x the size of a single one of the contained game…

How does this work? Do all the game series use the same engine code and assets?

I think you're slightly misinterpreting what the parent said. Take the game Super Mario World for the console Super Nintendo. It was released in Japan. It was released in the US. It was released in Europe. It was released in Korea. It was released in Australia. It was probably released in various minor regions and given unique translations. There are almost certainly re-releases of the game on Super Nintendo that issued new ROM files to correct minor bugs. Maybe there's a Greatest Hits version which might be the same game, but with an updated copyright date to reflect the re-release. This might amount to 10-12 versions of the same game, but 99.99% of what's in the ROM file is the same across all of them, so they can be represented compressed very well.

A copy of Super Mario Advance 2 for Game Boy Advance, which is also a re-release of Super Mario World, almost surely uses its own engine and would not be part of the same rom set. Likewise, other Mario games (like Mario 64, Super Mario Bros, etc.) would not be part of the same rom set. So it's nothing about the series using the same engine code or assets.

We're talking bugfixes and different regions for the same game on the same console. But this still has the effect of dropping the size for complete console collections by 50% or more, because most consoles have 2-3 regions per game for most games.

Re: Elfshaker: Version control system fine-tuned for binaries

#77

Post by Siira »

peterwaller-arm wrote: post #37
nh2 wrote: post #35

I experimented with something similar with a Linux distribution's package binary cache. Using `bup` (deduplicating backup tool using git packfile format) I deduplicated 4 Chromium builds into the size of 1. It could probably pack thousands into the size of a few. Large download/storage requirements for updates are one of NixOS's few drawbacks, and I think deduplication could solve that pretty much completely. Details…

Author here. I've used bup, and elfshaker was partially inspired by it! It's great. However, during initial experiments on this project I found bup to be slow, taking quite a long time to snapshot and extract. I think this could in principle be fixed in bup one day, perhaps.

Is elfshaker any good for backuping non-text data?

Re: Elfshaker: Version control system fine-tuned for binaries

#78

Post by ybkshaw »

peterwaller-arm wrote: post #37
nh2 wrote: post #35

I experimented with something similar with a Linux distribution's package binary cache. Using `bup` (deduplicating backup tool using git packfile format) I deduplicated 4 Chromium builds into the size of 1. It could probably pack thousands into the size of a few. Large download/storage requirements for updates are one of NixOS's few drawbacks, and I think deduplication could solve that pretty much completely. Details…

Author here. I've used bup, and elfshaker was partially inspired by it! It's great. However, during initial experiments on this project I found bup to be slow, taking quite a long time to snapshot and extract. I think this could in principle be fixed in bup one day, perhaps.

Thank you for having such a good description on the project! Sometimes the links from HN lead to a page that takes a few minutes of puzzling to figure out what is going on but not yours.

Re: Elfshaker: Version control system fine-tuned for binaries

#79

Post by peterwaller-arm »

mhx77 wrote: post #75

Somewhat related (and definitely born out of a very similar use case): https://github.com/mhx/dwarfs I initially built this for having access to 1000+ Perl installations (spanning decades of Perl releases). The compression in this case is not quite as impressive (50 GiB to around 300 MiB), but access times are typically in the millisecond region.

Nice, I bet dwarfs would do well at our use case too. Thanks for sharing.

Re: Elfshaker: Version control system fine-tuned for binaries

#80

Post by peterwaller-arm »

necovek wrote: post #70
peterwaller-arm wrote: post #23

Earlier quoted context omitted.

Author here, I don't think it would apply well to that scenario. elfshaker is good for manyclangs where we ship 2,000 revisions in one file (pack), so the cost of individual revision is amortized. If one build of llvm+clang costs you some ~400 MiB; a single elfshaker pack containing 2,000 builds has an amortized cost of around 40kiB/build. But this amazing win is only happening because you are shipping 2,000 builds a…

How fast would it be to get a delta between any two of the 2,000 builds in a single elfshaker pack? If that's reasonably fast, perhaps an approach like that could work: server stores the entire pack, but upon user request extracts a delta between user's version and target binary. Still, the devil is in the details of building all revisions of all software a single distribution has.

Yes you could do that. On the other hand, all revisions for a month is 100MiB, and all revisions we've built spanning 2019-now are a total of 2.8GiB, so we opted to forego implementing any object negotiation and just say 'you have to download the 100MiB for the month to access it'. I think you could a push/pull protocol could be implemented, but at that point probably git might do a reasonable job in that case :)
Post reply on HN