91–100 of 115 posts
> There are many files, > Most of them don't change very often so there are a lot of duplicate files, > When they do change, the deltas of the [binaries] are not huge. We need this but for node_modules
Node_modulea is already tons and tons of files, and when they are large, they are usually minified and hard to split on any "natural" boundary (like elf sections/symbols etc)
I'd like to see a version of this built into things like IPFS. It seems obvious that whenever something is saved into IPFS, there might be a similar object already stored. If there is, go make a diff, and only store the diff.
[1] https://en.wikipedia.org/wiki/Rolling_hash#Content-based_sli...
Earlier quoted context omitted.
Same here. There is a usage guide, which helped a tiny bit: https://github.com/elfshaker/elfshaker/blob/main/docs/users/... Honestly, I sort of looked at it for conventional backup strategy...as in, i wonder if it could work as a replacement for tar-zipping up a directory, etc. But, not sure if the use cases is appropriate.
Author here. We'd love this to be a thing, but this is young software, so we don't recommend relying on this as a single way of doing a backup for now. Bear in mind that our main use case is for things that you can reproduce in principle (builds of a commit history, see manyclangs).
I appreciate your response, and thanks very much for the clarification of use case; very helpful! Thanks also of course for building this!
Earlier quoted context omitted.
For backup you probably want something like Borg to handle deduplication of identical content between backups.
Author here, I agree with xdfgh1112, please take care before using brand new software to store your backups!
I'd like to see a version of this built into things like IPFS. It seems obvious that whenever something is saved into IPFS, there might be a similar object already stored. If there is, go make a diff, and only store the diff.
It should be possible to do this in IPFS already if you use the go-ipfs --chunker option with a content-sensitive chunking algorithm like rabin or buzhash [1]. With this there's a good chance that a file with small changes from something already on IPFS will have some chunks that hash identically, so they'll be shared. [1] https://en.wikipedia.org/wiki/Rolling_hash#Content-based_sli...
Earlier quoted context omitted.
How does this work? Do all the game series use the same engine code and assets?
I think you're slightly misinterpreting what the parent said. Take the game Super Mario World for the console Super Nintendo. It was released in Japan. It was released in the US. It was released in Europe. It was released in Korea. It was released in Australia. It was probably released in various minor regions and given unique translations. There are almost certainly re-releases of the game on Super Nintendo that iss…
Sometimes, ROM-image-based game titles were based on the same "engine" (i.e. the same core set of assembler source-files with fixed address-space target locations, and so fixed locations in a generated ROM image), but with a few engine modifications, and entirely different assets.
In a sense, this makes these different games effectively into mutual "full conversion ROMhacks" of one-another.
You'll usually find these different game titles compressed together into the same ROMset (with one game title — usually the one with the oldest official release — being considered the prototype for the others, and so naming the ROMset), because they do compress together very well — not near-totally, the way bugfix patches do, but adding only the total amount to the archive size that you'd expect for the additional new assets.
Well-known examples of this are Doki Doki Panic vs. Super Mario Bros 2; Panel de Pon vs. Tetris Attack; Gradius III vs. Parodius; and any game with editions, e.g. Pokemon or Megaman Battle Network.
But there are more "complete" examples as well, where you'd never even suspect the two titles are related, with the games perhaps existing in entirely-different genres. (I don't have a ROMset library on-hand to dig out examples, but if you dig through one, you'll find some amazing examples of engine reuse.)
Earlier quoted context omitted.
It should be possible to do this in IPFS already if you use the go-ipfs --chunker option with a content-sensitive chunking algorithm like rabin or buzhash [1]. With this there's a good chance that a file with small changes from something already on IPFS will have some chunks that hash identically, so they'll be shared. [1] https://en.wikipedia.org/wiki/Rolling_hash#Content-based_sli...
But that isn't quite as good as something like this that can 'understand' diffs in files, rather than simply relying on the fact a bunch of bytes in a row might be the same.
Earlier quoted context omitted.
But that isn't quite as good as something like this that can 'understand' diffs in files, rather than simply relying on the fact a bunch of bytes in a row might be the same.
I don't think elfshaker actually does do any binary diffing (e.g. xdelta or bsdiff). It works well because it uses pre-link objects which are built to change as little as possible between versions. Then when it compresses similar files together in a pack, Zstandard can recognize the trivial repeats.
I find the description a bit confusing, is there and example where we can see the usage?
My top level being that it's a VCS (like Git) specialized for binaries; with commands baked in to prevent the slowdown that often comes with large git repositories.
I wonder if this concept could be extended to other binary types that git has problems with, were you able to know/control more about the underlying binary format.