Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Reproducible Git Bundles

baecher.dev

Post reply on HN
Print view View original

11–20 of 38 posts

Re: Reproducible Git Bundles

#11

Post by CharlesW »

charcircuit wrote: post #10
mikhailt wrote: post #6

Earlier quoted context omitted.

Mostly for security, data integrity, and QA purposes, it ensures the same data can be rebuilt at any given time. Let's say we have a recipe for a food that we really like and we want it to be the same all the time. We figured out the list of ingredients, we used it and it worked to our taste. Two months later, we decided we want that food again. We followed the same recipe with the same ingredients and it turns out t…

Reproducibility does not neccessarily require having perfectly matching bits. To me it seems like md5sum is not the best program to check if two bundles are equivalent. In this case going for bit for bit reproducibility does not seem to have much of a practical benefit.

> Reproducibility does not neccessarily require having perfectly matching bits.

https://en.wikipedia.org/wiki/Reproducible_builds: "Reproducible builds, also known as deterministic compilation, is a process of compiling software which ensures the resulting binary code can be reproduced. Source code compiled using deterministic compilation will always output the same binary."

Re: Reproducible Git Bundles

#12

Post by pbhn »

stepupmakeup wrote: post #8

Why not just use tar or any other archive tool on the repository .git folder? Unless your repository is a un-gc'd mess with millions of unpacked objects...

I think that is a fair alternative, but restoring the backup means that the repository is in a bit of a weird state. Whereas a bundle can be cloned from nicely. Your way does have the property that it includes hooks and config, though (which could be desired or not).

Re: Reproducible Git Bundles

#14

Post by raimue »

The question is whether you can expect this format to stay stable and reproducible across git versions. Remember the fallout from git 2.38 when the output of 'git archive' changed. Although for this backup use case it would just mean the next backup with a new format would make a full copy once.

Re: Reproducible Git Bundles

#15

Post by CuriousCosmic »

raimue wrote: post #14

The question is whether you can expect this format to stay stable and reproducible across git versions. Remember the fallout from git 2.38 when the output of 'git archive' changed. Although for this backup use case it would just mean the next backup with a new format would make a full copy once.

git bundles have a standardised format (defined in [1]) while git archives did not and continue to not have one. Git can still handle all previous bundle versions and you can specify which version of the bundle standard you want to use when creating bundles.

So no the git 2.38 issue should not be a problem for bundles even if the format changes in the future.

1. https://git-scm.com/docs/gitformat-bundle

Re: Reproducible Git Bundles

#16

Post by avar »

CuriousCosmic wrote: post #15
raimue wrote: post #14

The question is whether you can expect this format to stay stable and reproducible across git versions. Remember the fallout from git 2.38 when the output of 'git archive' changed. Although for this backup use case it would just mean the next backup with a new format would make a full copy once.

git bundles have a standardised format (defined in [1]) while git archives did not and continue to not have one. Git can still handle all previous bundle versions and you can specify which version of the bundle standard you want to use when creating bundles. So no the git 2.38 issue should not be a problem for bundles even if the format changes in the future. 1. https://git-scm.com/docs/gitformat-bundle

Standardized format != guaranteed reproducible. Git makes no promises that it'll keep PACK contents stable, just that they're guaranteed to "deflate" to the same contents.

Which is what the linked article discovered. Threading is a trivial way to discover this, but there's other ways PACK contents might differ across versions.

Re: Reproducible Git Bundles

#17

Post by JoshTriplett »

Parallelism doesn't inherently have to break reproducibility. If the output depends on the scheduling order of the threads, then it will. But it's possible to use threads to farm out work while still doing the same work deterministically.

Re: Reproducible Git Bundles

#18

Post by kzrdude »

raimue wrote: post #14

The question is whether you can expect this format to stay stable and reproducible across git versions. Remember the fallout from git 2.38 when the output of 'git archive' changed. Although for this backup use case it would just mean the next backup with a new format would make a full copy once.

Absolute stability is not needed for this use case. Just reproducible per git version would be enough, to ensure mostly no redundant backups.

Re: Reproducible Git Bundles

#19

Post by Ferret7446 »

pbhn wrote: post #12
stepupmakeup wrote: post #8

Why not just use tar or any other archive tool on the repository .git folder? Unless your repository is a un-gc'd mess with millions of unpacked objects...

I think that is a fair alternative, but restoring the backup means that the repository is in a bit of a weird state. Whereas a bundle can be cloned from nicely. Your way does have the property that it includes hooks and config, though (which could be desired or not).

There are very few/no operations that will put a git repository in a "weird" state.

Actually, any snapshot of a Git repository is consistent (due to its CAS nature), minus the index which doesn't really need backup anyway.

Re: Reproducible Git Bundles

#20

Post by nektro »

> The naive solution of simply backing up the entire file-system tree is clearly not desirable since that would clutter the backup with useless build artifacts.

`git rm -r --cached .`

> One solution is to create a fresh clone (with --mirror), but that will typically consist of many small files which isn't ideal for backups, either.

this is precisely what tar(1) was made for..

Post reply on HN