Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Git archive checksums may change

github.blog

Post reply on HN
Print view View original

231–240 of 255 posts

Re: Git archive checksums may change

#231

Post by kzrdude »

kentonv wrote: post #25
acdha wrote: post #15

Earlier quoted context omitted.

If those tools incorrectly assume an API contract which doesn't exist, isn't the right answer to fix those tools?

In theory, sure, that's what we'd do in an ideal world. In the real world it will take millions of dollars of eng labor just to update the hashes to fix everything that's currently broken and millions more to actually implement something better and move everyone over to it. This isn't worth it, GitHub needs to just revert the change and then engineer a way to keep hashes stable going forward.

I know it's superficial but I think the problem would have been reduced if they used a download URL that looked like github.com/archive.php?project=rust&version=deadbeef it's just something that sends a signal and a different expectation on the same artifact.

Re: Git archive checksums may change

#232

Post by modderation »

dharmab wrote: post #55
phphphphp wrote: post #32

Earlier quoted context omitted.

Consumers often mistake hasn’t changed for a commitment to never change: any sufficiently large product will be littered with these kind of implicit commitments made by the product to consumers that nobody has visibility into. You’re unfortunate that we were all relying on this commitment you’ve never made, but the quick reversion is the best we can hope for. People will theorise how this could have been avoided but…

Hyrum's Law: With a sufficient number of users of an API, it does not matter what you promise in the contract: all observable behaviors of your system will be depended on by somebody.

Incidentally, I'm still waiting for GitHub to implement Spacebar Heating: https://m.xkcd.com/1172/

Re: Git archive checksums may change

#233

Post by JonChesterfield »

missingdays wrote: post #204
ZephyrBlu wrote: post #130

Earlier quoted context omitted.

You just described >90% of users. Everyone does this for something, most people do it for most things. You minimally read the docs, get something working and then leave it alone. Of course you're going to be pissed off when an implicit assumption which has been stable for a long time is broken.

Yes, but if you implement the checksum algorithm for GitHub archives, shouldn't you read the documentation about archives checksum?

Turns out scripts contain download an archive from github and check against a hardcoded checksum copy&pasted into that script. All of those broke. None of the authors will have looked up exactly how github had calculated said checksum.

Re: Git archive checksums may change

#234

Post by JonChesterfield »

reindeerer wrote: post #167
bentley wrote: post #136

Earlier quoted context omitted.

We’ve regressed from the previous norm of open source projects providing stable source tarballs with fixed checksums, sometimes even with cryptographic signatures.

That norm still exists, and it's offered by Github in form of Github Releases feature as well. It's the downstream tooling ( i.e. all the builds and package managers ) that need to clean their act up.

If the source tar changes, how do you propose the downstream tooling distinguishes between data corruption, MITM attack and upstream deciding to change the number without notifying anyone?

Re: Git archive checksums may change

#235

Post by Zababa »

nick__m wrote: post #9
frankjr wrote: post #5

GitHub will need to revert this change. They've just crippled pretty much every "from source" package manager out there.

I prefer that tool be adapted to be more resilient and not depend on github particular implementation.

They're all waiting for your pull requests.

Re: Git archive checksums may change

#236

Post by JonChesterfield »

ecnahc515 wrote: post #159
Denvercoder9 wrote: post #135

Earlier quoted context omitted.

You could also say that some maintainers made that decision for their convenience of not having to build and upload source archives. It is possible to upload your own artifacts to a release on GitHub, and lots of projects do. Those are correctly treated as immutable by GitHub.

I imagine GitHub would be equally upset if every project every started uploading their source tarballs themselves, as presumably the point of their source tarball feature was not needing to store the source tarball for every repository ever, but generating it on demand.

Bit of a shame they've just provided every project with huge incentive to upload their own source tarballs then. Coming soon, complaints that users upload too many opaque binaries.

Re: Git archive checksums may change

#237

Post by c4mpute »

nilsbunger wrote: post #215
c4mpute wrote: post #91

Earlier quoted context omitted.

Yes, but not in this bug. I guess lots of people missed that distinction: The stable git SHA hash is the commit hash, which is an hash over gits internal representation of the commit object (containing a tree of all file hashes, and parents' hashes). The hash that pops out of 'git archive' has nothing whatsoever to do with the commit hash and was historically stable more or less by accident: git feeds all files to 't…

Oh interesting. But if an archive hash isn’t stable, how is it meant to be used? What’s it good for?

In git, there is no intended use for it.

That people use it comes from how releases were usually published (independent of any version control system) as tgz/zip archives on some project website or ftp server. Websites and ftp servers were often mirrored to e.g. ISP or university mirrors because bandwith was scarce and CDNs were expensive/absent. To make sure that your release download from the university of somestrangeplace ftp matches the official release, you would compare the archive hash from the official project website with the hash of the archive you downloaded (bonus points for a GPG signature on the archive).

This then got automated by build/install/package tools to check the package downloaded from some mirror against the hash from the package description. Then GitHub happened, where GitHub replaced the mirror servers, serving autogenerated 'git archive' output instead of static files. And thats where things went wrong here...

Re: Git archive checksums may change

#238

Post by Aissen »

It was publicly known that Github was breaking automatic git archives consistency for many years. Here is a bug on a project to stop relying on fake github archives (as opposed to stable git-archive(1)):

https://bugzilla.tianocore.org/show_bug.cgi?id=3099

At some point it was impossible to go a few weeks (or even days) without a github archive change (depending on which part of the "CDN" you hit), I guess they must have stabilized it at some point. Here is an old issue before GitHub had a community issue tracker:

https://github.com/isaacs/github/issues/1483

I am glad this is getting more attention, maybe now github will finally have a stable endpoint for archives.

Re: Git archive checksums may change

#239

Post by rsc »

matthewcroughan wrote: post #230
rsc wrote: post #123

Earlier quoted context omitted.

Thanks for the quick rollback. I want to encourage you to think about locking in the current archive details, at least for archives that have already been served. Verifying that downloaded archives have the expected checksum is a critical best practice for software supply chain security. Training people to ignore checksum changes is training them to ignore attacks. GitHub is a strong leader in other parts of supply c…

I've just had a thought. When GitHub do update the hashing for better compression, everyone relying on the tar hash will update their hashes. This is the ultimate opportunity to change the tar contents, effect the supply chain, introduce vulnerabilities, and have everyone trust you. Something like Nix which computes the NAR Hash (the result of the tar contents) will not be effected by this, since it only cares about…

You're not wrong, but you're also not being realistic.

Nix is not the only system that takes this approach. The Go modules "directory hash" is roughly equivalent, although we defined it in terms of somewhat more standard tooling: it is the output of

    sha256sum $(find . -type f | sort) | sha256sum
I am not here advocating that everyone switch to this basic directory hash either, because it's not a solution to the more general problem that many systems are solving, namely validating _any_ downloaded file, not just file archives.

There are widespread, standard tools to run a SHA256 over a downloaded file, and those tools work on _any_ downloaded file. Essentially every programming language ships with or has easily accessible libraries to do the same. In contrast, there are not widespread, standard tools or libraries for the "NAR Hash" nor the Go "directory hash". Even if there were, such tools would need to be able to parse every kind of file that people might be downloading as part of a build, not just tar files.

It's a good solution in limited cases such as Nix and Go modules, but it's not the right end-to-end solution for all cases.

Re: Git archive checksums may change

#240

Post by matthewcroughan »

rsc wrote: post #239
matthewcroughan wrote: post #230

Earlier quoted context omitted.

I've just had a thought. When GitHub do update the hashing for better compression, everyone relying on the tar hash will update their hashes. This is the ultimate opportunity to change the tar contents, effect the supply chain, introduce vulnerabilities, and have everyone trust you. Something like Nix which computes the NAR Hash (the result of the tar contents) will not be effected by this, since it only cares about…

You're not wrong, but you're also not being realistic. Nix is not the only system that takes this approach. The Go modules "directory hash" is roughly equivalent, although we defined it in terms of somewhat more standard tooling: it is the output of sha256sum $(find . -type f | sort) | sha256sum I am not here advocating that everyone switch to this basic directory hash either, because it's not a solution to the more…

When you say it is not the right end-to-end solution for all cases, I am wondering what case you have in mind that a NAR Hash would not be suitable for.

If you adopt Nix fully, the .narinfo file that cache.nixos.org (a Nix substituted) serves that is signed, contains both the NAR Hash and the hash of the NAR Archive File as well. Additionally, NAR packs and unpacks deterministically, and you can read the implementation in the Nix thesis.

A .narinfo file looks like this:

```

StorePath: /nix/store/xvp2wr01fi27j0ycxqmdg6q4frsiv82s-libnotify-0.8.1 URL: nar/0a4jjqxwjcnnaia76l64drq9bjw7jczgmrirzshgp0bnw621f1c9.nar.xz Compression: xz FileHash: sha256:0a4jjqxwjcnnaia76l64drq9bjw7jczgmrirzshgp0bnw621f1c9 FileSize: 24324 NarHash: sha256:02bh3qjxgph5g9di3q553k87w4kbc4drmflkfz9knqbp9jip98c5 NarSize: 101776 References: 7ncncvnr864iangwbvbgbanx1r6wpf79-gdk-pixbuf-2.42.10 i4dqcpppyyq5yqcvw95mv5s11yfyy8pf-glib-2.74.3 xvp2wr01fi27j0ycxqmdg6q4frsiv82s-libnotify-0.8.1 yzjgl0h6a3qh1mby405428f16xww37h0-glibc-2.35-224 Deriver: 2vjs6q5j5vqckcwsvmh5lajvx3p7arkj-libnotify-0.8.1.drv Sig: cache.nixos.org-1:IqCAJROaqNx4TthRv9V47/dM7KP4sR+bBWBfL+9xSqQHAezcfczYdJhKj8nl5l+iFnj8O4uTIJMWNOcwVq8+AA==

```

Post reply on HN