Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Git archive checksums may change

github.blog

Post reply on HN
Print view View original

81–90 of 255 posts

Re: Git archive checksums may change

#81

Post by ajross »

ErikCorry wrote: post #34
swarfield wrote: post #10

Earlier quoted context omitted.

Using SHA hashes when building guarantees that the code that you are building is what you think it is. How else would you verify dependencies like this, GPG signatures would have the same issue if you change the underlying bits.

This seems like a weak argument. Firstly SHA is not a secure hash. Secondly if your build step involves uploading data to a third party then allowing them to transform it as they see fit and then checksumming the result then it's not really a reproducible build. For all you know, Github inserts a virus during the compression of the archive. What am I missing?

> Firstly SHA is not a secure hash.

It's... literally the Secure Hash Algorithm. (Yes, yes, SHA-1 was broken a while back, but SHA and derivatives were absolutely intended to provide secure collision resistance).

I think you're mixing things up here. Github didn't change the SHA-1 commit IDs in the repositories[1]. They changed the compression algorithm used for (and thus the file contents of) "git archive" output. So your tarballs have the same unpacked data but different hashes under all algorithms, secure or not.

> Secondly if your build step involves uploading data to a third party then allowing them to transform it as they see fit and then checksumming the result then it's not really a reproducible build. For all you know, Github inserts a virus during the compression of the archive.

Indeed. So you take and record a SHA-256 of the archive file you are tagging such that no one can feasibly do that!

Again, what's happened here is that the links pointing to generated archive files that projects assumed were immutable turned out not to be. It's got nothing to do with security or cryptography.

[1] Which would be a whole-internet-breaking catastrophe, of course. They didn't do that and never will.

Re: Git archive checksums may change

#82

Post by VWWHFSfQ »

johnklos wrote: post #74
vtbassmatt wrote: post #27

Earlier quoted context omitted.

We updated our Git version which made this change for the reasons explained. At the time we didn't foresee the impact. We're quickly rolling back the change now, as it's clear we need to look at this more closely to see if we can make the changes in a less disruptive way. Thanks for letting us know.

[flagged]

Ironic that "open source packaging systems" rely on proprietary Microsoft hosting and distribution to function.

I think you meant _poorly implemented_ open source packaging systems.

Re: Git archive checksums may change

#83

Post by tinus_hn »

scyrybdis wrote: post #69
denom wrote: post #40

Earlier quoted context omitted.

In my particular use-case, I'm using a set of local dev tools hosted as a homebrew tap. The build looks up the github tar.gz release for each tag and commits the sha256sum of that file to the formula What's odd is that all the _historical_ tags have broken release shasums. Does this mean the entire set of zip/tar.gz archives has been rebuilt? That could be a problem, as perhaps you cannot easily back out of this chan…

I think the zip/tar.gz archives are being created on the fly when you download them, probably with a caching layer in front.

Pretty bizarre this ever was stable in the first place.

Unfortunately for this kind of service you need to actively fiddle with the bytes to prevent people from relying on an implementation detail like this and prevent them from digging you into a too big to fail api stability hole.

Re: Git archive checksums may change

#84

Post by naikrovek »

sneak wrote: post #23
vlovich123 wrote: post #17

Hyrum's Law strikes again. It kind of doesn't matter what you document. If you weren't randomizing your checksum previously [1], you can't just spring this on the community and blame it for the fallout. I'm more shocked that there's resistance from the GitHub team saying "but we documented this isn't stable". Default stance for the team should be rollback & reevaluate an alternate path forward when the scope is this…

It's Microsoft. Just as the Apple of today is not the Apple of ten years ago, the GitHub today is not the GitHub of ten years ago. It's literally different people. The people who made the things you love have mostly moved on, and the brand is being run by different people with different values now. There's a little bit of an argument that such things are a bait-and-switch, but such is the nature of a large and multig…

The Microsoft of today isn't the Microsoft of 10 years ago, either, but that doesn't stop anyone from assuming that today's Microsoft is the same as the Microsoft of 10 years ago.

the logic people use to blame Microsoft is intense, man. literally any logical leap is valid except one that absolves Microsoft of anything, no matter how small.

Re: Git archive checksums may change

#85

Post by naikrovek »

nick__m wrote: post #9
frankjr wrote: post #5

GitHub will need to revert this change. They've just crippled pretty much every "from source" package manager out there.

I prefer that tool be adapted to be more resilient and not depend on github particular implementation.

the change was to git, not GitHub.

Re: Git archive checksums may change

#86

Post by vlovich123 »

blueflow wrote: post #45
vlovich123 wrote: post #17

Hyrum's Law strikes again. It kind of doesn't matter what you document. If you weren't randomizing your checksum previously [1], you can't just spring this on the community and blame it for the fallout. I'm more shocked that there's resistance from the GitHub team saying "but we documented this isn't stable". Default stance for the team should be rollback & reevaluate an alternate path forward when the scope is this…

But look at it from the other side. Users that don't read your documentation and expect your software to work like they imagined are just a huge pain in the ass.

Fact of life: the vast majority of your users do not read your documentation (or do not do so carefully enough that what you put in your docs is an ironclad proof that all users adhere to). That's literally what Hyrum's law is about. Of course, you can choose to do whatever you want. It's valuable to recognize of course that you're trading off good will from your users with whatever technical improvement is getting made. Sometimes it's appropriate and inevitable (e.g. old behavior is just wrong or harmful and better to cut off). In the vast majority of cases though it's better to just have a better process in place to manage this with minimal disruption, identifying and communicating with broken users, and only then making that change.

Re: Git archive checksums may change

#87

Post by SSLy »

johnklos wrote: post #74
vtbassmatt wrote: post #27

Earlier quoted context omitted.

We updated our Git version which made this change for the reasons explained. At the time we didn't foresee the impact. We're quickly rolling back the change now, as it's clear we need to look at this more closely to see if we can make the changes in a less disruptive way. Thanks for letting us know.

[flagged]

Most source-based pkg managers build from release tarballs, or even preprocessed tarballs after doing autoconf

Re: Git archive checksums may change

#88

Post by pxc »

frankjr wrote: post #5

GitHub will need to revert this change. They've just crippled pretty much every "from source" package manager out there.

Nixpkgs' so-called binary cache actually also caches source tarballs. Any Nix users out there who ran updates during the change?

Did cache hits save you? Did cache misses break your builds?

Re: Git archive checksums may change

#89

Post by vlovich123 »

nilsbunger wrote: post #72
vlovich123 wrote: post #17

Hyrum's Law strikes again. It kind of doesn't matter what you document. If you weren't randomizing your checksum previously [1], you can't just spring this on the community and blame it for the fallout. I'm more shocked that there's resistance from the GitHub team saying "but we documented this isn't stable". Default stance for the team should be rollback & reevaluate an alternate path forward when the scope is this…

This is especially true of something like a git SHA, which is drilled into your head as THE stable hash of your code and git tree at a certain state. It should be expected that lots of tools use it as an identifier -- heck, I've done so myself to confirm which version of a piece of software is deployed on a particular machine, etc.

To be fair this isn't the git SHA. This is the generated archive (apparently dynamically per request) when you ask for a source tarball.

Re: Git archive checksums may change

#90

Post by pxc »

groestl wrote: post #66
swarfield wrote: post #7

They have broken almost every open source project that builds external deps. Also broke homebrew apparently.

> every open source project that builds external deps and relies on checksumming ephemeral artefacts for integrity.

Such tools should definitely checksum package sources lol
Post reply on HN