Live data from Hacker News

Git archive checksums may change

github.blog

91–100 of 255 posts

Re: Git archive checksums may change

#91

Hyrum's Law strikes again. It kind of doesn't matter what you document. If you weren't randomizing your checksum previously [1], you can't just spring this on the community and blame it for the fallout. I'm more shocked that there's resistance from the GitHub team saying "but we documented this isn't stable". Default stance for the team should be rollback & reevaluate an alternate path forward when the scope is this…

This is especially true of something like a git SHA, which is drilled into your head as THE stable hash of your code and git tree at a certain state. It should be expected that lots of tools use it as an identifier -- heck, I've done so myself to confirm which version of a piece of software is deployed on a particular machine, etc.

Yes, but not in this bug. I guess lots of people missed that distinction: The stable git SHA hash is the commit hash, which is an hash over gits internal representation of the commit object (containing a tree of all file hashes, and parents' hashes).

The hash that pops out of 'git archive' has nothing whatsoever to do with the commit hash and was historically stable more or less by accident: git feeds all files to 'tar' in tree order (which is fixed) and (unless you specify otherwise) always uses gzip with the same options. Since they no longer use gzip but an internal call to zlib, compression output will look different but will still contain the same tar inside.

That people have relied on this archive hash being stable is an indication of a major problem imho, because it might mean that people in their heads project integrity guarantees from the commit hash (which has such guarantees) onto the archive hash (which doesn't have those guarantees). I would suggest randomizing the archive hash on purpose by introducing randomness somewhere, so that people no longer rely on it.

Re: Git archive checksums may change

#92

Hyrum's Law strikes again. It kind of doesn't matter what you document. If you weren't randomizing your checksum previously [1], you can't just spring this on the community and blame it for the fallout. I'm more shocked that there's resistance from the GitHub team saying "but we documented this isn't stable". Default stance for the team should be rollback & reevaluate an alternate path forward when the scope is this…

This is especially true of something like a git SHA, which is drilled into your head as THE stable hash of your code and git tree at a certain state. It should be expected that lots of tools use it as an identifier -- heck, I've done so myself to confirm which version of a piece of software is deployed on a particular machine, etc.

The Git commit hashes didn't change (that'd actually be a serious problem). The hash of a compressed archive of the contents of a Git commit changed.

Re: Git archive checksums may change

#94

Earlier quoted context omitted.

They're still using zlib to do the heavy lifting. It's not a large patch. https://public-inbox.org/git/1328fe72-1a27-b214-c226-d239099...

> So the internal implementation takes 17% longer on the Linux repo, but > uses 2% less CPU time. That's because the external gzip can run in > parallel on its own processor, while the internal one works sequentially > and avoids the inter-process communication overhead. > What are the benefits? Only an internal sequential implementation can > offer this eco mode, and it allows avoiding the gzip(1) requirement. It se…

Because they pay for the 2% CPU time, not for the 17% local time. In theory the user also pays for 2% less CPU time, but they are much less likely to be CPU limited in their build processes.

Of course 17% more time may not really be that much for most processes. Are we talking about 17% more of a second or of an hour?

Re: Git archive checksums may change

#95

Yet another reason why GitHub is not a good Artifactory/Nexus replacement. Anyone remember the crazyness when Homebrew had problems with using GitHub for the same thing?

this is a git behavior, not a GitHub behavior.

files uploaded to GH Packages are not modified by GitHub.

only the "Source Code (.zip)" and "Source Code (.tgz)" files that are part of releases and tags are affected because git generates them on demand, and git does not guarantee hash stability.

if you upload a package to GH Packages or upload a release asset to a GitHub releases those are never modified, and you can rely on those hashes.

Re: Git archive checksums may change

#96
post #55

Earlier quoted context omitted.

Consumers often mistake hasn’t changed for a commitment to never change: any sufficiently large product will be littered with these kind of implicit commitments made by the product to consumers that nobody has visibility into. You’re unfortunate that we were all relying on this commitment you’ve never made, but the quick reversion is the best we can hope for. People will theorise how this could have been avoided but…

Hyrum's Law: With a sufficient number of users of an API, it does not matter what you promise in the contract: all observable behaviors of your system will be depended on by somebody.

[deleted]

Re: Git archive checksums may change

#97

Earlier quoted context omitted.

[flagged]

Ironic that "open source packaging systems" rely on proprietary Microsoft hosting and distribution to function. I think you meant _poorly implemented_ open source packaging systems.

Of the 11,656 packages in OpenBSD’s package repos, 2,984 are built from source originally hosted on GitHub or Sourceforge. That’s a full 25%.

Moralize all you want about where these upstreams should host their software, but why claim that the downstream package manager is “poorly implemented” to fetch source code from those hosts? Your complaint was not technical—you imply the proprietariness of Microsoft servers is the problem (although open source servers like GitLab also have the problem of unstable checksums)—but HTTPS is HTTPS.

Re: Git archive checksums may change

#98

Hey folks. I'm the product manager for Git at GitHub. We're sorry for the breakage, we're reverting the change, and we'll communicate better about such changes in the future (including timelines). Also posted here: https://github.com/bazel-contrib/SIG-rules-authors/issues/11...

We are seeing an npm install failure inside our docker builds pointing at a github URL with a SHA change. Is this possibly related?

  #15 [dev-builder 4/7] RUN --mount=type=secret,id=npm,dst=/root/.npmrc npm ci
  #0 4.743 npm WARN deprecated querystring@0.2.0: The querystring API is considered Legacy. new code should use the URLSearchParams API instead.
  #0 8.119 npm WARN tarball tarball data for http2@https://github.com/node-apn/node-http2/archive/apn-2.1.4.tar.gz (sha512-ad4u4I88X9AcUgxCRW3RLnbh7xHWQ1f5HbrXa7gEy2x4Xgq+rq+auGx5I+nUDE2YYuqteGIlbxrwQXkIaYTfnQ==) seems to be corrupted. Trying again.
  #0 8.164 npm ERR! code EINTEGRITY
  #0 8.169 npm ERR! sha512-ad4u4I88X9AcUgxCRW3RLnbh7xHWQ1f5HbrXa7gEy2x4Xgq+rq+auGx5I+nUDE2YYuqteGIlbxrwQXkIaYTfnQ== integrity checksum failed when using sha512: wanted sha512-ad4u4I88X9AcUgxCRW3RLnbh7xHWQ1f5HbrXa7gEy2x4Xgq+rq+auGx5I+nUDE2YYuqteGIlbxrwQXkIaYTfnQ== but got sha512-GWBlkDNYgpkQElS+zGyIe1CN/XJxdEFuguLHOEGLZOIoDiH4cC9chggBwZsPK/Ls9nPikTzMuRDWfLzoGlKiRw==. (72989 bytes)
  #0 8.176 
  #0 8.177 npm ERR! A complete log of this run can be found in:
  #0 8.177 npm ERR!     /root/.npm/_logs/2023-01-30T23_19_36_986Z-debug-0.log
  #15 ERROR: process "/bin/sh -c npm ci" did not complete successfully: exit code: 1
This was working earlier today and the docker build/package.json haven't changed.

Re: Git archive checksums may change

#99

Earlier quoted context omitted.

They're still using zlib to do the heavy lifting. It's not a large patch. https://public-inbox.org/git/1328fe72-1a27-b214-c226-d239099...

> So the internal implementation takes 17% longer on the Linux repo, but > uses 2% less CPU time. That's because the external gzip can run in > parallel on its own processor, while the internal one works sequentially > and avoids the inter-process communication overhead. > What are the benefits? Only an internal sequential implementation can > offer this eco mode, and it allows avoiding the gzip(1) requirement. It se…

It seems like if they really wanted to save CPU they'd be caching the outputs. I fail to see why they would be recompressing years-old release tags. This seems like optimization at the wrong level.

That's without even mentioning the absurdity of saving 2% CPU but still using zlib.

Re: Git archive checksums may change

#100
post #87

Earlier quoted context omitted.

[flagged]

Most source-based pkg managers build from release tarballs, or even preprocessed tarballs after doing autoconf

Sadly there has been a sharp uptick in software that provides no release tarballs anymore. With the rise of GitHub many upstreams choose to make a tag and let people download the autogenerated tarballs, despite the fact that they won’t contain preprocessed autoconf or (more importantly) any Git submodules.

The situation is deteriorating further as some projects make no releases at all, assuming users will add the project’s own package mirror to the user’s trusted package repositories, or use Docker.

Post reply on HN