Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Git archive checksums may change

github.blog

Post reply on HN
Print view View original

151–160 of 255 posts

Re: Git archive checksums may change

#151

Post by denom »

tinus_hn wrote: post #83
scyrybdis wrote: post #69

Earlier quoted context omitted.

I think the zip/tar.gz archives are being created on the fly when you download them, probably with a caching layer in front.

Pretty bizarre this ever was stable in the first place. Unfortunately for this kind of service you need to actively fiddle with the bytes to prevent people from relying on an implementation detail like this and prevent them from digging you into a too big to fail api stability hole.

Alternatively, they could extract the compression code and maintain it for repo tags created before the git algo update release date.

Isn’t that the only humane course given all that depends on this?

Re: Git archive checksums may change

#152

Post by shakow »

bentley wrote: post #143
shakow wrote: post #140

Earlier quoted context omitted.

> That’s expensive, complicated, That sounds like prejudice. Just as a test, I cloned the git repo, which took 29 seconds, then took its hash with `guix hash`, which took 0.387ms. I think that if you can't handle a 0.4s delay in a build, you have problem problems.

Package builders work on the scale of thousands of packages. The increased time and CPU usage multiplies greatly. “Complicated” is indisputable. Cloning a repository is absolutely complicated. Fetching a single file over HTTPS is as simple as it gets, these days.

And you really believe that downloading & extracting a source .tar.gz and compiling it will have a run time much shorter than 0.4s?

Just executing the ./configure will take more than that.

Re: Git archive checksums may change

#153

Post by skobovm »

yakubin wrote: post #148
DoctorNick wrote: post #141

Earlier quoted context omitted.

With what? The abomination that is `git submodules`?

No. Just copy files into the repo. Any way you like. In a GUI, in a terminal — it doesn’t require a dedicated tool. Although cargo in Rust e.g. provides a dubcommand for it (cargo vendor). Alternatively you can host the tarballs somewhere you control in static storage — be it a static web server, object storage or whatever. How it’s done in Chromium: https://source.chromium.org/chromium/chromium/src/+/main:thi... >.

Woof. At the rate packages get updated these days, and the amount of dependencies between them, that just isn't sustainable for any reasonably-sized project in server and -- especially -- frontend land.

Re: Git archive checksums may change

#154

Post by denom »

VWWHFSfQ wrote: post #115
bentley wrote: post #97

Earlier quoted context omitted.

Of the 11,656 packages in OpenBSD’s package repos, 2,984 are built from source originally hosted on GitHub or Sourceforge. That’s a full 25%. Moralize all you want about where these upstreams should host their software, but why claim that the downstream package manager is “poorly implemented” to fetch source code from those hosts? Your complaint was not technical—you imply the proprietariness of Microsoft servers is…

I'm fairly certain that Homebrew doesn't function without GitHub. The index (and Cargo's index, and probably others) is hard-coded to be hosted on GitHub.

Exactly this.

Tbh, i would expect some developer working on this feature to have an ‘a-ha’ moment: ‘I’m a homebrew user … hey wait a minute’

Re: Git archive checksums may change

#155

Post by misnome »

bentley wrote: post #131
rsc wrote: post #123

Earlier quoted context omitted.

Thanks for the quick rollback. I want to encourage you to think about locking in the current archive details, at least for archives that have already been served. Verifying that downloaded archives have the expected checksum is a critical best practice for software supply chain security. Training people to ignore checksum changes is training them to ignore attacks. GitHub is a strong leader in other parts of supply c…

I would also appreciate stronger advertising of the ability to turn a Git tag into a GitHub release and upload stable source code files to it. Maybe even a button in the GitHub releases interface to “generate source tarball and attach as stable tarball to this release.”

But this isn’t a great solution, because afterwards there is now three, or four source download links, some of which are stable.

Not to mention, forcing people to use GitHub releases instead of just tags (which excludes every mirror of somewhere else)

Re: Git archive checksums may change

#156

Post by ecnahc515 »

Denvercoder9 wrote: post #139
pxc wrote: post #117

Earlier quoted context omitted.

> I think you meant _poorly implemented_ open source packaging systems. or under-resourced ones. If the upstream source only appears on GitHub, without formal release tarballs, your only options as a downstream packager are literally to get the source from GitHub or host your own mirror of every source tarball you build yourself.

Or get the source code using Git, which actually (by design) guarantees that its checksums are stable.

Downloading a source tarball is significantly cheaper on both sides than git. A source tarball is 100% served from CDNs, whereas I don't believe the same isn't quite true for git (even over https).

Re: Git archive checksums may change

#157

Post by thirtyseven »

c4mpute wrote: post #91
nilsbunger wrote: post #72

Earlier quoted context omitted.

This is especially true of something like a git SHA, which is drilled into your head as THE stable hash of your code and git tree at a certain state. It should be expected that lots of tools use it as an identifier -- heck, I've done so myself to confirm which version of a piece of software is deployed on a particular machine, etc.

Yes, but not in this bug. I guess lots of people missed that distinction: The stable git SHA hash is the commit hash, which is an hash over gits internal representation of the commit object (containing a tree of all file hashes, and parents' hashes). The hash that pops out of 'git archive' has nothing whatsoever to do with the commit hash and was historically stable more or less by accident: git feeds all files to 't…

The people that this broke weren't directly depending on the output of git archive being stable, but were assuming that the response data for a particular URL would stay constant. Maybe not a great idea either but not entirely unreasonable IMO.

Re: Git archive checksums may change

#158

Post by forgotpwd16 »

georgyo wrote: post #67
forgotpwd16 wrote: post #39

Earlier quoted context omitted.

What was the thought behind this change?

If you read the commit message you would see that it is up drop a third party dependency.

Yeah, read that. Just don't understand, if git already had an internal gzip implementation, why wasn't it used since it was added?

Re: Git archive checksums may change

#159

Post by ecnahc515 »

Denvercoder9 wrote: post #135
catiopatio wrote: post #116

Earlier quoted context omitted.

Source archives have never , in the entire history of open source, been considered ephemeral. GitHub unilaterally made that decision for their own convenience, and violated a decades-long universal community norm in the process.

You could also say that some maintainers made that decision for their convenience of not having to build and upload source archives. It is possible to upload your own artifacts to a release on GitHub, and lots of projects do. Those are correctly treated as immutable by GitHub.

I imagine GitHub would be equally upset if every project every started uploading their source tarballs themselves, as presumably the point of their source tarball feature was not needing to store the source tarball for every repository ever, but generating it on demand.

Re: Git archive checksums may change

#160

Post by DoctorNick »

skobovm wrote: post #153
yakubin wrote: post #148

Earlier quoted context omitted.

No. Just copy files into the repo. Any way you like. In a GUI, in a terminal — it doesn’t require a dedicated tool. Although cargo in Rust e.g. provides a dubcommand for it (cargo vendor). Alternatively you can host the tarballs somewhere you control in static storage — be it a static web server, object storage or whatever. How it’s done in Chromium: https://source.chromium.org/chromium/chromium/src/+/main:thi... >.

Woof. At the rate packages get updated these days, and the amount of dependencies between them, that just isn't sustainable for any reasonably-sized project in server and -- especially -- frontend land.

Exactly. Unless the package manager has a mechanism for doing that, good fucking luck updating any of your packages ever again.
Post reply on HN