Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Git archive checksums may change

github.blog

Post reply on HN
Print view View original

161–170 of 255 posts

Re: Git archive checksums may change

#161

Post by ecnahc515 »

denom wrote: post #151
tinus_hn wrote: post #83

Earlier quoted context omitted.

Pretty bizarre this ever was stable in the first place. Unfortunately for this kind of service you need to actively fiddle with the bytes to prevent people from relying on an implementation detail like this and prevent them from digging you into a too big to fail api stability hole.

Alternatively, they could extract the compression code and maintain it for repo tags created before the git algo update release date. Isn’t that the only humane course given all that depends on this?

That's my thought as well. They could also potentially retroactively generate the source tarballs using the old method for every possible repository/tag on Github, store it, and serve that, and then only generate it on-demand for new tags, but I doubt they'll do that. They might though, given this is what led to the problem in the first place (ie; the on-demand generation vs generating on push+storing).

Re: Git archive checksums may change

#162

Post by lupire »

blueflow wrote: post #45
vlovich123 wrote: post #17

Hyrum's Law strikes again. It kind of doesn't matter what you document. If you weren't randomizing your checksum previously [1], you can't just spring this on the community and blame it for the fallout. I'm more shocked that there's resistance from the GitHub team saying "but we documented this isn't stable". Default stance for the team should be rollback & reevaluate an alternate path forward when the scope is this…

But look at it from the other side. Users that don't read your documentation and expect your software to work like they imagined are just a huge pain in the ass.

If you don't want users, feel free to ignore them.

Re: Git archive checksums may change

#163

Post by rabexc23 »

yakubin wrote: post #148
DoctorNick wrote: post #141

Earlier quoted context omitted.

With what? The abomination that is `git submodules`?

No. Just copy files into the repo. Any way you like. In a GUI, in a terminal — it doesn’t require a dedicated tool. Although cargo in Rust e.g. provides a dubcommand for it (cargo vendor). Alternatively you can host the tarballs somewhere you control in static storage — be it a static web server, object storage or whatever. How it’s done in Chromium: https://source.chromium.org/chromium/chromium/src/+/main:thi... >.

vendoring even with a tool has always worked poorly with me. Here are a few reasons:

1. You work in a company, you are in a team, you want some reasonable code review process in place. Now you want to check in a 3rd party dependency, "let's vendor it!" so you send out a PR with ... 10,000 - 100,000 lines of code. Your reviewer has no reasonable way to know if a) the dependency was downloaded from a reputable source, if b) the code was not modified maliciously, c) there was some local patch / local change either voluntarily added or accidentally added (maybe you tried running configure/make locally, and didn't realize that one .h file was generated from your machine. A diligent reviewer would have to re-download the source tarball from a reputable source (is there the url in the commit message? A README? better hope there is!), unpack it locally, generate the set of files and all hashes, compare with your PR. And ensure that the PR / vendored dependency comes with a README or METADATA file so the download URL and LICENSE is recorded for posterity.

2. Now you need to update the dependency. Either it's a new directory (so you vendor both versions), or you have to delete all files that are gone. The PR review will be worse, as it will show a diff, except the reviewer won't review it, except to repeat the steps in 1. Without considering patches applied in the mean time, as the code was simply checked in the repository, and anyone could easily change it.

3. For anything but small/tiny projects, the vendoring will take up most of the download / checkout time of your repository.

If you use git for vendoring, the problem is not significantly better: if you care about the integrity of the vendored code, you need to verify the final tree, or the log / hash / set of commits.

Compare to using a simple file with a 1) url, 2) secure hash, 3) list of patches to apply. Reviewing and ensuring correctness is trivial, upgrading is trivial, PRs are trivial.

To avoid problems like the github problem here, a simple proxy or local cache is enough, a tool that takes the hash (or a hash of a url) and reads it from disk, is good enough. And detects corruption.

Re: Git archive checksums may change

#164

Post by voidbip »

mdouglass wrote: post #98
vtbassmatt wrote: post #20

Hey folks. I'm the product manager for Git at GitHub. We're sorry for the breakage, we're reverting the change, and we'll communicate better about such changes in the future (including timelines). Also posted here: https://github.com/bazel-contrib/SIG-rules-authors/issues/11...

We are seeing an npm install failure inside our docker builds pointing at a github URL with a SHA change. Is this possibly related? #15 [dev-builder 4/7] RUN --mount=type=secret,id=npm,dst=/root/.npmrc npm ci #0 4.743 npm WARN deprecated querystring@0.2.0: The querystring API is considered Legacy. new code should use the URLSearchParams API instead. #0 8.119 npm WARN tarball tarball data for http2@https://github.com/…

Just want to second this. Still seeing an issue in our build right now that seems related.

``` Building aws-sdk-cpp[core,dynamodb,kinesis,s3]:x64-linux... -- Downloading https://github.com/aws/aws-sdk-cpp/archive/a72b841c91bd421fb... -> aws-aws-sdk-cpp-a72b841c91bd421fbb6deb516400b51c06bc596c.tar.gz... [DEBUG] To include the environment variables in debug output, pass --debug-env [DEBUG] Feature flag 'binarycaching' unset [DEBUG] Feature flag 'manifests' = off [DEBUG] Feature flag 'compilertracking' unset [DEBUG] Feature flag 'registries' unset [DEBUG] Feature flag 'versions' unset [DEBUG] 5612: popen( curl --fail -L https://github.com/aws/aws-sdk-cpp/archive/a72b841c91bd421fb... --create-dirs --output /home/*redacted*/vcpkg/downloads/aws-aws-sdk-cpp-a72b841c91bd421fbb6deb516400b51c06bc596c.ta r.gz.5612.part 2>&1) [DEBUG] 5612: cmd_execute_and_stream_data() returned 0 after 12643779 us Error: Failed to download from mirror set: File does not have the expected hash: url : [ https://github.com/aws/aws-sdk-cpp/archive/a72b841c91bd421fb... ] File path : [ /home/*redacted*/vcpkg/downloads/aws-aws-sdk-cpp-a72b841c91bd421fbb6deb516400b51c06bc596c.tar.gz.5612.part ] Expected hash : [ 9b7fa80ee155fa3c15e3e86c30b75c6019dc1672df711c4f656133fe005f104e4a30f5a99f1c0a0c6dab42007b5695169cd312bd0938b272c4c7b05765ce3421 ] Actual hash : [ 503d49a8dc04f9fb147c0786af3c7df8b71dd3f54b8712569500071ee24c720a47196f4d908d316527dd74901cb2f92f6c0893cd6b32aaf99712b27ae8a56fb2 ] ```

Re: Git archive checksums may change

#166

Post by reindeerer »

yakubin wrote: post #104

Now I’m having a laugh at all those times someone tried to explain to me that vendoring dependencies doesn’t make sense, when you have package managers which verify checksums of the things downloaded from GitHub/wherever. A good laugh. Keep it simple, just vendor your deps.

This is a false choice. "Vendoring" is much more of a mess than this is, and second, there's no reason to rely on these on the fly tarballs for anything, when proper versioned software releases exist.

Github has pretty much a one-click ( or one API call ) workflow to create properly versioned and archived tarballs. Just because lots of people try to skirt proper version management doesn't mean you should commit the world into your repo

Re: Git archive checksums may change

#167

Post by reindeerer »

bentley wrote: post #136
metrognome wrote: post #132

I wonder if this incident will encourage our industry to build more robust forms of artifact integrity verification, or if we will instead codify the status quo of "we guarantee repos to be archived deterministically." To me, the latter seems like a more troubling precedent.

We’ve regressed from the previous norm of open source projects providing stable source tarballs with fixed checksums, sometimes even with cryptographic signatures.

That norm still exists, and it's offered by Github in form of Github Releases feature as well.

It's the downstream tooling ( i.e. all the builds and package managers ) that need to clean their act up.

Re: Git archive checksums may change

#168

Post by JoshTriplett »

bentley wrote: post #121
JoshTriplett wrote: post #112

Earlier quoted context omitted.

> many upstreams choose to make a tag and let people download the autogenerated tarballs Which is fine as long as you rely on the hash of the tag rather than the hash of the tarball. > despite the fact that they won’t contain preprocessed autoconf This is a feature; run `autoreconf -vfi` at build time, so that you don't depend on the maintainer's idiosyncratic autotools setup and local macros, and so that you can rel…

> This is a feature; run `autoreconf -vfi` at build time, so that you don't depend on the maintainer's idiosyncratic autotools setup and local macros, and so that you can reliably regenerate it all if you want to change configure.ac or Makefile.am. On a package bulk build machine, that’s a lot (like, a lot ) of wasted CPU cycles multiplied by the thousands of packages that use autoconf. For the majority of packages t…

There are a lot of things that waste cycles on a build machine, but they're worth the reproduciblity. I think that problem would be better solved via caching, ideally.

I would hazard a guess that there are far fewer people these days who download a tarball and `./configure` `make` `make install` than there are distros (who often need to patch) and developers (who will be working from git anyway).

Re: Git archive checksums may change

#169

Post by semiquaver »

jzelinskie wrote: post #46

Does anyone have the motivation for why the git project wants to use their own implementation of gzip? Did this implementation already exist and was being used for something else? I understand wanting fewer dependencies, but gut-reaction is that it's a bad move in the unsafe world of C to rewrite something that already has a far more audited, ubiquitous implementation.

“Their own” implementation is just zlib, already in use throughout git since the dawn of the project for other purposes like blob storage [1].

Depending on how you measure it, zlib might be considered significantly more ubiquitous than gzip itself. At any rate it’s certainly no less battle tested.

[1] https://git-scm.com/book/en/v2/Git-Internals-Git-Objects

Post reply on HN