Git archive checksums may change • Hacker News Board

Re: Git archive checksums may change

#181

Post by ilyt » Tue, Jan 31, 2023, 3:43 AM UTC

Denvercoder9 wrote: post #127

catiopatio wrote: post #122

Earlier quoted context omitted.

That’s expensive, complicated, exposes a greater attack surface, and requires new tooling to maintain considerably more complex metadata covering the full contents of source archives. For the entire multi-decade history of open source, the norm has been — for very good reason — that source archives are immutable and will not change. The solution here isn’t to change the entire open source ecosystem.

> For literally the entire multi-decade history of open source, the norm has been — for very good reason — that source archives are immutable and will not change. Well, the norm has been that maintainers generated and distributed a source archive, and that archive being immutable. That workflow is still perfectly fine with GitHub and not impacted by this change. The problem is that a bunch of maintainers stopped gene…

The problem was people assuming github works like that - saves a archive of every commit, which is obviously silly if you think about it (why save it if you can regenerate it on a whim from any commit you want?)

Re: Git archive checksums may change

#182

Post by ilyt » Tue, Jan 31, 2023, 3:46 AM UTC

vlovich123 wrote: post #22

ArchOversight wrote: post #16

Earlier quoted context omitted.

a git checkout of the code at that particular tag hasn't changed. Just the tarball that git archive generates has.

The two main problems are: A) How do you catch tarballs that have extra files injected that aren't part of your manifest B) What does the performance of this look like? Certainly for traditional HDDs this is going to kill performance, but even for SSDs I think verifying a bunch of small files is going to be less efficient than verifying the tarball.

Well, the simplest way would be to make checksum after decompression, that doesn't need per file verify and relies on files being put in same order into tar file.

The other method would be having Manifest file with checksum of every file inside the tar and compare that in-flight, could be simple "read from tar, compare to hash, write to disk" (with maybe some tmpfiles for the bigger ones)

Re: Git archive checksums may change

#183

Post by ilyt » Tue, Jan 31, 2023, 3:47 AM UTC

lopkeny12ko wrote: post #171

I can't fathom how no one internally at Microsoft-Github realized how widespread the breakage would be before rolling this out to all public users. Surely, Microsoft-Github's own internal builds would have started failing as a result of this change? Or do they not even canary releases internally at all?

I can

"didn't read every commit in new version of git, realized after the fact"

Re: Git archive checksums may change

#184

Post by ilyt » Tue, Jan 31, 2023, 3:50 AM UTC

bentley wrote: post #97

VWWHFSfQ wrote: post #82

Earlier quoted context omitted.

Ironic that "open source packaging systems" rely on proprietary Microsoft hosting and distribution to function. I think you meant _poorly implemented_ open source packaging systems.

Of the 11,656 packages in OpenBSD’s package repos, 2,984 are built from source originally hosted on GitHub or Sourceforge. That’s a full 25%. Moralize all you want about where these upstreams should host their software, but why claim that the downstream package manager is “poorly implemented” to fetch source code from those hosts? Your complaint was not technical—you imply the proprietariness of Microsoft servers is…

> Moralize all you want about where these upstreams should host their software, but why claim that the downstream package manager is “poorly implemented” to fetch source code from those hosts?

Because it should validate checksums of content of the tarball, instead of just the outside blob.

Then:

* you don't care about compression method or implementation * you don't care about archive method or implementation * your system works just as well for "download a tarball" as for "shallow copy the remote repo

Re: Git archive checksums may change

#185

Post by ilyt » Tue, Jan 31, 2023, 3:51 AM UTC

pxc wrote: post #149

Denvercoder9 wrote: post #139

Earlier quoted context omitted.

Or get the source code using Git, which actually (by design) guarantees that its checksums are stable.

That's a good point. It's way more resource-intensive and much slower, which is why it's not preferred in Nixpkgs, for example. But it's also vulnerable to the same problem in that your package manager's build system is still dependent on GitHub. It will take more to screw you up, but a whole GitHub outage, for example, will definitely still hurt.

It's not depending on github-specific functionality tho. You can just redirect it to another mirror of git repo of the project

Re: Git archive checksums may change

#186

Post by ilyt » Tue, Jan 31, 2023, 3:54 AM UTC

blueflow wrote: post #45

vlovich123 wrote: post #17

Hyrum's Law strikes again. It kind of doesn't matter what you document. If you weren't randomizing your checksum previously [1], you can't just spring this on the community and blame it for the fallout. I'm more shocked that there's resistance from the GitHub team saying "but we documented this isn't stable". Default stance for the team should be rollback & reevaluate an alternate path forward when the scope is this…

But look at it from the other side. Users that don't read your documentation and expect your software to work like they imagined are just a huge pain in the ass.

I'd imagine they broke their own stuff doing it, considering npm broke on it

Re: Git archive checksums may change

#187

Post by pabs3 » Tue, Jan 31, 2023, 4:50 AM UTC

I note that diffoscope is useful for verifying which parts of git/other archives have changed:

https://diffoscope.org/

You can try it online here:

https://try.diffoscope.org/

Re: Git archive checksums may change

#188

Post by rswail » Tue, Jan 31, 2023, 4:52 AM UTC

metrognome wrote: post #132

I wonder if this incident will encourage our industry to build more robust forms of artifact integrity verification, or if we will instead codify the status quo of "we guarantee repos to be archived deterministically." To me, the latter seems like a more troubling precedent.

This is being driven in industry by the push by US FedGov (via NIST) to have supply chain verification after the recent hacks.

POTUS issued an EO and NIST have been following up, leading to the promotion of schemes such as spdx https://tools.spdx.org/app/about/

Where I work is also required to start documenting our supply chain as part of the (new, replacing PCI-DSS) PCI-SFF certification requirements, which requires end-to-end verification of artifacts that are deployed within PCI scope.

So really, the arguments about CPU time etc are basically silly. The use of SHA hashes for artifacts that don't change will be a requirement for anyone building industrial software, or supplying to government, or in the money transacting business.

Re: Git archive checksums may change

#189

Post by [deleted] » Tue, Jan 31, 2023, 4:53 AM UTC

JoshTriplett wrote: post #112

bentley wrote: post #100

Earlier quoted context omitted.

Sadly there has been a sharp uptick in software that provides no release tarballs anymore. With the rise of GitHub many upstreams choose to make a tag and let people download the autogenerated tarballs, despite the fact that they won’t contain preprocessed autoconf or (more importantly) any Git submodules. The situation is deteriorating further as some projects make no releases at all, assuming users will add the pro…

> many upstreams choose to make a tag and let people download the autogenerated tarballs Which is fine as long as you rely on the hash of the tag rather than the hash of the tarball. > despite the fact that they won’t contain preprocessed autoconf This is a feature; run `autoreconf -vfi` at build time, so that you don't depend on the maintainer's idiosyncratic autotools setup and local macros, and so that you can rel…

[deleted]

Re: Git archive checksums may change

#190

Post by pabs3 » Tue, Jan 31, 2023, 4:55 AM UTC

chlorion wrote: post #133

VWWHFSfQ wrote: post #82

Earlier quoted context omitted.

Ironic that "open source packaging systems" rely on proprietary Microsoft hosting and distribution to function. I think you meant _poorly implemented_ open source packaging systems.

I am not sure how you could not rely on GitHub when packaging code that is hosted on GitHub. My personal Gentoo ebuilds for example contain a URI variable that points to the GitHub auto-generated archives for projects that use GitHub for hosting. What am I supposed to do in this case? The only option here is to setup a mirror and have a backup of the data. Packages that are in the official Gentoo repository do get mi…

Software Heritage backs up GitHub, they aren't really designed to be a mirror though.

https://www.softwareheritage.org/

Some GitHub repos also end up on archive.org, but not systematically.