Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Git archive checksums may change

github.blog

Post reply on HN
Print view View original

111–120 of 255 posts

Re: Git archive checksums may change

#111

Post by shakow »

swarfield wrote: post #10
nick__m wrote: post #9

Earlier quoted context omitted.

I prefer that tool be adapted to be more resilient and not depend on github particular implementation.

Using SHA hashes when building guarantees that the code that you are building is what you think it is. How else would you verify dependencies like this, GPG signatures would have the same issue if you change the underlying bits.

By checking the hash of the extracted files. The hash of the archive is dependent on the order in which the file were compressed, the compression, some metadata, etc.

Re: Git archive checksums may change

#112

Post by JoshTriplett »

bentley wrote: post #100
SSLy wrote: post #87

Earlier quoted context omitted.

Most source-based pkg managers build from release tarballs, or even preprocessed tarballs after doing autoconf

Sadly there has been a sharp uptick in software that provides no release tarballs anymore. With the rise of GitHub many upstreams choose to make a tag and let people download the autogenerated tarballs, despite the fact that they won’t contain preprocessed autoconf or (more importantly) any Git submodules. The situation is deteriorating further as some projects make no releases at all, assuming users will add the pro…

> many upstreams choose to make a tag and let people download the autogenerated tarballs

Which is fine as long as you rely on the hash of the tag rather than the hash of the tarball.

> despite the fact that they won’t contain preprocessed autoconf

This is a feature; run `autoreconf -vfi` at build time, so that you don't depend on the maintainer's idiosyncratic autotools setup and local macros, and so that you can reliably regenerate it all if you want to change configure.ac or Makefile.am.

Re: Git archive checksums may change

#113

Post by shakow »

bentley wrote: post #103
Denvercoder9 wrote: post #78

Earlier quoted context omitted.

I wouldn't check the hash of the compressed archive, but of the actual files themselves. It's a bit more metadata, but it's also a lot more robust, and allows you to detect changes after unpacking as well.

It’s generally a bad idea to process (extract) a tarball of unknown provenance. Verifying the tarball is from a known source beforehand mitigates the risk of, say, a malicious tarball that exploits a tar or gzip 0‐day.

But then that's the role of the httpS query with which you will fetch your data.

And if you don't trust your http layer and/or Github's certificate, then you should not trust their archive anyway.

Re: Git archive checksums may change

#114

Post by dataflow »

blueflow wrote: post #110
dataflow wrote: post #101

Earlier quoted context omitted.

I don't think expecting users to go look for a user manual on each website whose links they download from is a realistic expectation.

Worse, you can't expect other people to host your data for free, forever. If you want your data distributed, you need to check first if the platform is suitable for your purposes.

I don't believe paid users saw any different behavior here?

Re: Git archive checksums may change

#115

Post by VWWHFSfQ »

bentley wrote: post #97
VWWHFSfQ wrote: post #82

Earlier quoted context omitted.

Ironic that "open source packaging systems" rely on proprietary Microsoft hosting and distribution to function. I think you meant _poorly implemented_ open source packaging systems.

Of the 11,656 packages in OpenBSD’s package repos, 2,984 are built from source originally hosted on GitHub or Sourceforge. That’s a full 25%. Moralize all you want about where these upstreams should host their software, but why claim that the downstream package manager is “poorly implemented” to fetch source code from those hosts? Your complaint was not technical—you imply the proprietariness of Microsoft servers is…

I'm fairly certain that Homebrew doesn't function without GitHub. The index (and Cargo's index, and probably others) is hard-coded to be hosted on GitHub.

Re: Git archive checksums may change

#116

Post by catiopatio »

groestl wrote: post #66
swarfield wrote: post #7

They have broken almost every open source project that builds external deps. Also broke homebrew apparently.

> every open source project that builds external deps and relies on checksumming ephemeral artefacts for integrity.

Source archives have never, in the entire history of open source, been considered ephemeral.

GitHub unilaterally made that decision for their own convenience, and violated a decades-long universal community norm in the process.

Re: Git archive checksums may change

#117

Post by pxc »

VWWHFSfQ wrote: post #82
johnklos wrote: post #74

Earlier quoted context omitted.

[flagged]

Ironic that "open source packaging systems" rely on proprietary Microsoft hosting and distribution to function. I think you meant _poorly implemented_ open source packaging systems.

> I think you meant _poorly implemented_ open source packaging systems.

or under-resourced ones. If the upstream source only appears on GitHub, without formal release tarballs, your only options as a downstream packager are literally to get the source from GitHub or host your own mirror of every source tarball you build yourself.

Re: Git archive checksums may change

#118

Post by misnome »

skobovm wrote: post #65

I wonder what monetary loss in productivity was due to this change. We noticed this issue a bit before noon, tracked it down to GH, sent out company-wide comms notifying others of the problem, filed tickets with GH, had to modify numerous repos across multiple teams, and now it's 3pm and I'm here reading about it. It's crazy how such a seemingly innocuous change, like this, could lead to such widespread loss in produ…

Our conda-forge package builds broke. We had someone declare to us that tag downloads were never stable, just releases. This seems to be the opposite of the known truth about the previous status quo - but does go some way to demonstrating how little the state of the actual guarantees for this system were understood.

Re: Git archive checksums may change

#119

Post by vlovich123 »

ArchOversight wrote: post #37
vlovich123 wrote: post #22

Earlier quoted context omitted.

The two main problems are: A) How do you catch tarballs that have extra files injected that aren't part of your manifest B) What does the performance of this look like? Certainly for traditional HDDs this is going to kill performance, but even for SSDs I think verifying a bunch of small files is going to be less efficient than verifying the tarball.

A wouldn't be an issue since you are checking out a git tag. B would just be a normal git checkout, which already validates that all the objects are reachable and git tags (and commits for that matter) can be signed, and since the sha1 hash is signed as well it validates that the entire tree of commits has not been tampered with. So as long you trust git to not lie about what it is writing to disk, you have a valid c…

I know GitHub had asked that clones from package manager use shallow clones. It wouldn't surprise me if downloading tarballs is similarly beneficial to GitHub because it's trivially cacheable in a CDN and thus lowers their operational footprint to support package managers.

Re: Git archive checksums may change

#120

Post by vlovich123 »

blueflow wrote: post #105
vlovich123 wrote: post #86

Earlier quoted context omitted.

Fact of life: the vast majority of your users do not read your documentation (or do not do so carefully enough that what you put in your docs is an ironclad proof that all users adhere to). That's literally what Hyrum's law is about. Of course, you can choose to do whatever you want. It's valuable to recognize of course that you're trading off good will from your users with whatever technical improvement is getting m…

Thats support you could expect if you paid for it.

Look. Even vcpkg broke which is a Microsoft product. I agree that there can be a continuum some times, but can we agree that this specific instance isn't anything like that? Even without vcpkg, the list of things impacted are anything that depends on Bazel, homebrew, conan, etc. The blast radius is quite wide regardless of documentation.
Post reply on HN