Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Git archive checksums may change

github.blog

Post reply on HN
Print view View original

201–210 of 255 posts

Re: Git archive checksums may change

#201

Post by grepfru_it »

ZephyrBlu wrote: post #130
blueflow wrote: post #45

Earlier quoted context omitted.

But look at it from the other side. Users that don't read your documentation and expect your software to work like they imagined are just a huge pain in the ass.

You just described >90% of users. Everyone does this for something, most people do it for most things. You minimally read the docs, get something working and then leave it alone. Of course you're going to be pissed off when an implicit assumption which has been stable for a long time is broken.

>Of course you're going to be pissed off when an implicit assumption which has been stable for a long time is broken.

This accurately describes my beef with golang

Re: Git archive checksums may change

#202

Post by ilyt »

vlovich123 wrote: post #195
ilyt wrote: post #182

Earlier quoted context omitted.

Well, the simplest way would be to make checksum after decompression , that doesn't need per file verify and relies on files being put in same order into tar file. The other method would be having Manifest file with checksum of every file inside the tar and compare that in-flight, could be simple "read from tar, compare to hash, write to disk" (with maybe some tmpfiles for the bigger ones)

It’s not just about the integrity of the files you’re processing, but also the integrity of the archive itself. If you extract the tarball from a random place, there’s a larger security risk. Now granted HTTPS probably mitigates a lot of it, but cert pinning isn’t that common so MITM attacks aren’t thaaat theoretical.

You can do validation in flight during extraction. Signed file manifests are how distros like Debian did it since forever, althought in their cases its two step process, the packages themselves contain their own signature and whole directory tree also gets signed (to avoid shenaningans like "attacker putting older, still vulnerable, but signed version into the repo)

Re: Git archive checksums may change

#204

Post by missingdays »

ZephyrBlu wrote: post #130
blueflow wrote: post #45

Earlier quoted context omitted.

But look at it from the other side. Users that don't read your documentation and expect your software to work like they imagined are just a huge pain in the ass.

You just described >90% of users. Everyone does this for something, most people do it for most things. You minimally read the docs, get something working and then leave it alone. Of course you're going to be pissed off when an implicit assumption which has been stable for a long time is broken.

Yes, but if you implement the checksum algorithm for GitHub archives, shouldn't you read the documentation about archives checksum?

Re: Git archive checksums may change

#205

Post by ErikCorry »

blueflow wrote: post #60
ErikCorry wrote: post #34

Earlier quoted context omitted.

This seems like a weak argument. Firstly SHA is not a secure hash. Secondly if your build step involves uploading data to a third party then allowing them to transform it as they see fit and then checksumming the result then it's not really a reproducible build. For all you know, Github inserts a virus during the compression of the archive. What am I missing?

1) SHA-256 is reasonably secure 2) The checksum assures you that the file you have is the same your upstream looked at

1) Ah of course, this is SHA256, my mistake.

2) If I and the upstream are both looking at a file that was generated by Github then the Sha may match, but that doesn't prove we weren't both owned by Github.

Perhaps what I am missing is that this isn't part of a reproducible build scenario. There's no attempt to ensure that the file Github had built is the one I would build with the same starting point.

Re: Git archive checksums may change

#206

Post by shakow »

viraptor wrote: post #175
shakow wrote: post #113

Earlier quoted context omitted.

But then that's the role of the httpS query with which you will fetch your data. And if you don't trust your http layer and/or Github's certificate, then you should not trust their archive anyway.

The certificate guarantees the source of the file, not the trust you should put in its contents. I can upload malware as a github project release file and https doesn't change that you shouldn't download/run it. For software distribution this actually sometimes goes the other way - debian/ubuntu uses http (no s) for their packages, because the content itself is signed by the distribution and this way you can easily c…

> I can upload malware as a github project release file and https doesn't change that you shouldn't download/run it.

If you can't trust the archive published by the owner themselves, you are already screwed; a stable hash will just make sure that you trust harder that you are, indeed, downloading contaminated code.

I'm not sure most people here understand how checksums/hashs work, what they protect you against, and what they don't.

Re: Git archive checksums may change

#207

Post by shakow »

bentley wrote: post #193
shakow wrote: post #152

Earlier quoted context omitted.

And you really believe that downloading & extracting a source .tar.gz and compiling it will have a run time much shorter than 0.4s? Just executing the ./configure will take more than that.

> And you really believe … Huh? What I fully believe is that downloading a source tarball over HTTPS, verifying its checksum, and extracting it will take less time than cloning the repository from Git, then verifying the checksum of all files—which you said would take 29 seconds plus 0.4s.

My point is that either spending 0.08s computing the md5 of the zip (I just measured) or 0.3s computing the hash of the repo does not matter the slightest if you are managing software repos, as just extracting the source and preparing to build it will be an order of magnitude slower.

Re: Git archive checksums may change

#208

Post by kelnos »

The thing I don't get is how this ever worked.

The change was upstream from git itself, and it was to use the builtin (zlib-based) compression code in git, rather than shelling out to gzip.

But would the gzip binary itself give reproducible results across versions of gzip (and zlib)? Intuition seems to suggest it wouldn't, at least not always. And if not, was the "strategy" just to never update gzip or zlib on GitHub's servers? That seems like a non-starter...

Re: Git archive checksums may change

#209

Post by kzrdude »

acdha wrote: post #15
frankjr wrote: post #5

GitHub will need to revert this change. They've just crippled pretty much every "from source" package manager out there.

If those tools incorrectly assume an API contract which doesn't exist, isn't the right answer to fix those tools?

Well, Github presents a file that looks like it comes from a file server, an old "ftp" archive or so. So they model it on that. Already published versions and tar balls should not change in those systems.

I think everyone knows these files are generated on the fly, but it comes from old habits.

Re: Git archive checksums may change

#210

Post by kzrdude »

catiopatio wrote: post #116
groestl wrote: post #66

Earlier quoted context omitted.

> every open source project that builds external deps and relies on checksumming ephemeral artefacts for integrity.

Source archives have never , in the entire history of open source, been considered ephemeral. GitHub unilaterally made that decision for their own convenience, and violated a decades-long universal community norm in the process.

All these projects relying on github, they are using a free service they don't control. It could go away someday. That will be a bigger crisis than this was..
Post reply on HN