Live data from Hacker News

Git archive checksums may change

github.blog

171–180 of 255 posts

Re: Git archive checksums may change

#171
I can't fathom how no one internally at Microsoft-Github realized how widespread the breakage would be before rolling this out to all public users.

Surely, Microsoft-Github's own internal builds would have started failing as a result of this change? Or do they not even canary releases internally at all?

Re: Git archive checksums may change

#172
post #71

I didn't even know I should be depending on compression, file ordering, created-at file metadata, etc. being stable when pressing 'download repository as zip' (if I understand correctly what this is about, since the article doesn't really say). Perhaps it could be stable due to caching for a while after you first press it, but when it gets re-generated? I'm very surprised this was reproducible to begin with, given ho…

I once had a small issue with a deployment at work because of ordering issues within a zip file. That order is important with Spring since that determines which classes are initialized first.

Re: Git archive checksums may change

#173
Forever problem 0:

Tar/zipball archives on the same ref never have a stable hash.

Forever problem 1:

No sha256/512/3 hashes of said tar/zipballs.

Forever problem 2:

No metalinks for those.

Forever problem 3:

Not IPv6. Some of our network is IPv6 only.

Forever problem 4:

Hitting secondary rate limiting because I can browse fast.

Re: Git archive checksums may change

#174
post #71

I didn't even know I should be depending on compression, file ordering, created-at file metadata, etc. being stable when pressing 'download repository as zip' (if I understand correctly what this is about, since the article doesn't really say). Perhaps it could be stable due to caching for a while after you first press it, but when it gets re-generated? I'm very surprised this was reproducible to begin with, given ho…

Many tools set mtime to zero to avoid checksum drift

Re: Git archive checksums may change

#175
post #113

Earlier quoted context omitted.

It’s generally a bad idea to process (extract) a tarball of unknown provenance. Verifying the tarball is from a known source beforehand mitigates the risk of, say, a malicious tarball that exploits a tar or gzip 0‐day.

But then that's the role of the httpS query with which you will fetch your data. And if you don't trust your http layer and/or Github's certificate, then you should not trust their archive anyway.

The certificate guarantees the source of the file, not the trust you should put in its contents. I can upload malware as a github project release file and https doesn't change that you shouldn't download/run it.

For software distribution this actually sometimes goes the other way - debian/ubuntu uses http (no s) for their packages, because the content itself is signed by the distribution and this way you can easily cache it at multiple levels.

Re: Git archive checksums may change

#176

Earlier quoted context omitted.

Woof. At the rate packages get updated these days, and the amount of dependencies between them, that just isn't sustainable for any reasonably-sized project in server and -- especially -- frontend land.

Exactly. Unless the package manager has a mechanism for doing that, good fucking luck updating any of your packages ever again.

It is implemented pretty well in a few languages. For ruby for example it's almost trivial to maintain a `vendor` directory that matches the current `Gemfile` and `Gemfile.lock`. The size changes without LFS mean that's a bad idea, but... you can do it.

Re: Git archive checksums may change

#177

Earlier quoted context omitted.

Fact of life: the vast majority of your users do not read your documentation (or do not do so carefully enough that what you put in your docs is an ironclad proof that all users adhere to). That's literally what Hyrum's law is about. Of course, you can choose to do whatever you want. It's valuable to recognize of course that you're trading off good will from your users with whatever technical improvement is getting m…

Thats support you could expect if you paid for it.

Aint nobody give a shit about you if you aren't bringing five or six figures as customer. Nobody is stopping rewrite that happened to break undocumented stuff you relied on if you $10/mo.

This case is different as breakage probably affected github/microsoft themselves

Re: Git archive checksums may change

#178
post #123

Earlier quoted context omitted.

Thanks for the quick rollback. I want to encourage you to think about locking in the current archive details, at least for archives that have already been served. Verifying that downloaded archives have the expected checksum is a critical best practice for software supply chain security. Training people to ignore checksum changes is training them to ignore attacks. GitHub is a strong leader in other parts of supply c…

I would also appreciate stronger advertising of the ability to turn a Git tag into a GitHub release and upload stable source code files to it. Maybe even a button in the GitHub releases interface to “generate source tarball and attach as stable tarball to this release.”

I agree this would be great. However, it should also stop you from providing useless tarballs (as `/archive/` does today) if:

- you use autoconf (or any other tool(s) that require generating code into the source archive; or - you have submodules (to which `git archive` is completely blind).

Note that `git-archive-all`[1] can help as long as your submodules don't do things like `[attr]custom-attr` in their `.gitattributes` as it is only allowed in the top-level `.gitattributes` file and cannot be added to the tree otherwise.

[1]https://github.com/roehling/git-archive-all

Re: Git archive checksums may change

#179
post #67

Earlier quoted context omitted.

If you read the commit message you would see that it is up drop a third party dependency.

Yeah, read that. Just don't understand, if git already had an internal gzip implementation, why wasn't it used since it was added?

Because not everyone refactors whole codebase after adding one function that might be useful somewhere else.

I'd imagine motivation for this change in particular is multiplatform use, not every platform just have gzip in path.

Re: Git archive checksums may change

#180

Earlier quoted context omitted.

I wouldn't check the hash of the compressed archive, but of the actual files themselves. It's a bit more metadata, but it's also a lot more robust, and allows you to detect changes after unpacking as well.

It’s generally a bad idea to process (extract) a tarball of unknown provenance. Verifying the tarball is from a known source beforehand mitigates the risk of, say, a malicious tarball that exploits a tar or gzip 0‐day.

Or just contains 100TB of zeroes
Post reply on HN