Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Git archive checksums may change

github.blog

Post reply on HN
Print view View original

191–200 of 255 posts

Re: Git archive checksums may change

#191

Post by pabs3 »

bentley wrote: post #121
JoshTriplett wrote: post #112

Earlier quoted context omitted.

> many upstreams choose to make a tag and let people download the autogenerated tarballs Which is fine as long as you rely on the hash of the tag rather than the hash of the tarball. > despite the fact that they won’t contain preprocessed autoconf This is a feature; run `autoreconf -vfi` at build time, so that you don't depend on the maintainer's idiosyncratic autotools setup and local macros, and so that you can rel…

> This is a feature; run `autoreconf -vfi` at build time, so that you don't depend on the maintainer's idiosyncratic autotools setup and local macros, and so that you can reliably regenerate it all if you want to change configure.ac or Makefile.am. On a package bulk build machine, that’s a lot (like, a lot ) of wasted CPU cycles multiplied by the thousands of packages that use autoconf. For the majority of packages t…

Having autocruft in the default tarball is a design flaw in autotools, you should never ship prebuilt files in git nor in source tarballs. I think `make distcheck` should by default put the autotools files into a separate foo-1.2.3.4-2023-01-31-autocruft.tar alongside the real source tarball generated by git-archive.

Re: Git archive checksums may change

#192

Post by pabs3 »

bentley wrote: post #121
JoshTriplett wrote: post #112

Earlier quoted context omitted.

> many upstreams choose to make a tag and let people download the autogenerated tarballs Which is fine as long as you rely on the hash of the tag rather than the hash of the tarball. > despite the fact that they won’t contain preprocessed autoconf This is a feature; run `autoreconf -vfi` at build time, so that you don't depend on the maintainer's idiosyncratic autotools setup and local macros, and so that you can rel…

> This is a feature; run `autoreconf -vfi` at build time, so that you don't depend on the maintainer's idiosyncratic autotools setup and local macros, and so that you can reliably regenerate it all if you want to change configure.ac or Makefile.am. On a package bulk build machine, that’s a lot (like, a lot ) of wasted CPU cycles multiplied by the thousands of packages that use autoconf. For the majority of packages t…

The distros always run `autoreconf` these days, so that they can verify that they can still build the build system from the source configure.ac/Makefile.am files.

Re: Git archive checksums may change

#193

Post by bentley »

shakow wrote: post #152
bentley wrote: post #143

Earlier quoted context omitted.

Package builders work on the scale of thousands of packages. The increased time and CPU usage multiplies greatly. “Complicated” is indisputable. Cloning a repository is absolutely complicated. Fetching a single file over HTTPS is as simple as it gets, these days.

And you really believe that downloading & extracting a source .tar.gz and compiling it will have a run time much shorter than 0.4s? Just executing the ./configure will take more than that.

> And you really believe …

Huh? What I fully believe is that downloading a source tarball over HTTPS, verifying its checksum, and extracting it will take less time than cloning the repository from Git, then verifying the checksum of all files—which you said would take 29 seconds plus 0.4s.

Re: Git archive checksums may change

#194

Post by hobofan »

vlovich123 wrote: post #17

Hyrum's Law strikes again. It kind of doesn't matter what you document. If you weren't randomizing your checksum previously [1], you can't just spring this on the community and blame it for the fallout. I'm more shocked that there's resistance from the GitHub team saying "but we documented this isn't stable". Default stance for the team should be rollback & reevaluate an alternate path forward when the scope is this…

This isn't even a case of "we didn't documented this".

I know that the Bazel team reached out to GitHub in the past to get a confirmation that this behaviour could be relied on, and only after that was confirmed did they set that as recommendation across their ecosystem.

Re: Git archive checksums may change

#195

Post by vlovich123 »

ilyt wrote: post #182
vlovich123 wrote: post #22

Earlier quoted context omitted.

The two main problems are: A) How do you catch tarballs that have extra files injected that aren't part of your manifest B) What does the performance of this look like? Certainly for traditional HDDs this is going to kill performance, but even for SSDs I think verifying a bunch of small files is going to be less efficient than verifying the tarball.

Well, the simplest way would be to make checksum after decompression , that doesn't need per file verify and relies on files being put in same order into tar file. The other method would be having Manifest file with checksum of every file inside the tar and compare that in-flight, could be simple "read from tar, compare to hash, write to disk" (with maybe some tmpfiles for the bigger ones)

It’s not just about the integrity of the files you’re processing, but also the integrity of the archive itself. If you extract the tarball from a random place, there’s a larger security risk. Now granted HTTPS probably mitigates a lot of it, but cert pinning isn’t that common so MITM attacks aren’t thaaat theoretical.

Re: Git archive checksums may change

#196

Post by metrognome »

rswail wrote: post #188
metrognome wrote: post #132

I wonder if this incident will encourage our industry to build more robust forms of artifact integrity verification, or if we will instead codify the status quo of "we guarantee repos to be archived deterministically." To me, the latter seems like a more troubling precedent.

This is being driven in industry by the push by US FedGov (via NIST) to have supply chain verification after the recent hacks. POTUS issued an EO and NIST have been following up, leading to the promotion of schemes such as spdx https://tools.spdx.org/app/about/ Where I work is also required to start documenting our supply chain as part of the (new, replacing PCI-DSS) PCI-SFF certification requirements, which requires…

Oh, I'm not arguing that using checksums, SHA for example, for integrity verification is a bad idea. That's what they're designed for, after all.

However, I do think it's a bad idea to enforce the content of compressed archives to be deterministic. tar has never specified an ordering of its contents. Compression algorithms are parameterized for time and space, so their output should not be deterministic either. Both of these principles apply to zip as well. But we now have a situation where we are depending on both the archive format and the compression algorithm to produce a deterministic output. If we expect archives to behave this way in general, we set a bad precedent for all sorts of systems, not just git and GitHub.

Re: Git archive checksums may change

#197

Post by yakubin »

rabexc23 wrote: post #163
yakubin wrote: post #148

Earlier quoted context omitted.

No. Just copy files into the repo. Any way you like. In a GUI, in a terminal — it doesn’t require a dedicated tool. Although cargo in Rust e.g. provides a dubcommand for it (cargo vendor). Alternatively you can host the tarballs somewhere you control in static storage — be it a static web server, object storage or whatever. How it’s done in Chromium: https://source.chromium.org/chromium/chromium/src/+/main:thi... >.

vendoring even with a tool has always worked poorly with me. Here are a few reasons: 1. You work in a company, you are in a team, you want some reasonable code review process in place. Now you want to check in a 3rd party dependency, "let's vendor it!" so you send out a PR with ... 10,000 - 100,000 lines of code. Your reviewer has no reasonable way to know if a) the dependency was downloaded from a reputable source,…

1. If you follow my link, you’ll see the process involves adding a README documenting git hashes, licence and changes made (I don't see what seems to you so arduous about it). You can checkout the git repo by hash and run a diff between the files added to the review and what you checked out. It’s also common practice to delete irrelevant files to make the review smaller.

2. Same as in 1.

3. It’s not an issue in my experience. A much bigger issue are large JSON files captured for snapshot testing or just big binary files. If your repo is so small that its deps are its majority, then it really shouldn’t take all that much time (or you use too many/too big deps, but I doubt you can beat Chromium which has Skia-sized deps).

> Compare to using a simple file with a 1) url, 2) secure hash, 3) list of patches to apply. Reviewing and ensuring correctness is trivial, upgrading is trivial, PRs are trivial.

Using a url doesn’t remove the need for reviewing the code of your dependencies. If you don’t, you’re essentially running “curl | sh” at scale. Checksums without code review don’t mean much.

Re: Git archive checksums may change

#198

Post by groestl »

slaymaker1907 wrote: post #172
lucb1e wrote: post #71

I didn't even know I should be depending on compression, file ordering, created-at file metadata, etc. being stable when pressing 'download repository as zip' (if I understand correctly what this is about, since the article doesn't really say). Perhaps it could be stable due to caching for a while after you first press it, but when it gets re-generated? I'm very surprised this was reproducible to begin with, given ho…

I once had a small issue with a deployment at work because of ordering issues within a zip file. That order is important with Spring since that determines which classes are initialized first.

One of the first things I check with every jvm packaging/deployment tool I investigate: does it preserve classpath ordering. Some offenders think -jar * is enough.

Re: Git archive checksums may change

#199

Post by katbyte »

naikrovek wrote: post #84
sneak wrote: post #23

Earlier quoted context omitted.

It's Microsoft. Just as the Apple of today is not the Apple of ten years ago, the GitHub today is not the GitHub of ten years ago. It's literally different people. The people who made the things you love have mostly moved on, and the brand is being run by different people with different values now. There's a little bit of an argument that such things are a bait-and-switch, but such is the nature of a large and multig…

The Microsoft of today isn't the Microsoft of 10 years ago, either, but that doesn't stop anyone from assuming that today's Microsoft is the same as the Microsoft of 10 years ago. the logic people use to blame Microsoft is intense, man. literally any logical leap is valid except one that absolves Microsoft of anything, no matter how small.

Trust is lost quickly and easily and earned back slowly with great difficulty

Re: Git archive checksums may change

#200

Post by throwawaylinux »

blueflow wrote: post #45
vlovich123 wrote: post #17

Hyrum's Law strikes again. It kind of doesn't matter what you document. If you weren't randomizing your checksum previously [1], you can't just spring this on the community and blame it for the fallout. I'm more shocked that there's resistance from the GitHub team saying "but we documented this isn't stable". Default stance for the team should be rollback & reevaluate an alternate path forward when the scope is this…

But look at it from the other side. Users that don't read your documentation and expect your software to work like they imagined are just a huge pain in the ass.

If your product supports some particular behavior, it will be used regardless of what you document.

Microsoft was once renown for bug-compatibility so as not to break their users. The new wave of movers and breakers would forget that wisdom at their peril.

Post reply on HN