Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Dependencies should be fetched directly from VCS

arp242.net

Post reply on HN
Print view View original

31–40 of 46 posts

Re: Dependencies should be fetched directly from VCS

#31

Post by cygx »

g-b-r wrote: post #29
cygx wrote: post #27

Earlier quoted context omitted.

it's far from a given that the tools (e.g. browsers) used to read it are safe from malicious documentation files Generated HTML files are potentially easier to audit than the scripts/toolchains used to generate them on an end user's machine if you do not pre-generate them. Off the top of my head, other things I've done is committing RELAX NG *.rnc files, but shipping *.rng files, or generating C header files for vari…

Hmm, potentially, the problem is that hardly anyone ever audits packages, while at least someone occasionally gives a look at a repository history Can't the things you list be part of the build scripts?

Sure - but running build scripts on an end-user's machine requires the user to have all relevant tools installed, and isn't exactly reducing the attack surface...

I still like the idea of shipping tarballs that include generated files instead of pulling input files from source control. As mentioned, the first thing that came to mind to make things easier to audit is to stick their contents into a community-controlled VCS.

Re: Dependencies should be fetched directly from VCS

#32

Post by Charon77 »

cygx wrote: post #15

Not sure this is the right solution for 2 reasons: First, I'm uncomfortable with making a package creator's VCS provider part of the language's module infrastructure. Second, not all files under version control necessarily belong in a tarball, and not all files in a tarball necessarily need to be under version control. However, the point that there should be an easy way to track changes in your dependencies is well t…

Archlinux has this separation of official packages and the user submitted AUR, which has never been something that is officially supported by design.

Having big names maintain their own VCS is a good thing. There can be a centralized system to index / search packages just for findability. And this centralized system doesn't have to be anything special just like how anyone could use any search engine

Re: Dependencies should be fetched directly from VCS

#33

Post by mook »

inigyou wrote: post #18
kibwen wrote: post #5

Maybe everyone else is too young to remember left-pad, but in the wake of left-pad everyone learned that one of the primary selling points of dedicated dependency repositories is that they can refuse to support "unpublishing" a dependency, which is not a guarantee that Github (or any other popular forge) makes.

There's no such thing as refusing to unpublish, not if you want to avoid prison time. Package contains child porn, what now?

I think it's implied to be refusing to unpublish at the request of the author; unpublishing due to outside forces is probably inevitable. If some unsupportable content got introduced in a new version (e.g. xz), at least the historical versions would be kept.

That of course brings in questions of cost, trust, and governance, so there's definitely a trade-off…

Re: Dependencies should be fetched directly from VCS

#34

Post by g-b-r »

cygx wrote: post #31
g-b-r wrote: post #29

Earlier quoted context omitted.

Hmm, potentially, the problem is that hardly anyone ever audits packages, while at least someone occasionally gives a look at a repository history Can't the things you list be part of the build scripts?

Sure - but running build scripts on an end-user's machine requires the user to have all relevant tools installed, and isn't exactly reducing the attack surface... I still like the idea of shipping tarballs that include generated files instead of pulling input files from source control. As mentioned, the first thing that came to mind to make things easier to audit is to stick their contents into a community-controlled…

> Sure - but running build scripts on an end-user's machine requires the user to have all relevant tools installed, and isn't exactly reducing the attack surface...

Well, that's what building from source means; you could maybe distribute some compiled files for those who prefer them, and are willing to take the risk.

But you seem to be arguing that having users compile their software themselves doesn't increase their security?

> As mentioned, the first thing that came to mind to make things easier to audit is to stick their contents into a community-controlled VCS.

True

Re: Dependencies should be fetched directly from VCS

#35

Post by rendaw »

Cedricgc wrote: post #19

IIRC Go had to deal with a bunch of issue related to their initial source based package management system. The URL imports remain, but I believe fetches are served through the pkg.go.dev proxy now. Andrew Nesbitt has a good write [0] up on why using Git as a database is a bad idea [0] https://nesbitt.io/2025/12/24/package-managers-keep-using-gi...

Most of those deal with downloading large centralized indexes using git, not individual packages.

> The problem was that go get needed to fetch each dependency’s source code just to read its go.mod file and resolve transitive dependencies. Cloning entire repositories to get a single file.

Then it needs to compile the module, right? It's not downloading it twice.

> Grab’s engineering team went from 18 minutes for go get to 12 seconds after deploying a module proxy.

> ...

> GOPROXY became the default in Go 1.13.

The way this is written it sounds like go builds were extremely slow (18 minutes) so go adopted a proxy to speed things up. But Go 1.13 was released in 2020, and the blog post about 18 minutes is from 2023. There's no link to anything about GOPROXY.

The vcpkg example also isn't a git issue, it sounds like just bad engineering from Microsoft.

I'm willing to believe there are issues with using git, but that article is a letdown.

Re: Dependencies should be fetched directly from VCS

#36

Post by rendaw »

rendaw wrote: post #35
Cedricgc wrote: post #19

IIRC Go had to deal with a bunch of issue related to their initial source based package management system. The URL imports remain, but I believe fetches are served through the pkg.go.dev proxy now. Andrew Nesbitt has a good write [0] up on why using Git as a database is a bad idea [0] https://nesbitt.io/2025/12/24/package-managers-keep-using-gi...

Most of those deal with downloading large centralized indexes using git, not individual packages. > The problem was that go get needed to fetch each dependency’s source code just to read its go.mod file and resolve transitive dependencies. Cloning entire repositories to get a single file. Then it needs to compile the module, right? It's not downloading it twice. > Grab’s engineering team went from 18 minutes for go g…

https://go.dev/blog/module-mirror-launch seems like a better source.

It seems to imply the reasons were optimization and to protect from disappearing dependencies. It doesn't imply that there were any hidden dangers to git.

I think it's very possible that google simply thinks speed is more important than decentralization. And for google's purposes, a google-owned cache is probably a safe enough hedge against disappearing repos (read: if you're not google you'd need to evaluate this risk yourself).

Edit: Discussion from the time here: https://news.ycombinator.com/item?id=20837753 - in that thread too people don't really know why the proxy was needed (or like, "is it really just for speed? but it isn't particularly slow right now"). One additional reason is analytics collection.

Re: Dependencies should be fetched directly from VCS

#37

Post by SkiFire13 »

jchw wrote: post #25
throwaway27448 wrote: post #21

Earlier quoted context omitted.

All package managers I've used allow specifying where the package is pulled from. What am I missing? What ecosystems are you comparing go to?

Sure, you can tell your other package managers to pull a module from VCS, usually with some limitations and negative side effects. The Go module syste is not the same model. You can, for example, run an NPM proxy that caches locally, but what you're missing is that with Go, the proxy part is the centralized part. There is no NPM. There is no crates.io, no PyPI, no RubyGems, no Packigist, no PECL, no CPAN. The proxy's…

> The source of truth remains decentralized, in the individual VCSes.

That's not entirely true, if the VCS's tag changes the proxy might not pick it up.

Re: Dependencies should be fetched directly from VCS

#38

Post by SkiFire13 »

jchw wrote: post #28
zer00eyz wrote: post #23

Earlier quoted context omitted.

> When you install Go, it does just that, to some approximation; it will try to fetch from the module proxy first and fall back after that fails. Go is telling you that your VCS has a deficiency, without saying it out loud. That proxy should be your own repo... but git sub modules, sub trees, sub directories are non starters for 99 percent of cases... Git is an amazing tool, and if you want to manage your development…

To be honest with you, I don't know what you are suggesting. Piper is a more scalable Perforce clone. That's cool and all, but I don't see how it solves the problem that hosts can be unreliable and that data fetched from the internet can be changed later. Doesn't seem to me like there's anything a VCS itself can do to change this. You maybe could invent a fully decentralized peer-to-peer VCS hosting system that uses…

> and that data fetched from the internet can be changed later

So do you want the VCS to be the source of truth or not?

Re: Dependencies should be fetched directly from VCS

#39

Post by throwaway27448 »

jchw wrote: post #25
throwaway27448 wrote: post #21

Earlier quoted context omitted.

All package managers I've used allow specifying where the package is pulled from. What am I missing? What ecosystems are you comparing go to?

Sure, you can tell your other package managers to pull a module from VCS, usually with some limitations and negative side effects. The Go module syste is not the same model. You can, for example, run an NPM proxy that caches locally, but what you're missing is that with Go, the proxy part is the centralized part. There is no NPM. There is no crates.io, no PyPI, no RubyGems, no Packigist, no PECL, no CPAN. The proxy's…

> There is no NPM.

any package manager can be redirected towards particular sources. What is unique about go? I sense, though this is not articulated through your communication, that the source of the package is directly specified in the source code requiring the package. Is that what you're trying to say—source pinning is evident in the source?

Re: Dependencies should be fetched directly from VCS

#40

Post by cygx »

g-b-r wrote: post #34
cygx wrote: post #31

Earlier quoted context omitted.

Sure - but running build scripts on an end-user's machine requires the user to have all relevant tools installed, and isn't exactly reducing the attack surface... I still like the idea of shipping tarballs that include generated files instead of pulling input files from source control. As mentioned, the first thing that came to mind to make things easier to audit is to stick their contents into a community-controlled…

> Sure - but running build scripts on an end-user's machine requires the user to have all relevant tools installed, and isn't exactly reducing the attack surface... Well, that's what building from source means; you could maybe distribute some compiled files for those who prefer them, and are willing to take the risk. But you seem to be arguing that having users compile their software themselves doesn't increase their…

But you seem to be arguing that having users compile their software themselves doesn't increase their security?

On the contrary! I gave examples where source files get generated. This happens in one of two places: Either when the maintainer publishes a new version, or every time an end user builds the package.

I'm arguing for the former, you're arguing for the latter. There are pros and cons to either approach. Some cons for the latter:

First, the build process becomes more brittle, as every end user now has to install the necessary tools. In case of the project containing RELAX NG schemas I alluded to, this would require a recent version of Perl, a Java runtime, and the Trang utility written in Java. The alternative? Just shipping a single XML file.

Second, the build process often doesn't become more, but less auditable: Instead of just reviewing the source files that actually get compiled, you now have to track down how they get generated, and review all the scripts that do so.

Post reply on HN