Live data from Hacker News

Dependencies should be fetched directly from VCS

arp242.net

21–30 of 46 posts

Re: Dependencies should be fetched directly from VCS

#21
post #13

People keep replying seeming to miss what makes Go's solution so good. I think it deserves more attention. In many cases I think it's impossible to avoid the pain that comes with decentralization without defeating the point. It's challenging because certain things are hard to do decentralized so it's easier to just give up and rely on central authorities and components. When you install Go, it does just that, to some…

All package managers I've used allow specifying where the package is pulled from. What am I missing? What ecosystems are you comparing go to?

Re: Dependencies should be fetched directly from VCS

#22
post #20
post #15

Not sure this is the right solution for 2 reasons: First, I'm uncomfortable with making a package creator's VCS provider part of the language's module infrastructure. Second, not all files under version control necessarily belong in a tarball, and not all files in a tarball necessarily need to be under version control. However, the point that there should be an easy way to track changes in your dependencies is well t…

> First, I'm uncomfortable with making a package creator's VCS provider part of the language's module infrastructure. Why can't you store a (possibly shallow) clone of the repository in the "package" registry? > Second, not all files under version control necessarily belong in a tarball How likely is it that files under version control but "not belonging in a tarball" (not needed to compile the current version) take…

Why can't you store a (possibly shallow) clone of the repository in the "package" registry?

Yes, that would be an option.

As to your question, you generally want to version control input files, but distribute the generated artifacts. Some possible scenarios would be documentation, data tables in source or binary form, generated code including maybe even configure scripts if you cannot avoid it (though if I had to 'vendor' 3rdparty code that uses autotools, I think I probably would just unpack a tarball and commit the relevant artifacts).

Re: Dependencies should be fetched directly from VCS

#23
post #13

People keep replying seeming to miss what makes Go's solution so good. I think it deserves more attention. In many cases I think it's impossible to avoid the pain that comes with decentralization without defeating the point. It's challenging because certain things are hard to do decentralized so it's easier to just give up and rely on central authorities and components. When you install Go, it does just that, to some…

> When you install Go, it does just that, to some approximation; it will try to fetch from the module proxy first and fall back after that fails.

Go is telling you that your VCS has a deficiency, without saying it out loud.

That proxy should be your own repo... but git sub modules, sub trees, sub directories are non starters for 99 percent of cases...

Git is an amazing tool, and if you want to manage your development like the linux kernel its dam near perfect. Most orgs dont work that way, and the tools and machinations that we have built around these shortcomings are rather burdensome.

I long for google to productize piper for the rest of us.

Re: Dependencies should be fetched directly from VCS

#25
post #13

People keep replying seeming to miss what makes Go's solution so good. I think it deserves more attention. In many cases I think it's impossible to avoid the pain that comes with decentralization without defeating the point. It's challenging because certain things are hard to do decentralized so it's easier to just give up and rely on central authorities and components. When you install Go, it does just that, to some…

All package managers I've used allow specifying where the package is pulled from. What am I missing? What ecosystems are you comparing go to?

Sure, you can tell your other package managers to pull a module from VCS, usually with some limitations and negative side effects. The Go module syste is not the same model. You can, for example, run an NPM proxy that caches locally, but what you're missing is that with Go, the proxy part is the centralized part. There is no NPM. There is no crates.io, no PyPI, no RubyGems, no Packigist, no PECL, no CPAN. The proxy's entire job is just to stand in for requests that would be made to individual VCSes using their original source repositories. The sumdb's entire job is to ensure that whether proxied or not, the source code you get isn't tampered with. The source of truth remains decentralized, in the individual VCSes.

With NPM, uv, composer, Cargo, etc. you can indeed choose to make your dependencies pull from VCS, but:

- Not having universal module proxies with caching means you will run into the usual problems. Repos can disappear, history can be rewritten, hosts can go down. Even if you invented an equivalent caching service, without making it an ecosystem default it won't achieve the same effectiveness as Go because less would be in the cache.

- Because it isn't an ecosystem standard, dependencies you pull would then still reference dependencies from centralized repos anyways, more or less defeating the point. You'd have to go and override every recursive dependency to really make it decentralized.

(And again, it usually comes with other downsides depending on the specific ecosystem. Performance is a big one, the module cache definitely helps make fetching dependencies in Go faster.)

Not having centralized package repos comes with its ups and downs, but the lack of one existing means there are a limited number of interesting attacks you can really pull in the Go ecosystem and they are heavily tamper-evident. You can try to poison the module cache with a given revision for a dependency with malicious code then overwrite it with an inoccuous commit, but this will probably be detected when someone (possibly in Nixpkgs or another packaging system) runs GOPROXY=direct and the sumdb doesn't line up. So the least noticeable thing you can really do is just leave your malicious code in the repo. Go also has the kind of nice property that fetching and building modules is supposed to be safe with untrusted code. That's pretty good, it would be hard to do better than that.

It also means there really isn't any central namespace to poison. You can still trick someone into downloading and using malicious software but you're mostly limited to doing it the old fashioned way. Again, hard to do much better than that.

While everyone has been losing their minds over supply chain issues, the Go ecosystem has been concerned but less so. It's still a huge risk still, dependencies can get compromised still, but attackers have no real central target to go after and get a bunch of damage at once. Nothing realistical, anyway. So, it's back to just trying to pwn individual GitHub repos.

Re: Dependencies should be fetched directly from VCS

#26
post #22
post #20

Earlier quoted context omitted.

> First, I'm uncomfortable with making a package creator's VCS provider part of the language's module infrastructure. Why can't you store a (possibly shallow) clone of the repository in the "package" registry? > Second, not all files under version control necessarily belong in a tarball How likely is it that files under version control but "not belonging in a tarball" (not needed to compile the current version) take…

Why can't you store a (possibly shallow) clone of the repository in the "package" registry? Yes, that would be an option. As to your question, you generally want to version control input files, but distribute the generated artifacts. Some possible scenarios would be documentation, data tables in source or binary form, generated code including maybe even configure scripts if you cannot avoid it (though if I had to 've…

If you distribute generated artifacts you're not really distributing the source (and thus xz can happen)

Even for documentation, you'd think it would be harmless to generate it, but it's far from a given that the tools (e.g. browsers) used to read it are safe from malicious documentation files

Re: Dependencies should be fetched directly from VCS

#27
post #26
post #22

Earlier quoted context omitted.

Why can't you store a (possibly shallow) clone of the repository in the "package" registry? Yes, that would be an option. As to your question, you generally want to version control input files, but distribute the generated artifacts. Some possible scenarios would be documentation, data tables in source or binary form, generated code including maybe even configure scripts if you cannot avoid it (though if I had to 've…

If you distribute generated artifacts you're not really distributing the source (and thus xz can happen) Even for documentation, you'd think it would be harmless to generate it, but it's far from a given that the tools (e.g. browsers) used to read it are safe from malicious documentation files

it's far from a given that the tools (e.g. browsers) used to read it are safe from malicious documentation files

Generated HTML files are potentially easier to audit than the scripts/toolchains used to generate them on an end user's machine if you do not pre-generate them.

Off the top of my head, other things I've done is committing RELAX NG *.rnc files, but shipping *.rng files, or generating C header files for various types of data (think `xxd -i` in case of binary files, but also just large chunks of plain text that gets wrapped into a C string).

Re: Dependencies should be fetched directly from VCS

#28
post #13

People keep replying seeming to miss what makes Go's solution so good. I think it deserves more attention. In many cases I think it's impossible to avoid the pain that comes with decentralization without defeating the point. It's challenging because certain things are hard to do decentralized so it's easier to just give up and rely on central authorities and components. When you install Go, it does just that, to some…

> When you install Go, it does just that, to some approximation; it will try to fetch from the module proxy first and fall back after that fails. Go is telling you that your VCS has a deficiency, without saying it out loud. That proxy should be your own repo... but git sub modules, sub trees, sub directories are non starters for 99 percent of cases... Git is an amazing tool, and if you want to manage your development…

To be honest with you, I don't know what you are suggesting. Piper is a more scalable Perforce clone. That's cool and all, but I don't see how it solves the problem that hosts can be unreliable and that data fetched from the internet can be changed later. Doesn't seem to me like there's anything a VCS itself can do to change this.

You maybe could invent a fully decentralized peer-to-peer VCS hosting system that uses Blockchain(TM) technology to ensure integrity, or something. But... You'd probably still want a cache, perhaps even moreso than before, for performance reasons, even if you really truly did solve the availability and integrity problems.

Re: Dependencies should be fetched directly from VCS

#29
post #27
post #26

Earlier quoted context omitted.

If you distribute generated artifacts you're not really distributing the source (and thus xz can happen) Even for documentation, you'd think it would be harmless to generate it, but it's far from a given that the tools (e.g. browsers) used to read it are safe from malicious documentation files

it's far from a given that the tools (e.g. browsers) used to read it are safe from malicious documentation files Generated HTML files are potentially easier to audit than the scripts/toolchains used to generate them on an end user's machine if you do not pre-generate them. Off the top of my head, other things I've done is committing RELAX NG *.rnc files, but shipping *.rng files, or generating C header files for vari…

Hmm, potentially, the problem is that hardly anyone ever audits packages, while at least someone occasionally gives a look at a repository history

Can't the things you list be part of the build scripts?

Re: Dependencies should be fetched directly from VCS

#30
post #5

Maybe everyone else is too young to remember left-pad, but in the wake of left-pad everyone learned that one of the primary selling points of dedicated dependency repositories is that they can refuse to support "unpublishing" a dependency, which is not a guarantee that Github (or any other popular forge) makes.

> Maybe everyone else is too young to remember left-pad It wasn't that long ago...

In the AI age, 6 months seems to be percieved as a long time ago. 6 months feels like yesterday to me.
Post reply on HN