Live data from Hacker News

Dependencies should be fetched directly from VCS

arp242.net

41–46 of 46 posts

Re: Dependencies should be fetched directly from VCS

#41
post #25

Earlier quoted context omitted.

Sure, you can tell your other package managers to pull a module from VCS, usually with some limitations and negative side effects. The Go module syste is not the same model. You can, for example, run an NPM proxy that caches locally, but what you're missing is that with Go, the proxy part is the centralized part. There is no NPM. There is no crates.io, no PyPI, no RubyGems, no Packigist, no PECL, no CPAN. The proxy's…

> There is no NPM. any package manager can be redirected towards particular sources. What is unique about go? I sense , though this is not articulated through your communication, that the source of the package is directly specified in the source code requiring the package. Is that what you're trying to say—source pinning is evident in the source?

What's unique about Go is the lack of a centralized package repo. You are saying "I could point NPM to another registry". I am saying, Go doesn't have a registry and you can't really point it to one. I hope it is apparent that these are two extremely different statements. The fact that it is pinning source to source is just the approach that they take to get there.

And again, if you just bolted a solution onto the side of an existing ecosystem it wouldn't accomplish as much. "Technically this could be done elsewhere" is true, but I hope you know I know that. The point is that Go deployed a mostly decentralized package management solution to the entire ecosystem as the primary and essentially only way to manage packages with Go. You actually could approximate what Go does using NPM, sans the caching module proxy part, using VCS URLs as dependencies, but the "approximate" and "sans caching module proxy part" is doing some heavy lifting.

There isn't really anything that does what Go does exactly. The closest example to me is OCI images. Like Go modules, their identity is essentially a sort of URI. However, it isn't a language package manager and has fairly different challenges, and doesn't really seem to need the caching proxy stuff due to the relatively few registries (and running it would be very expensive due to image sizes.) So I stand by with my belief that what Go does here is pretty unique.

Re: Dependencies should be fetched directly from VCS

#42
post #9

Earlier quoted context omitted.

if you mean nix the state declaration, it is a ilusion. when you have packge for debian 12... you just install it. when you have debian 13, you need the package for debian 13 and there's no way around it. nix lies that you don't have to worry about all that. which is only as true as is true with packages. if you can replace -12 with -13 in a non nix setup, your nix "package x" will still work. otherwise you will have…

I’m super interested in your comment about nix but am not following your example. Would you mind elaborating why (presumably) nixos (since you gave a Debian example), doesn’t help with this.

it is a distro. it packages code from upstream. it have the same problems as other distros when it came from depencies and managing breaking changes.

Re: Dependencies should be fetched directly from VCS

#43
post #28

Earlier quoted context omitted.

To be honest with you, I don't know what you are suggesting. Piper is a more scalable Perforce clone. That's cool and all, but I don't see how it solves the problem that hosts can be unreliable and that data fetched from the internet can be changed later. Doesn't seem to me like there's anything a VCS itself can do to change this. You maybe could invent a fully decentralized peer-to-peer VCS hosting system that uses…

> and that data fetched from the internet can be changed later So do you want the VCS to be the source of truth or not?

First of all, can you just simply not speak in gotchas? I don't even like engaging with this sort of thing. It's a valid question to ask how VCS is the source of truth in this case, but the way you're responding suggests you're not interested in discussing it productively, just a shit-flinging where we argue semantics about what it means for something to be a source of truth, without regards to what the actual important implications are supposed to be. I'm going to try to interpret this in good faith but I feel like I'm probably wasting my time doing so.

Yes, the VCS is still the source of truth, ultimately nothing in the cache or sumdb exists without having existed in the VCS source first. The sumdb is an integrity mechanism bolted on top, and then the module proxy cache is an availability mechanism bolted on top.

Consider the following: let's say Kubernetes depends on some library you have Git access to. So you, being evil, rewrite the tag Kubernetes currently uses to contain malware. There are two possible scenarios:

- By default: the user is using a module cache, so changing a tag that already existed is no different than a Git host being pulled offline from the perspective of the user. The cache continues to serve the original source code, no malware delivered.

- The decentralized case: a user disables both sumdb and the module cache. Guess what? The pull fails. The pull fails because the project has a go.sum file and your tag doesn't match it.

Let's say someone compromised Google's sumdb or module proxy cache. Same basic thing: the go get will fail, because of the local integrity checks.

See, it really changes nothing. The module proxy cache essentially just ensures that a module source code that was once available remains available indefinitely so that you can continue to build old software exactly and securely.

The only place where behavior ever changes meaningfully (as in, not just "available or unavailable") is when go getting a new package or package update. If you have GOPROXY=direct, the poisoned fetch will fail because of global sumdb. If you disable that too, it may still fail because of local sumdb. But if not, you may get someone to pull your mutated tag. (And then the resulting go.sum would fail for other people.)

If you knew for absolute certain that nobody had ever pulled the package before, then theoretically mutating an existing tag would be safe. However, I call shenanigans on this: if you knew for sure nobody had ever pulled it, how in the world did it end up in the sumdb and module proxy cache? Granted, it may have been because of your own CI or something, but still, it stands to reason that this mechanism is introducing a highly desirable property. If a given tag was available once with a given source code I would want to be able to pull it again later and know it isn't tampered with.

In general, I have never seen it interpreted that the possibility of a stale cache changes what the canonical source of truth is. I've also never seen anyone argue that a lockfile changes the canonical source of truth. The Go sumdb being a global integrity mechanism does make it fairly unique, but I don't see it as being categorically different than the integrity checks from lockfiles, it's just that but across multiple users.

Even if you don't like Go's implementation particularly, I still think it is highly worth copying. You can always change a few things. Like for example, you could make module fetch operations not backed by a local checksum skip the global module proxy cache as a safeguard against the proxy itself, only allowing it to be trusted once you already have evidence that the sumdb and canonical source currently agree by virtue of already having a sum.

Sorry I have dropped another text wall, I don't have time to make it shorter. It will happen again.

(I realize that I have ranted in two directions here, but my intention was to 1. Argue that the properties introduced by the design are actually important and desirable, 2. Then argue semantics about whether it changes what one would consider to be the canonical source of truth, second mainly because I consider this to be unimportant. I mean it could be discussed further but, there really should be a point to it other than just semantics.)

Re: Dependencies should be fetched directly from VCS

#44
post #25

Earlier quoted context omitted.

Sure, you can tell your other package managers to pull a module from VCS, usually with some limitations and negative side effects. The Go module syste is not the same model. You can, for example, run an NPM proxy that caches locally, but what you're missing is that with Go, the proxy part is the centralized part. There is no NPM. There is no crates.io, no PyPI, no RubyGems, no Packigist, no PECL, no CPAN. The proxy's…

> The source of truth remains decentralized, in the individual VCSes. That's not entirely true, if the VCS's tag changes the proxy might not pick it up.

Changed tags are not allowed in Go (though you can work around it if you really want to boil yourself). All references are hashed so any attempt to alter a published package version will get blocked.

Go has a decentralized append only auditable hashsum db protocol.

Re: Dependencies should be fetched directly from VCS

#45
post #17
post #10

Not sure relying on a bunch of various VCS to stay online is necessarily a great approach either. I think go is also a little more amenable to source library distribution since there's a pretty broad pure go ecosystem. For interpreted languages, a lot of performance sensitive stuff tends to be offloaded to arbitrary compiled languages so you end up needing a bunch of different toolchains to get everything working. A…

2nd paragraph of blogpost: > A hash of all files is checked against known hashes on sum.golang.org to prevent tags from being replaced, and it uses a proxy to prevent repos from being left-pad’d. there is no need for 3rd party hosts to stay online. It's the best combination of de-centralized and centralized approaches. Re pre-buiild packages: Python makes a difference between source and binary wheels. This approach i…

Proxies are still 3rd party hosts.

I've ran into issues in my very limited go experience with them missing things and having hash mismatches (maybe someone changed a tag and it was working as intended? I don't remember the exact chain of events but I ended up with a hash that could no longer be found anywhere)

Re: Dependencies should be fetched directly from VCS

#46
This makes the problem harder instead of easier. The most practical conclusion is "the package registry is the VCS, but with stricter rules".

Those stricter rules are useful, the registry acts as a authority to ensure no version string is ever reused in a way that would make it non-canonical. It also ensures availability of the source code, so your software won't suddenly fail to compile because somebody decided to delete the repository from their server, delete their Github/codeberg user, or they can't pay for their domain anymore (which is a fail-open identity system, not a fail-close one like crates.io). Sure there's also the risk of crates.io ceasing to exist, but that's less likely than "some software engineer gets layed off, decides tech is a dead-end and becomes a domain-less goose farmer".

Post reply on HN