Live data from Hacker News

An Analysis of vgo

sdboyer.io

11–20 of 69 posts

Re: An Analysis of vgo

#11
post #2

Could anyone distill this down for the casual reader? The author has a fairly laborious style and seems to be writing for an audience that is intimately familiar with the internals of dep and vgo.

The article is about vgo, a proposal (with working prototype) to build package management into Go itself. vgo bundles several improvements, a major one being the introduction of "modules", which are versioned collections of packages. Modules, among other features, will let us decouple the import path from the file system structure and source repository, making the awkward and much-maligned $GOPATH finally obsolete. v…

MVS has another quirk, which is that there are no version ranges. Only minimum versions.[1]

It relies completely on the community adopting "Semantic Import Versioning", which is basically "v2 is any breaking API change, which must live along-side v1 because it's actually a totally different import". So it blends some of the pros and cons of both dep/cargo-like package management with those of npm's.

It's... interesting. And I love that it's not an easy answer either way - it's a significant alternative, and it's bringing up lots of new(ish) ideas and approaches that have kinda fallen on the side. That said, I've had concerns about vgo from the beginning, primarily around the "does this make sense in an ecosystem". Package managers are not just "this algorithm is efficient", they have major code-culture effects (and the ecosystem can support or destroy the package manager's goals). This article is hitting most of my concerns squarely on the head, in quite a bit more detail than I've managed to figure out.

Also, https://medium.com/@sdboyer/so-you-want-to-write-a-package-m... is very good. sdboyer has been thinking about this stuff for quite a while - emotions may be there, but he has a LOT of experience backing it up.

[1]: To work off / echo pa7ch's comment here: it picks the highest minimum when there are conflicts in transitive dependencies. So if you use A (which needs C v1.1+) and B (which needs C v1.3+) you get C v1.3, even though v1.4 is available. You can upgrade to v1.4, but only by putting "C at v1.4" in your top-level dependencies list (which the tool automates for you) to override those values, and it's assumed safe because it's still importable, therefore still v1-compliant.

Re: An Analysis of vgo

#12
> If there are two algorithms that satisfy the same requirements, and only one is NP-complete, you pick the other one.

Problems can be NP-complete. Algorithms cannot - they can be exponential time (and if your choice is between exponential and double exponential, you pick the former!)

In practice, and especially for SAT solving, as far as I understand the state of the art is that the general case is NP-hard but we have tools that work surprisingly well in those cases that reality tends to produce. In some more academic cases, we even have heuristic solvers that are not even guaranteed to terminate, but usually do so within a couple of seconds, so you can get away with the "beginner's solution to the halting problem".

Re: An Analysis of vgo

#13
post #10

Earlier quoted context omitted.

The article is about vgo, a proposal (with working prototype) to build package management into Go itself. vgo bundles several improvements, a major one being the introduction of "modules", which are versioned collections of packages. Modules, among other features, will let us decouple the import path from the file system structure and source repository, making the awkward and much-maligned $GOPATH finally obsolete. v…

Minor correction: MVS requires modules to specify the minimum version to use of a module. The algorithm will then select the newest among those based on the idea that semver is forward compatible among minor versions. The name minimal version selection throws people off. Other algorithms would select the newest release within a major version even if no module has ever tested it and it was released 5 seconds ago. When…

Thanks for the correction; I worded that unclearly. My point is that with MVS, the solver will favour the oldest package that is allowed by the semver version constraint; given transitive dependencies on multiple packages, it will conservatively pick the oldest that satisfies all the constraint. You can bump the version you want by updating the version constraint, but unlike other tools (e.g. Cargo [1]), it will always favour the oldest allowed version.

[1] https://research.swtch.com/cargo-newest.html

Re: An Analysis of vgo

#14
post #8
post #3

The basic upshot of all this is that vgo's attempt to avoid NP-completeness in the dependency resolution algorithm is solving a problem that doesn't matter in practice, and is simultaneously creating downsides that do matter in practice. This is consistent with what I've seen with systems like Cargo. If I had to make a list of top 10 issues I run into with Cargo, theoretical scalability of the core dependency resolut…

I'm curious to hear what problems you've had with cargo. I've found it to be incredibly reliable and easy to use compared to most other similar tools in other languages like java, python, c++ (if its fair to pick on c++ here...)

My pet peeve with cargo is lack of binary dependencies.

Sure you can get around it with having a common workspace where all build artifacts are dumped, but still have to go for a coffee while dependencies like gtkrs-pango get compiled.

Re: An Analysis of vgo

#15
post #5
post #3

The basic upshot of all this is that vgo's attempt to avoid NP-completeness in the dependency resolution algorithm is solving a problem that doesn't matter in practice, and is simultaneously creating downsides that do matter in practice. This is consistent with what I've seen with systems like Cargo. If I had to make a list of top 10 issues I run into with Cargo, theoretical scalability of the core dependency resolut…

On the other hand, git became successful by being fast and by not solving certain of the hard theoretical problems that specialists were obsessed with solving. It's hard to know before hand what the healthy compromise might be.

git may have become successful, but that doesn't mean it's right. I'd argue that most developers still don't really understand git. I know seasoned, senior developers who struggle when git gets into an unfamiliar state, or who are completely mystified as to how it actually works. And don't forget that git spent years improving its initially horrific UI to make it easier and more palatable to people who aren't kernel developers.

If you compare git to rival tools such as Mercurial and Fossil or even Darcs, it's pretty evident that git didn't really win on technical merit, user friendliness or indeed hardly any other metric. It was fast, and it had the dubious advantage of being better than a lot of crappy alternatives at the time, and of being the product of a famous developer. But if technical merit was how we chose software, we might all be using Mercurial today. It's quite possible we'd be better off, too.

git's punting of "certain of the hard theoretical problems" lead to major shortcomings in its design. For example, git does not track branching histories. Once you merge your changes back into a branch and delete the origin branch, the path that your commits took are lost. Another example: Since git's data model works entirely on repository snapshots (there are no "patches" or logical operations, only snapshots), it doesn't know about file renames. Some commands, like log and diff, do understand how to detect renames (but not out of the box!), but it's fundamentally a fuzzy-matching operation that can only work perfectly if you separate your rename commits from modification commits, which is rarely feasible.

Don't get me wrong, git has overall been a force for good, but I think it's a pretty terrible example in this particular case. In fact, I'm not sure I can think of many cases where the "worse is better" philosophy was ultimately a good idea. In every instance that comes to mind (MySQL, MongoDB, PHP), the fast-and-loose attitude ultimately came back to hurt everyone, and years were spent paying for those mistakes, plugging one hole at a time.

The original decision by the Go team to build package management into the language ("go get") was made without really understanding the problem, and this decision has been haunting us pretty much since day one, with developers having to chase unofficial package management tools of the week in order to get their work done. I really want it to be done right, even if takes a bit longer. It's a problem worth solving well.

Re: An Analysis of vgo

#16
post #4

I think the current approach of announcing the death of `dep` before even having a stable alternative (`vgo`) is deeply flawed. With `dep` the ecosystem finally had something most of the developers agreed on, despite whatever shortcomings it had. The model was working. They could have adopted dep and improved on it. The packaging story for Golang has gone from worse to OK to worse again. After all this progress, we s…

> They could have adopted dep and improved on it.

Not really because:

- The scope is different. dep is a package management tool. vgo is more than that. vgo intends to fully replace the go command. This is how we deprecate GOPATH.

- The underlying principles are different enough to make it difficult to "improve" on dep. It looked easier to start from scratch.

References:

"Go += Package Versioning", Russ Cox, https://research.swtch.com/vgo-intro

"FAQ: Why is the proposal not “use Dep”?", Russ Cox, https://github.com/golang/go/issues/24301#issuecomment-37122...

Re: An Analysis of vgo

#17
post #7

This badly needs a TL;DR. If there's something seriously wrong with vgo's dependency versioning then it should be possible to explain it in less than 100k words.

> If there's something seriously wrong with vgo's dependency versioning then it should be possible to explain it in less than 100k words.

That is a very lazy argument.

It's perfectly reasonable to claim something is flawed for numerous complex reasons. It's equally reasonable to claim that a complex flaw may be difficult to explain concisely.

Most importantly though, I don't think Sam is claiming that there's an obvious fatal flaw in vgo, but rather that it is not the ideal choice given the problem space.

To make that more subtle argument, it is necessary to fully define the alternatives and problem space, which is what a lot of the words are attempting to do.

The tl;dr would be along the lines of "vgo knowingly makes a set of tradeoffs which I think are worse than the tradeoffs that are possible with another hypothetical option, which I am also proposing"... but that tl;dr is not really a very useful one, and attempting to condense a more meaningful one will lose a bit too much nuance I think.

Re: An Analysis of vgo

#18
post #10

Earlier quoted context omitted.

Minor correction: MVS requires modules to specify the minimum version to use of a module. The algorithm will then select the newest among those based on the idea that semver is forward compatible among minor versions. The name minimal version selection throws people off. Other algorithms would select the newest release within a major version even if no module has ever tested it and it was released 5 seconds ago. When…

Thanks for the correction; I worded that unclearly. My point is that with MVS, the solver will favour the oldest package that is allowed by the semver version constraint; given transitive dependencies on multiple packages, it will conservatively pick the oldest that satisfies all the constraint. You can bump the version you want by updating the version constraint, but unlike other tools (e.g. Cargo [1]), it will alwa…

Well, sort of. It's more like it has implicit versioning based on a lock file.

In other systems like Cargo and NPM you have a "lock" file which specifies the exact current version of dependencies to use, e.g. if you say you depend on "foo >=1.0" and when you build your project it actually downloads "foo 1.4", this will be written to the lock file. When foo 1.5 is released, it won't be automatically upgraded even though you said it should be fine.

vgo is exactly the same. When you first add foo1 it will implicitly (via semver) assume you meant "foo >=1.0" and download the latest version - foo 1.5. That version will be written to the lock file (go.mod) and when foo 1.6 is released it won't be automatically used. Exactly the same as NPM and Cargo.

And just like NPM and Cargo you can trivially update to the latest compatible versions of your dependencies with `vgo get -u` (like `cargo update`).

It's not that different. The main differences are you can't specify maximum versions, and different major versions are treated as different packages so you can easily have them both.

Re: An Analysis of vgo

#19
post #5
post #3

The basic upshot of all this is that vgo's attempt to avoid NP-completeness in the dependency resolution algorithm is solving a problem that doesn't matter in practice, and is simultaneously creating downsides that do matter in practice. This is consistent with what I've seen with systems like Cargo. If I had to make a list of top 10 issues I run into with Cargo, theoretical scalability of the core dependency resolut…

On the other hand, git became successful by being fast and by not solving certain of the hard theoretical problems that specialists were obsessed with solving. It's hard to know before hand what the healthy compromise might be.

This rings as a wrong interpretation of history.

When git became successful, there were scant few other options with a comparable feature-set regardless of speed.

Its distributed nature, ability to easily merge and rebase sets of changes, etc, were all wonderful solutions to real problems.

I'm unconvinced that its success was because it solved fewer problems than the state of the art, but rather that it solved more. This is a stark contrast to vgo which is intentionally solving fewer problems than other modern dependency management tools.

In addition, git must be performant enough to handle a git repo (namely the linux kernel, its original usecase). Beyond that, I think there's no evidence it intentionally cut features or complexity to be faster.

In the case of vgo, it cuts additional user-facing features in order to be faster, but there's 0 evidence that the speed matters, that there are any go projects in the world that cannot be solved quickly enough with an approximation of a sat solver.

Re: An Analysis of vgo

#20
post #19
post #5

Earlier quoted context omitted.

On the other hand, git became successful by being fast and by not solving certain of the hard theoretical problems that specialists were obsessed with solving. It's hard to know before hand what the healthy compromise might be.

This rings as a wrong interpretation of history. When git became successful, there were scant few other options with a comparable feature-set regardless of speed. Its distributed nature, ability to easily merge and rebase sets of changes, etc, were all wonderful solutions to real problems. I'm unconvinced that its success was because it solved fewer problems than the state of the art, but rather that it solved more.…

(from memory) there was arch (larch,arx,) and monotone and others.

They were glacially slow and difficult to use. But they had very roughly the feature set; in particular the distributed nature.

Post reply on HN