Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The vgo proposal is accepted. Now what?

research.swtch.com

Post reply on HN
Print view View original

71–80 of 138 posts

Re: The vgo proposal is accepted. Now what?

#71

Post by baq »

pcx wrote: post #64
pcwalton wrote: post #50

Earlier quoted context omitted.

Rust was actually around for a good while before Cargo was adopted. (In fact, there were two Cargos, the older of which bore very little resemblance to the Cargo of today.) Of course, Go was stable for longer. I have to admit I'm a bit confused as to why the dependency resolution algorithm in dep is seen as slow. The speed of the solver is not a problem in any other package management system I've seen. If it is indee…

> The speed of the solver is not a problem in any other package management system I've seen This! I've never heard anyone complain about this aspect of a package manager, EVER. vgo seems to be optimizing for a problem no one has.

i've seen aptitude get really confused about what to install and computing a solution for a few minutes.

once.

10 years ago or so, i don't even remember.

Re: The vgo proposal is accepted. Now what?

#72

Post by pcwalton »

baq wrote: post #71
pcx wrote: post #64

Earlier quoted context omitted.

> The speed of the solver is not a problem in any other package management system I've seen This! I've never heard anyone complain about this aspect of a package manager, EVER. vgo seems to be optimizing for a problem no one has.

i've seen aptitude get really confused about what to install and computing a solution for a few minutes. once. 10 years ago or so, i don't even remember.

I should clarify that I'm referring to language package managers. Their problem domains are significantly different than those of system package managers.

Re: The vgo proposal is accepted. Now what?

#73

Post by pcwalton »

TheDong wrote: post #70
kibwen wrote: post #60

Earlier quoted context omitted.

That's quite weird. When I run `rm Cargo.lock && cargo generate-lockfile` on the Servo repo (test performed on the cheapest VPS that money can buy) it exits near-instantly (after first spending three seconds trying to git-fetch new versions of the dozen custom dependencies that live on Github rather than crates.io). For reference, here's what Servo's dependency graph looked like two years ago (July 2016): https://dir…

It's not strange at all for go. Because third party go packages may not have a dep file, and because go programmers expect vendor directories to be minimal and not include unused imports, dep parses all of the go code of the project, and all the project's transitive dependencies. It has to parse every .go file to find all 'import' statements, and it also has to find remote versions by making multiple network requests…

Thanks for the insight, that's very helpful. That would confirm what I suspected: it's not the core solving algorithm that's slow. Rather what's slow is building the graph in the first place.

Re: The vgo proposal is accepted. Now what?

#74

Post by TheDong »

lobster_johnson wrote: post #53
pcwalton wrote: post #50

Earlier quoted context omitted.

Rust was actually around for a good while before Cargo was adopted. (In fact, there were two Cargos, the older of which bore very little resemblance to the Cargo of today.) Of course, Go was stable for longer. I have to admit I'm a bit confused as to why the dependency resolution algorithm in dep is seen as slow. The speed of the solver is not a problem in any other package management system I've seen. If it is indee…

Here [1] is the "dep ensure -v" output for a project of mine. It takes almost 12 seconds even when there are no changes to the actual file. I don't know why, or whether it's actually the solver (though the output seems to indicate it). [1] https://gist.github.com/atombender/7c28f1d371fcb139e1e742a08...

As you can see from the output, it is not the solver per-se, but the weird idiosyncrasies of go imports and gopath layout. 'satisfy' and 'select-atom' and such are the solver bit and take about 20ms all together. A SAT solver is 20ms, MVS might be 1ms, but who cares about that difference, right?

The top 3 items there are slow because they're:

1. 'source-exists' (~6s) which will do network traffic to find if a project exists to be downloaded or is in the cache; it's network io heavy in most cases.

2. list-packages (~3s) which parses the downloaded source code for import statements to find further dependencies; disk-io heavy + go loader has to do some work

3. gmal - GetManifestAndLock (~2s) which looks for lock files, including of other dependency solvers; disk io mostly I think

Any system designed with the constraint that it cannot use a centralized registry / list, must be compatible with things not using this system (and so must parse their code), etc will have these problems regardless of the algorithm.

Those steps are all doing network/disk-io/go-parsing, and none of that is SAT solving.

I don't think vgo has these problems because vgo is built by the go team and can dictate far more, such as the use of a centralized repo, that all dependencies must use vgo, etc.

Re: The vgo proposal is accepted. Now what?

#76

Post by lobster_johnson »

TheDong wrote: post #74
lobster_johnson wrote: post #53

Earlier quoted context omitted.

Here [1] is the "dep ensure -v" output for a project of mine. It takes almost 12 seconds even when there are no changes to the actual file. I don't know why, or whether it's actually the solver (though the output seems to indicate it). [1] https://gist.github.com/atombender/7c28f1d371fcb139e1e742a08...

As you can see from the output, it is not the solver per-se, but the weird idiosyncrasies of go imports and gopath layout. 'satisfy' and 'select-atom' and such are the solver bit and take about 20ms all together. A SAT solver is 20ms, MVS might be 1ms, but who cares about that difference, right? The top 3 items there are slow because they're: 1. 'source-exists' (~6s) which will do network traffic to find if a project…

Thanks for the explanation!

The fact that dep parses import statements (as does Glide) is something I've never liked. It means that if you run "dep ensure --add" on something not yet imported, it will complain, and the next "ensure" will remove it. This is never in line with how I actually work. I need the dependencies before I can import them! There's no editor/IDE in existence that lets you autocomplete libraries that haven't been installed yet.

It also means that "dep ensure" parses my code to discover things not yet added to Gopkg.toml. That's upside down to me. I want it to parse its lockfile and nothing else; the lockfile is what should inform its decisions about what to install so that my code works, my code shouldn't be driving the lockfile! If I try to compile my code and it imports stuff that isn't in the lockfile, it should fail, and dep shouldn't try to "repair" itself with stuff that I didn't list as an explicit dependency.

I'm sure there are edge cases where the current behaviour can be considered rational, but I don't know what they are. As you point out, dep has to do a lot of work -- but why? Running "dep ensure" when the vendor directory is in perfect sync with the lockfile should take no time at all, and certainly shouldn't need to access the network. Yet it takes the same amount of time with or without a lockfile.

Re: The vgo proposal is accepted. Now what?

#77

Post by pcx »

marcus_holmes wrote: post #65
pcx wrote: post #64

Earlier quoted context omitted.

> The speed of the solver is not a problem in any other package management system I've seen This! I've never heard anyone complain about this aspect of a package manager, EVER. vgo seems to be optimizing for a problem no one has.

Go has always optimised for build speed. I guess they considered dependency resolution as part of the build process. Which it technically is, I suppose, but when you're coding and iterating the code-build-run loop you generally don't need to add new dependencies each time. And that's when the build speed matters, of course.

And, most CIs cache build dependencies these days which is an easy work around.

Re: The vgo proposal is accepted. Now what?

#78

Post by kibwen »

pcwalton wrote: post #73
TheDong wrote: post #70

Earlier quoted context omitted.

It's not strange at all for go. Because third party go packages may not have a dep file, and because go programmers expect vendor directories to be minimal and not include unused imports, dep parses all of the go code of the project, and all the project's transitive dependencies. It has to parse every .go file to find all 'import' statements, and it also has to find remote versions by making multiple network requests…

Thanks for the insight, that's very helpful. That would confirm what I suspected: it's not the core solving algorithm that's slow. Rather what's slow is building the graph in the first place.

Yes, I had overlooked that Go probably doesn't have anything like the crates.io index (https://github.com/rust-lang/crates.io-index) to allow instant discovery of versioning metadata. And, AFAICT, even MVS would have the same problem here and would take the same time to resolve, since it still needs to access the network to fetch remote repos to discover versioning metadata; rather than pointing the finger at SAT solvers, it looks like vgo should be tackling Go's lack of a central package host (the vgo manifesto mentions "proxies", but it seems that those are just intended for solving the problem of persistent availability).

Re: The vgo proposal is accepted. Now what?

#79

Post by kibwen »

iambvk wrote: post #69
vhbit wrote: post #49

Earlier quoted context omitted.

> The lock file stops future upgrades; once it is written, your build stays on serde 1.0.27 even when 1.0.28 is released. That means that the only difference is that `vgo` doesn't require lock file for reproducible builds, now the question is what's considered so terrible wrong/dangerous with having a lock file?

If there was a security fix in a package, MVS allows build infra to automatically upgrade (assuming packages imported are well maintained by their owners) where as lock-files require manual intervention. Imagine the packaging and development ecosystem in internet scale.

I'm not sure what this is asserting. There's no association between MVS and build infra. If you have a security fix to apply, you'd run `vgo get -u` to upgrade all your modules; systems like Cargo that use lockfiles would run `cargo upgrade`. This is just as automatic. In fact, vgo is less automatic here because systems like Cargo allow users to decide when they want certain upgrades automatically, which means that new downstream users can get the security fix even without needing to explicitly upgrade. And vgo also requires more manual intervention to upgrade than traditional build systems, because if the upgrade requires a major version bump then you also have to edit the import statements in your source files as well. vgo is both less automatic and requires more manual labor; that appears to be an intentional choice on behalf of the vgo manifesto.

Re: The vgo proposal is accepted. Now what?

#80

Post by Spiritus »

throwaway243425 wrote: post #33

To an outsider this may sound like there is some sort of process that resulted in this solution, there actually isn't any. The committee is nothing more than a simple bureaucracy to the point that it is almost a joke how Russ makes a proposal, community is against it, then it gets accepted by the Committee. It is all just a funny joke.

Who says the community is against it? I happen to like it.
Post reply on HN