Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The vgo proposal is accepted. Now what?

research.swtch.com

Post reply on HN
Print view View original

61–70 of 138 posts

Re: The vgo proposal is accepted. Now what?

#61

Post by kibwen »

amasad wrote: post #57
kibwen wrote: post #55

Earlier quoted context omitted.

> The standard answer is lock files which is extra bloat and leads to merge conflicts. Bloat how? A lockfile is a handful of bytes. A thousand lockfiles could fit in the space of a single Go hello world binary. And I've never heard of a merge conflict from a lockfile. What are the actual arguments against lockfiles?

More things, more state = bloat. Don't you agree that if you can do something without holding state it's invariably simpler? That's what MVS does. There's also cases where your lock file and manifest are not in agreement. e.g. https://github.com/rust-lang/cargo/issues/4100

> Don't you agree that if you can do something without holding state it's invariably simpler?

Only if the two solutions under consideration are really solving the same problems. Sam Boyer alludes to this very thing in his discussion of MVS:

"If there are two algorithms that satisfy the same requirements, and only one is NP-complete, you pick the other one. That’s axiomatic. Moreover, if you have only an NP-complete algorithm for a particular problem, finding a less complex alternative that does the same job is an electrifying discovery. When such an alternative algorithm is proposed, however, the inevitable question to be answered is whether it actually does meet the original requirements. [...] But, in avoiding SAT, MVS also cuts out some of the complexities that I believe are essential to the domain. Being essential, the problems don’t go away when MVS ignores them. Instead, they’re redistributed into other, often less obvious places. If reading the vgo blog posts gave you a general sense of unease that you couldn’t put your finger on, that might’ve been you intuitively sensing some of these redistributions." https://sdboyer.io/vgo/intro/

Re: The vgo proposal is accepted. Now what?

#62

Post by blaisio »

kibwen wrote: post #55
amasad wrote: post #52

Earlier quoted context omitted.

There are a lot of suboptimal design decisions that's shared across the state of the art in package management (cargo, pipenv, npm, yarn, etc) that the Go team is not taking for granted. One is reproducible builds. The standard answer is lock files which is extra bloat and leads to merge conflicts. vgo is trying to avoid it with the "minimal versions" strategy ( https://research.swtch.com/vgo-repro ). Another is shar…

> The standard answer is lock files which is extra bloat and leads to merge conflicts. Bloat how? A lockfile is a handful of bytes. A thousand lockfiles could fit in the space of a single Go hello world binary. And I've never heard of a merge conflict from a lockfile. What are the actual arguments against lockfiles?

It's pretty easy to get a merge conflict from a lock file. If two developers add a dependency and then try to merge their changes together, it can happen easily.

Re: The vgo proposal is accepted. Now what?

#63

Post by kibwen »

blaisio wrote: post #62
kibwen wrote: post #55

Earlier quoted context omitted.

> The standard answer is lock files which is extra bloat and leads to merge conflicts. Bloat how? A lockfile is a handful of bytes. A thousand lockfiles could fit in the space of a single Go hello world binary. And I've never heard of a merge conflict from a lockfile. What are the actual arguments against lockfiles?

It's pretty easy to get a merge conflict from a lock file. If two developers add a dependency and then try to merge their changes together, it can happen easily.

In that scenario, you're going to get a merge conflict anyway in your manifest (e.g. in go.mod).

Re: The vgo proposal is accepted. Now what?

#64

Post by pcx »

pcwalton wrote: post #50
lobster_johnson wrote: post #28

Earlier quoted context omitted.

Rust/Cargo had the luxury of being a greenfield project that could adopt semver from the beginning, whereas Go made the mistake of starting out, and then going years , without any official package management solution. As a result, Go has a swathe of applications and libraries that use specific workflows as well as a mélange of community-developed package management tools such as godep, Glide and dep. The semver stand…

Rust was actually around for a good while before Cargo was adopted. (In fact, there were two Cargos, the older of which bore very little resemblance to the Cargo of today.) Of course, Go was stable for longer. I have to admit I'm a bit confused as to why the dependency resolution algorithm in dep is seen as slow. The speed of the solver is not a problem in any other package management system I've seen. If it is indee…

> The speed of the solver is not a problem in any other package management system I've seen

This! I've never heard anyone complain about this aspect of a package manager, EVER. vgo seems to be optimizing for a problem no one has.

Re: The vgo proposal is accepted. Now what?

#65

Post by marcus_holmes »

pcx wrote: post #64
pcwalton wrote: post #50

Earlier quoted context omitted.

Rust was actually around for a good while before Cargo was adopted. (In fact, there were two Cargos, the older of which bore very little resemblance to the Cargo of today.) Of course, Go was stable for longer. I have to admit I'm a bit confused as to why the dependency resolution algorithm in dep is seen as slow. The speed of the solver is not a problem in any other package management system I've seen. If it is indee…

> The speed of the solver is not a problem in any other package management system I've seen This! I've never heard anyone complain about this aspect of a package manager, EVER. vgo seems to be optimizing for a problem no one has.

Go has always optimised for build speed. I guess they considered dependency resolution as part of the build process.

Which it technically is, I suppose, but when you're coding and iterating the code-build-run loop you generally don't need to add new dependencies each time. And that's when the build speed matters, of course.

Re: The vgo proposal is accepted. Now what?

#66

Post by mseepgood »

eberkund wrote: post #20

I wish they had just copied Rust/Cargo. I remember reading a comment on GitHub somewhere from one of the Go maintainers who responded to someone expressing a similar sentiment and his reply was basically that Go is somehow different than every other language and they need to explore and find a unique custom solution for their particular use case. Has it ever been addressed anywhere why the tried and true "list of pac…

> I wish they had just copied Rust/Cargo

Cargo is really a suboptimal design. With vgo Go advances the state of the art of dependency management.

Re: The vgo proposal is accepted. Now what?

#67

Post by lilyball »

amasad wrote: post #52
eberkund wrote: post #20

I wish they had just copied Rust/Cargo. I remember reading a comment on GitHub somewhere from one of the Go maintainers who responded to someone expressing a similar sentiment and his reply was basically that Go is somehow different than every other language and they need to explore and find a unique custom solution for their particular use case. Has it ever been addressed anywhere why the tried and true "list of pac…

There are a lot of suboptimal design decisions that's shared across the state of the art in package management (cargo, pipenv, npm, yarn, etc) that the Go team is not taking for granted. One is reproducible builds. The standard answer is lock files which is extra bloat and leads to merge conflicts. vgo is trying to avoid it with the "minimal versions" strategy ( https://research.swtch.com/vgo-repro ). Another is shar…

The "minimal versions" strategy basically just replaces the lockfile with your dependency list.

Two branches want to update the same dependency? With a lockfile, they both update the lockfile and will have a merge conflict (assuming the updates aren't identical). With "minimal versions", they both update the file that declares the dependency, and, you guessed it, have a merge conflict.

Re: The vgo proposal is accepted. Now what?

#68

Post by jchw »

lobster_johnson wrote: post #36
jchw wrote: post #31

Earlier quoted context omitted.

I think Cargo-like is the direction things are headed even with vgo. In a way, though, I don't think the answer is that Go is "special," just that they wanted to explore the problem space of package management, dependencies, etc. in more detail before locking things down. It may have ended in a slightly weakened ecosystem temporarily, but I hope it helps to ensure that the semantics and behavior of Go's package manag…

The vgo proposal explicitly rejects the "Cargo way" [1]. There's no lock file, and the MVS algorithm requires, as far as I recall, that the go.mod file is modified whenever the developer wants to update to a new version. [1] https://research.swtch.com/vgo-repro

Indeed, but it does seem like a central repository is possibly the direction it's headed, or at least away from source repositories. Personally, I won't be too upset by MVS if it's one of the only deviations, though I do agree with Sam Boyer on the matter, who I actually had the pleasure of speaking with a bit ago.

Re: The vgo proposal is accepted. Now what?

#69

Post by iambvk »

vhbit wrote: post #49
lobster_johnson wrote: post #37

Earlier quoted context omitted.

Cox discusses Cargo here, and why he doesn't like it: https://research.swtch.com/vgo-repro

> The lock file stops future upgrades; once it is written, your build stays on serde 1.0.27 even when 1.0.28 is released. That means that the only difference is that `vgo` doesn't require lock file for reproducible builds, now the question is what's considered so terrible wrong/dangerous with having a lock file?

If there was a security fix in a package, MVS allows build infra to automatically upgrade (assuming packages imported are well maintained by their owners) where as lock-files require manual intervention.

Imagine the packaging and development ecosystem in internet scale.

Re: The vgo proposal is accepted. Now what?

#70

Post by TheDong »

kibwen wrote: post #60
lobster_johnson wrote: post #53

Earlier quoted context omitted.

Here [1] is the "dep ensure -v" output for a project of mine. It takes almost 12 seconds even when there are no changes to the actual file. I don't know why, or whether it's actually the solver (though the output seems to indicate it). [1] https://gist.github.com/atombender/7c28f1d371fcb139e1e742a08...

That's quite weird. When I run `rm Cargo.lock && cargo generate-lockfile` on the Servo repo (test performed on the cheapest VPS that money can buy) it exits near-instantly (after first spending three seconds trying to git-fetch new versions of the dozen custom dependencies that live on Github rather than crates.io). For reference, here's what Servo's dependency graph looked like two years ago (July 2016): https://dir…

It's not strange at all for go.

Because third party go packages may not have a dep file, and because go programmers expect vendor directories to be minimal and not include unused imports, dep parses all of the go code of the project, and all the project's transitive dependencies.

It has to parse every .go file to find all 'import' statements, and it also has to find remote versions by making multiple network requests per dependency (typically 1 http-get + 1 git pull operation).

This is obviously going to be much slower than cargo where it's assumed every dependency is also using cargo and all needed information is present in metadata files... and there's one single fast api to download data from and cache (crates.io).

If cargo had to do the equivalent of `cargo check`-style parsing to find all 'extern crate' and 'use' statements before it could spit out a valid lock, and it couldn't use only 1 request to update all crates.io data, it would probably be closer to the speed of dep.

I think the speed difference is thus largely a result of go's lack of a central repository and lack of a unified packaging solution.

Post reply on HN