Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Taking Go modules for a spin

dave.cheney.net

Post reply on HN
Print view View original

51–60 of 75 posts

Re: Taking Go modules for a spin

#51

Post by perfmode »

greenhouse_gas wrote: post #49
dilap wrote: post #28

Earlier quoted context omitted.

From cargo's source: "Actually solving a constraint graph is an NP-hard problem. This algorithm is basically a nice heuristic to make sure we get roughly the best answer most of the time." ( https://github.com/rust-lang/cargo/blob/master/src/cargo/cor... ) vgo's more-constrained specification for dependencies means there is exactly one right answer, and it can be easily and quickly calculated by both computer and hum…

>Actually solving a constraint graph is an NP-hard problem How big of a deal is this IRL? Assuming you have 1000 modules, how long should it take to solve the graph?

depends on the complexity. if it is 2^1000 then forget about finding an optimal result.

Re: Taking Go modules for a spin

#52

Post by bbatha »

greenhouse_gas wrote: post #49
dilap wrote: post #28

Earlier quoted context omitted.

From cargo's source: "Actually solving a constraint graph is an NP-hard problem. This algorithm is basically a nice heuristic to make sure we get roughly the best answer most of the time." ( https://github.com/rust-lang/cargo/blob/master/src/cargo/cor... ) vgo's more-constrained specification for dependencies means there is exactly one right answer, and it can be easily and quickly calculated by both computer and hum…

>Actually solving a constraint graph is an NP-hard problem How big of a deal is this IRL? Assuming you have 1000 modules, how long should it take to solve the graph?

Not very the parts that make it NP hard are allowing libraries to specify maximum versions (and other more complex version ranges). Most of the time libraries use minimum constraints (~) or (^) which allows the heuristic to work like go's algorithm. For rust, node, and other languages libraries can be imported twice as different versions (without requiring a major version renaming like go) this also allows the heuristic to have an out: if it reaches a really complex case it can just give you both versions. Beyond that package management is a barely disguised 3-SAT solver which we have good, fast solvers. There are definitely some edge cases, but when's the last time you ran any of the following package managers and worried about dependency solve speed? cargo, apt-get, npm (and yarn), dnf, zypper. IO far and away dominates the profiles of these programs, solver speed is basically a non-issue in practice.

Re: Taking Go modules for a spin

#53

Post by dragonwriter »

eikenberry wrote: post #46
dragonwriter wrote: post #44

Earlier quoted context omitted.

The catch to the vgo approach required that no package in the ecosystem ever have even unintentional backwards incompatibilities, because you can't do anything other than specify minimum versions. Or, rather, it makes the resulting problems something that need to be addressed outside the scope of dependency specification and resolution. When you just decide not to address a significant part of the problem, the soluti…

> unintentional backwards incompatibilities You mean a bug? Because that's what that is and it is no different from any other bug, and like any other bug they are outside the scope of dependency specifications as they are unintended.

> like any other bug they are outside the scope of dependency specifications

Known relevant bugs in particular versions of dependencies are not outside the scope of what non-vgo dependency management solutions address.

Re: Taking Go modules for a spin

#54

Post by [deleted] »

dragonwriter wrote: post #53
eikenberry wrote: post #46

Earlier quoted context omitted.

> unintentional backwards incompatibilities You mean a bug? Because that's what that is and it is no different from any other bug, and like any other bug they are outside the scope of dependency specifications as they are unintended.

> like any other bug they are outside the scope of dependency specifications Known relevant bugs in particular versions of dependencies are not outside the scope of what non-vgo dependency management solutions address.

[deleted]

Re: Taking Go modules for a spin

#55

Post by akavel »

atombender wrote: post #17
benesch wrote: post #12

Earlier quoted context omitted.

I can't speak to this with absolute certainty, but I have some speculation to offer. The story of GOPATH is tightly intertwined with the story of package management. Go is a Google project, and Google has a very unique approach to package management: commit everything to the monorepo. The GOPATH is, in essence, a monorepo. If you want to change the API of a library, well, you can just change all its callers across yo…

The Go team, for years, actually insisted that package management was something the Go community had to go and figure out. As you say, Google uses a monorepo where they check everything into a single tree. One of the core Go developers -- I forget who, unfortunately -- actually claimed at one point that they didn't want to design a package management system because they didn't know how ; since Google used a monorepo,…

Personally, I actually never thought Dep was blessed by the Go team to the point that it "was going to be official". So I wouldn't be so fast to say "pretty much everyone". Notably, some loud people "thought" so; I've noticed that many quiet people quietly did not (as can be seen e.g. by the numerous voiced agreements and endorsements of the vgo prototype on the mailing list). Again, I personally expected exactly something similar to what was pulled off in the end to happen. Especially given that whenever somebody started claiming Dep "is going to be official", rsc/rob (don't remember) seemed fast to correct that it's not so, and that it's only an official experiment. Even Sam, after AFAIR being corrected so, was seemingly careful to say only about an official experiment in public emails, readme, etc.

I have numerous thoughts about developments like this one. Mostly, that I've seen a similar situation happen in numerous communities already, Go totally not being the first nor the last one, that the steering commitee have the last say, and may have different taste. I learnt to accept that their choice usually does have merit and usually actually ends for the better. I learnt that it requires a lot of humbleness and sometimes gritting one's teeth, learning to let go of hurt feelings, and accepting that someone else may have reasons you still need to grow to understand. Personally, my own view is that for Sam, this was probably the first time something like this happened, and he wasn't prepared for the hit. And I agree those never stop tasting bitter, given the work one has put in a project of this kind. A good will, hard contribution, being de facto rejected in the end. A child being "lost". But that's not the whole truth, because the child is in this case reborn, though in somewhat different shape. The experiment has served its purpose and brought a lot of value, a significant contribution. On the other hand, I do sometimes wonder, can such situations and misunderstandings be avoided somehow? Or is the world just not perfect enough? And by the way, I also think that Russ was actually taken by surprise by the extent of the reaction. I suppose that's why it took him so long to react, which let the situation and complains get somewhat louder than necessary.

But that's too just my personal opinion. One of many in this somewhat unfortunate situation. I just wanted to also let my steam off in the end, starting to grow more and more tired of the recurring claims that "everybody is surprised". On the contrary, I'm personally one of the people looking forward to vgo, and strongly unconvinced by what dep has become.

Re: Taking Go modules for a spin

#56

Post by Lazare »

flurrything wrote: post #38
dilap wrote: post #28

Earlier quoted context omitted.

From cargo's source: "Actually solving a constraint graph is an NP-hard problem. This algorithm is basically a nice heuristic to make sure we get roughly the best answer most of the time." ( https://github.com/rust-lang/cargo/blob/master/src/cargo/cor... ) vgo's more-constrained specification for dependencies means there is exactly one right answer, and it can be easily and quickly calculated by both computer and hum…

Is there a catch to vgo's approach? If not, why aren't the cargo people copying it?

The counterarguments are, basically:

1. vgo focuses on the wrong issue (if you're spending a ton of time resolving and re-resolving your dependency graph, maybe the issue is your build process).

2. vgo will get the wrong answers and/or make development much harder

There's a long writeup of some of the ways vgo can go wrong here: https://sdboyer.io/vgo/failure-modes/ and some background here: https://sdboyer.io/vgo/intro/ among other places. And there was a lot of discussion here: https://news.ycombinator.com/item?id=17183101

I'd say there's about a 3% chance vgo ends up being a smashing success that revolutionises package management and gets copied by everyone else, a 30% chance that vgo works well for golang due to their unique requirements but has nothing to offer anyone else, and about a 67% chance it ends up being a failure and being scrapped or heavily revised to scrap the novel, controversial and (arguably) fundamentally broken ideas that set it apart from every other package manager.

But fundamentally, the reason the cargo people aren't copying it right now is that it doesn't even really claim to have advantages over cargo for rust. (There are some quirks in the golang ecosystem which mean you end up analyzing your dependency tree way, way, more than you do in basically any other common language. That makes speed important for golang, but for everyone else, it's almost meaningless.) "We make the unimportant stuff fast at the expense of getting the importing stuff wrong" isn't very compelling. :)

Of course, the vgo people would phrase it as "we make the important stuff fast and we get the important stuff right", so...time will tell. But don't expect anyone to copy this quickly; it remains to be seen if it'll even work for golang, and it'd need to be a huge step up from the current state of the art to make it worth switching for other languages and ecosystems.

Re: Taking Go modules for a spin

#57

Post by dilap »

ainar-g wrote: post #50
dilap wrote: post #48

Earlier quoted context omitted.

Thx. OK, so that makes it sound like it has nothing to do with GOPATH. But interesting (annoying?) that vendor won't kick in unless you specifically ask for it.

So far, I get a feeling that vendoring will be a second-class citizen in the world of modules. Unfortunately, I will not be surprised if the team considers removing the support entirely in a future release. I just hope that a viable alternative will be proposed.

yeah, i feel like it's really handy to not be reliant on 3p servers to do a build.

Re: Taking Go modules for a spin

#58

Post by kaeshiwaza »

pinkgodzilla wrote: post #24
benesch wrote: post #12

Earlier quoted context omitted.

I can't speak to this with absolute certainty, but I have some speculation to offer. The story of GOPATH is tightly intertwined with the story of package management. Go is a Google project, and Google has a very unique approach to package management: commit everything to the monorepo. The GOPATH is, in essence, a monorepo. If you want to change the API of a library, well, you can just change all its callers across yo…

Can someone help me understand why people hate gopath? We have a script called govars.sh at the root of every project and it sets the gopath exactly the same way one would use virtualenv in Python. We remove the default one from .bashrc or .profile. This basically makes gopath disappear entirely - we never bother explaining to new devs what it "really" means. Just ask them to use our template project folder structure…

I've the same workflow, but like you said gopath disappear. You just hate gopath also :-) I mean, with go modules you will have the same workflow but without the need to adjust the envs.

Re: Taking Go modules for a spin

#59

Post by kaeshiwaza »

atombender wrote: post #23
benesch wrote: post #22

Earlier quoted context omitted.

> My personal theory is that what made Russ Cox cave in was his discussions with Sam Boyer. Cox thought Boyer was going down the wrong path, and thought he had a better solution. That's certainly my understanding of the situation. Matt Farina has a great commented history of dep and vgo [0] if you haven't already seen it. The comments are particularly enlightening. Still, it's not clear to me what made the Go team ge…

Indeed, as I said, Dep is okay when it works, until it doesn't. This [1], for example, is a blocking issue, and requires some manual editing of the lock file to get around. I've had other issues. The issues pale in comparison to the horribleness of Glide, but it's interesting just how these tools end up being so damn flaky. [1] https://github.com/golang/dep/issues/1207

Not only buggy, theses tools didn't follow the simplicity that we like in Go. For example gb was more in the Go philosophy for my taste. I was surprise that it was not chosen as the official experiment.

Re: Taking Go modules for a spin

#60

Post by slrz »

Lazare wrote: post #56
flurrything wrote: post #38

Earlier quoted context omitted.

Is there a catch to vgo's approach? If not, why aren't the cargo people copying it?

The counterarguments are, basically: 1. vgo focuses on the wrong issue (if you're spending a ton of time resolving and re-resolving your dependency graph, maybe the issue is your build process). 2. vgo will get the wrong answers and/or make development much harder There's a long writeup of some of the ways vgo can go wrong here: https://sdboyer.io/vgo/failure-modes/ and some background here: https://sdboyer.io/vgo/in…

> There are some quirks in the golang ecosystem which mean you end up analyzing your dependency tree way, way, more than you do in basically any other common language.

What are these?

Post reply on HN