Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Go += Package Versioning

research.swtch.com

Post reply on HN
Print view View original

141–150 of 213 posts

Re: Go += Package Versioning

#141

Post by dkarl »

munificent wrote: post #131
dkarl wrote: post #124

Earlier quoted context omitted.

For what it's worth, Dart does not let you have two versions of the same package in your application, even different major versions. This restriction does cause real pain, but it doesn't appear to be insurmountable. Most of the pain seems to be in the performance issues over-constrained dependencies caused in our old version solver and not in the user's code itself. In almost all cases, I think there is a single vers…

> I think this makes a strong case for not releasing major version upgrades that use the same package names. The very idea of two incompatible things having the same name should set off alarm bells. Instead of trying to make that work, we should be avoiding it. If you do that, I think you'll find in practice that one of two things happens (or more likely, both , in a confusing mixture): 1. People start releasing pack…

I think you'll find in practice that one of two things happens

I think the existence of those practices (like Java dependency shading) proves that people are struggling towards this solution on their own, without support from the language or the community. With official support, if major versions work the same way for everybody, it won't need to be so janky.

In practice, I predict that people would start behaving better, doing what they should have been doing (and what many have been doing) all along: avoiding unnecessary breaking changes in non-0.x libraries, choosing function signatures carefully, and growing by accretion and living with their mistakes. Right now, I think some developers see major version bumps as a convenient way to erase their mistakes, without taking into account the cost imposed on users who end up juggling dependency conflicts.

Re: Go += Package Versioning

#142

Post by joncalhoun »

munificent wrote: post #132
joncalhoun wrote: post #114

Earlier quoted context omitted.

You (and likely everyone else) should look at the tour as well before commenting, as I think many people are misunderstanding some of the subtler points. > If I'm starting a brand new from scratch Ruby on Rails application today, in 2017, there is no reason it should default to having me use Rails 1.0 from 2005. In the tour it states, "We've seen that when a new module must be added to a build to resolve a new import…

> In the tour it states, "We've seen that when a new module must be added to a build to resolve a new import, vgo takes the latest one." which means that the newest Rails would be used and set in your `go.mod` file. That works for adding a new dependency. But, as I understand it, if I decide to upgrade my dependency on foo by changing its already-present version in my app's module file, this does not upgrade any of t…

Let's say I create a program that is using foo and end up with the following dependencies:

main:

    requires "foo" v1.0.0
foo (v1.0.0):

    requires "bar" v1.0.0
Right now if I check my dependencies, I'll have something like this:

    MODULE    VERSION
    main      -
    bar       v1.0.0
    foo       v1.0.0
Now lets say some time passes, and both foo and bar release new versions:

foo:

    v1.0.0
    v1.1.0
bar:

    v1.0.0
    v1.0.1
    v1.1.0
    v1.1.1
    v1.1.2
And the deps for foo v1.1.0 are:

foo (v1.1.0):

    require "bar" v1.0.1
Realizing that foo has an update, I decide I want to upgrade. I'd do vgo get foo. My updated dependencies (shown with "vgo list -m") are:

    MODULE    VERSION
    main      -
    bar       v1.0.1
    foo       v1.1.0
bar gets its version increased as well, using the version specified by the foo package's module. This makes sense to me - the foo package maintainer has stated that he only needs v1.0.1 to be stable, so we default to what he specified.

Now imagine I want to add another package, say it is the wham package and it has the following dependencies:

wham (v1.0.0):

    require "bar" v1.1.1
If I add this to my code my versions will now be:

    MODULE    VERSION
    main      -
    wham      v1.0.0
    bar       v1.1.1
    foo       v1.1.0
bar now uses v1.1.1 because it is the minimal version that satisfies all of my modules. vgo DOES upgrade bar for us, but not beyond the lower version number required to satisfy all of our modules. That said, we can still upgrade it manually with "vgo get bar", after which it will be using v1.1.2 because our main dependencies would become:

main:

    requires "foo" v1.1.0
    requires "wham" v1.0.0
    requires "bar" v1.1.2
In short, upgrading foo WILL upgrade all of foo's dependencies in order to meet it's minimum version requirements, but no further. That said, you can still manually upgrade any of those dependencies.

To me this makes sense. The creator of foo may have avoided upgrading the dependency on bar for some performance reasons, so this upgrade only happens in your code if it is required by another package, you initiate it manually, or if the foo package releases a new version with updated dependencies in its go.mod file.

PS - I've tested this all using the prototype of vgo. You can see yourself by grabbing this code: github.com/joncalhoun/vgo_foo_main and then use vgo to list dependency versions and try upgrading foo which has a dep on demo.

Re: Go += Package Versioning

#143

Post by overcyn »

mCOLlSVIxp6c wrote: post #93

I predict that under the proposed minimum version selection system some package will decide it's important and flawless enough that users should always depend on the most recent version. The package will recommend that users depend on version v1.0.0 but the first release will be v1.100.0. Subsequent releases will decrement the minor version to make sure newer releases are selected.

User's wouldn't be able to depend on version v1.0.0 if it doesn't exist. From what I understand the tool isn't downloading the list of all versions from GitHub and selecting the minimum. Its searching for the maximum version number (i.e. minimal required version), from your go.mod file and all your dependencies' go.mod file.

Re: Go += Package Versioning

#144

Post by thefounder »

xrstf wrote: post #129

A lot of this seems to hinge on "go get" being the best way to get Go source code. Most discussions I've read so far come down to "well, this might be cool, but it would break go get, so we cannot ever do it". I for one could live without go get alltogether. In 99% of cases it boils down to a simple git clone anyway, which could (should?) be done by the package manager anyway. Maybe it's time to re-evaluate the exist…

go get, go install and go build are cool and great tools! Let's not turn them into a maven/pom/npm crap!

Re: Go += Package Versioning

#145

Post by [deleted] »

stickfigure wrote: post #135
abvdasker wrote: post #83

Earlier quoted context omitted.

Yeah it's crazy that there's still so much controversy around this topic considering Node and Ruby have had amazing dependency management for over half a decade at this point. Dependency management in those languages is pretty much a solved problem and the fact that go isn't there yet drives me nuts since I have to work with it every day for my job.

Node and Ruby have had amazing dependency management for over half a decade at this point. Eh? NPM didn't have package-lock.json until v5, released in 2017. Before then there was the optional shrinkwrap that nobody used, so builds were totally unreproducible. Ruby at least had Gemfile.lock from early days. Unfortunately there have been so many compatibility problems with different versions of Ruby itself that someone…

[deleted]

Re: Go += Package Versioning

#146

Post by joncalhoun »

munificent wrote: post #84
uluyol wrote: post #46

Earlier quoted context omitted.

Actually, I think the SAT solver is avoided by making the only version constraints of the form Using the min version appears to eschew the need for lock files. Want to upgrade? Bump your min version.

> Using the min version appears to eschew the need for lock files. This only works if the system also prevents you from publishing a version of a package with a lower number than any previously-published version of that package. So, for example, after you've shipped foo 1.3.0, if a critical security issue is found in foo 1.2.0, you can't ship foo 1.2.1. Instead, all of your users have to deal with revving all the way…

You CAN ship v1.2.1 after v1.3.0 is live. I have tested this with the vgo prototype and it works fine (see github.com/joncalhoun/vgo_main):

    $ vgo list -m -u
    MODULE                          VERSION                    LATEST
    github.com/joncalhoun/vgo_main  -                          -
    github.com/joncalhoun/vgo_demo  v1.0.1 (2018-02-20 18:26)  v1.1.0 (2018-02-20 18:25)
Notice that v1.0.1 was released AFTER v1.1.0

What the minimum version is doing is giving our code a way to automatically resolve upgrades if they are necessary. Eg if module X requires module Z w/ a version >= 1.0.1, while module Y requires Z with a version >= 1.1.0 we clearly CANNOt use v1.0.1, as it won't satisfy the requirements of Y, but we CAN use v1.1.0 because it satisfies both.

The "minimum" stuff basically means that even if a version 1.3.2 of Z is available, our code will still use v1.1.0 because this is the minimal version to satisfy our needs. You can still upgrade Z with vgo, or if you upgrade a module X and it now needs a newer version of Z vgo will automatically upgrade in that case (but to the minimal version that X needs), but random upgrades to new versions don't just occur between builds.

Re: Go += Package Versioning

#147

Post by TheDong »

j_bond wrote: post #125

I am curious why go doesn't just vendor dependencies like npm does when confronted with conflicting requirements instead of trying to figure out a version that satisfies all worlds? I always found this to be a nice feature when working with npm. I am also firm believer in version-pinning/lockfiles. Updating versions should be a separate workflow with first-class built-in tools to support it. I think that is the area…

nodejs's "require", which pulls different versions depending on which module calls it, is a cool trick.

Unfortunately, it also has a downside, and in go that downside would be noticeable.

Let's take one easy example: a logging library. Let's say I pull in "logrus v1.0.1" and one of my dependencies pulls in "logrus v1.0.2". In my "main" function, I set logrus's default loglevel to debug (logrus.SetLevel(logrus.DebugLevel)).

If go did the thing nodejs does (a different copy of logrus for my dependency than my main), the "logrus.SetLevel" would only affect my package, not any of my dependencies; they'd still log at a default level.

This would be true of other things; "func init" code would run once per dependency that had that library, not just once, maps wouldn't be shared, pools, etc.

This is a lot more memory usage, but it's also really surprising, especially in the case of logging.

I definitely prefer having only one copy of a library in memory and having package-level variables (like loglevel) work as expected.

If that ends up failing and I need two different versions, vendor+import-path rewriting allows an escape hatch

These days, npm actually tries to minimize and flatten dependency versions as much as possible to avoid the huge memory tax.

Re: Go += Package Versioning

#148

Post by joncalhoun »

joncalhoun wrote: post #114
munificent wrote: post #33

I'm going to comment mostly on the parts of the proposal that I think are wrong, but don't take this to be an overall negative response. I'm excited to see smart folks working on this, and package management is a really hard problem. There are no silver bullets to code reuse. Context for those who don't know: I along with Natalie Weizenbaum wrote pub[1], the package manager used for Dart. > Instead of concluding from…

You (and likely everyone else) should look at the tour as well before commenting, as I think many people are misunderstanding some of the subtler points. > If I'm starting a brand new from scratch Ruby on Rails application today, in 2017, there is no reason it should default to having me use Rails 1.0 from 2005. In the tour it states, "We've seen that when a new module must be added to a build to resolve a new import…

> When a security issue is found in a package, it's common to see point releases get released for older major/minor versions. So if foo has 1.1.0 and 1.2.0 out today and a security bug that affects both is found, the maintainers will likely release 1.1.1 and 1.2.1. This means 1.1.1 is released later than 1.2.0.

I should have addressed this in the original reply and its too late to edit now, but this isn't an issue. I downloaded vgo and verified that you CAN release a 1.1.1 AFTER 1.2.0 and it is treated correctly as far as I can tell.

See github.com/joncalhoun/vgo_main:

    $ vgo list -m -u
    MODULE                          VERSION                    LATEST
    github.com/joncalhoun/vgo_main  -                          -
    github.com/joncalhoun/vgo_demo  v1.0.1 (2018-02-20 18:26)  v1.1.0 (2018-02-20 18:25)
v1.0.1 is newer than v1.1.0, but isn't treated as the latest version. I suspect that RSC didn't mean "older" in the literal datetime sense, but rather in the context of semantic versioning where "older" means you don't release v1.3.4 AFTER you have released v1.3.5

Re: Go += Package Versioning

#149

Post by mook »

readams wrote: post #31
binarycrusader wrote: post #8

Earlier quoted context omitted.

As a long-time maintainer of various packaging systems (and co-author of one): I find myself wholly in agreement with the idea that maximal version selection is appropriate for operating systems and general software components, but not necessarily desirable for a build system. When you consider the evaluation of a dependency graph in the context of a SAT solver, you realize that the solver would consider both a minim…

We can only hope they'll at least have a tool that you can run that will bump up all the versions to specify the latest. But then of course that will make solving dependencies a lot harder since it's much more likely you'll get an unsolvable set of deps. It's hard to see how this could ever help anything. Lock files also have the major advantage that you have a hash of the thing you're downloading so you're not depen…

Because only the minimum version can be specified, you will always get a solvable set of deps. Those deps might not build but they will be solvable.

Re: Go += Package Versioning

#150

Post by kellysutton »

munificent wrote: post #33

I'm going to comment mostly on the parts of the proposal that I think are wrong, but don't take this to be an overall negative response. I'm excited to see smart folks working on this, and package management is a really hard problem. There are no silver bullets to code reuse. Context for those who don't know: I along with Natalie Weizenbaum wrote pub[1], the package manager used for Dart. > Instead of concluding from…

Now is a good time to mention the great blog post, "So you want to write a package manager," by Sam Boyer: https://medium.com/@sdboyer/so-you-want-to-write-a-package-m...
Post reply on HN