Can a Rust binary use incompatible versions of the same library?
1–10 of 89 posts
Re: Can a Rust binary use incompatible versions of the same library?
#2I thought this was about loading two incompatible versions of a shared object into the same address space at first :-)
The author correctly contrasts Rust (and NPM's) behavior with that of Python/pip, where only one version per package name is allowed. The Python packaging ecosystem could in theory standardize a form of package name mangling wherein multiple versions could be imported simultaneously (akin to what's currently possible with multiple vendored versions), but that would likely be a significant undertaking given that a lot of applications probably - accidentally - break the indirect relationship and directly import their transitive dependencies.
(The more I work in Python, the more I think that Python's approach is actually a good one: preventing multiple versions of the same package prevents dependency graph spaghetti when every subdependency depends on a slightly different version, and provides a strong incentive to keep public API surfaces small and flexible. But I don't think that was the intention, more of an accidental perk of an otherwise informal approach to packaging.)
Re: Can a Rust binary use incompatible versions of the same library?
#3I thought this was about loading two incompatible versions of a shared object into the same address space at first :-) The author correctly contrasts Rust (and NPM's) behavior with that of Python/pip, where only one version per package name is allowed. The Python packaging ecosystem could in theory standardize a form of package name mangling wherein multiple versions could be imported simultaneously (akin to what's c…
> dependency graph spaghetti
The worst spaghetti comes from hard dependencies on minor versions and revisions.
I will die on the hill that you should only ever specify dependencies on “at least this major-minor (and optionally and rarely revision for a bugfix)” in whatever the syntax is for your preferred language. Excepting of course a known incompatibility with a specific version or range of versions, and/or developers who refuse to get on the semver bandwagon who should collectively be rounded up and yelled at.
In Rust, Cargo makes this super easy: “x.y.z” means “>= x.y.z, It’s fine to ship a generated lock file that locks everything to a fixed, known-good version of all your dependencies. But you should be able to trivially run an update that will bring everything to the latest minor and revision (and alert on newer major versions).
Re: Can a Rust binary use incompatible versions of the same library?
#4I thought this was about loading two incompatible versions of a shared object into the same address space at first :-) The author correctly contrasts Rust (and NPM's) behavior with that of Python/pip, where only one version per package name is allowed. The Python packaging ecosystem could in theory standardize a form of package name mangling wherein multiple versions could be imported simultaneously (akin to what's c…
Another thing I appreciate about this in the Python world is it avoids an issue I've seen in node a lot, which is people being too clever by a half and pre-emptively adding major version bounds to their library. So foo depends on "barThe end result of this is that you end up with some random library in your stack (4 transitive layers deep because of course it is) holding back stuff like chokadir in a huge chunk of your dep tree for... no real good reason. So you now have several copies of a huge library.
Of course new major versions might break your usage! Minor versions might as well! Patch versions too sometimes! Upper bounds pre-emptively set help mainly in one thing, and that's reducing the number of people who would help "beta-test" new major versions because they don't care enough to pin their own dependencies.
Re: Can a Rust binary use incompatible versions of the same library?
#5I thought this was about loading two incompatible versions of a shared object into the same address space at first :-) The author correctly contrasts Rust (and NPM's) behavior with that of Python/pip, where only one version per package name is allowed. The Python packaging ecosystem could in theory standardize a form of package name mangling wherein multiple versions could be imported simultaneously (akin to what's c…
> (The more I work in Python, the more I think that Python's approach is actually a good one ...)
I've come to the opposite conclusion. I've "git cloned" several programs in both python and ruby (which has the same behaviour) only to discover that I can't actually install the project's dependencies. The larger your gemfile / requirements.txt is, the more likely this is to happen. All it takes is a couple packages in your tree to update their own dependencies out of sync with one another and you can run into this problem. A build that worked yesterday doesn't work today. Not because anyone made a mistake - but just because you got unlucky. Ugh.
Its a completely unnecessary landmine. Worse yet, new developers (or new teammembers) are very likely to run into this problem as it shows up when you're getting your dev environment setup.
This problem is entirely unnecessary. In (almost) every way, software should treat foo-1.x.x as a totally distinct package from foo-2.x.x. They're mutually incompatible anyway, and semantically the only thing they share is their name. There's no reason both packages can't be loaded into the package namespace at the same time. No reason but the mistakes of shortsighted package management systems.
RAM is cheap. My attention is expensive. Print a warning if you must, and I'll fix it when I feel like it.
Re: Can a Rust binary use incompatible versions of the same library?
#6I thought this was about loading two incompatible versions of a shared object into the same address space at first :-) The author correctly contrasts Rust (and NPM's) behavior with that of Python/pip, where only one version per package name is allowed. The Python packaging ecosystem could in theory standardize a form of package name mangling wherein multiple versions could be imported simultaneously (akin to what's c…
> dependency graph spaghetti The worst spaghetti comes from hard dependencies on minor versions and revisions. I will die on the hill that you should only ever specify dependencies on “at least this major-minor (and optionally and rarely revision for a bugfix)” in whatever the syntax is for your preferred language. Excepting of course a known incompatibility with a specific version or range of versions, and/or develo…
There's a subtle point there though. When you rely on something that was introduced in x.y.z, stating that your version requirement is x.y.0 is an error that can easily cause downstream breakage.
Re: Can a Rust binary use incompatible versions of the same library?
#7I thought this was about loading two incompatible versions of a shared object into the same address space at first :-) The author correctly contrasts Rust (and NPM's) behavior with that of Python/pip, where only one version per package name is allowed. The Python packaging ecosystem could in theory standardize a form of package name mangling wherein multiple versions could be imported simultaneously (akin to what's c…
> (The more I work in Python, the more I think that Python's approach is actually a good one ...) I've come to the opposite conclusion. I've "git cloned" several programs in both python and ruby (which has the same behaviour) only to discover that I can't actually install the project's dependencies. The larger your gemfile / requirements.txt is, the more likely this is to happen. All it takes is a couple packages in…
I'm not saying this hasn't happened to you, but I'm curious: are you working with scientific Python codebases or similar? I've done Python development off and on for the last ~10 years, and I think I can count the number of times I've had transitive conflicts on a single hand. But I almost never touch scientific/statistical/etc. Python codebases, so I'm curious is this is a discipline/practice concern in different subsets of the ecosystem.
(One of the ways I have seen this happen in this past is people attempting to use multiple requirements sources without synchronizing them or resolving them simultaneously. That's indeed a highway to pain city, and it's why modern Python packaging emphasizes either using a single standard metadata file like pyproject.toml or a fully locked environment specification like a frozen requirements file.)
Re: Can a Rust binary use incompatible versions of the same library?
#8Earlier quoted context omitted.
> dependency graph spaghetti The worst spaghetti comes from hard dependencies on minor versions and revisions. I will die on the hill that you should only ever specify dependencies on “at least this major-minor (and optionally and rarely revision for a bugfix)” in whatever the syntax is for your preferred language. Excepting of course a known incompatibility with a specific version or range of versions, and/or develo…
There's a subtle point there though. When you rely on something that was introduced in x.y.z, stating that your version requirement is x.y.0 is an error that can easily cause downstream breakage.
I’m confused. If you rely on a feature introduced in X.y.z why would you specify X.y.0 to begin with (and not just X.y.z)?
In practice, usual rust projects that have not put a ton of work into their dependencies encode X.y.z in Cargo.toml matching the current release at the time they developed the system. So you get at worst an unnecessarily higher version requirement but never a lower one.
Moreover, rust semver would normally imply that new features should only be introduced in X.y releases, so this doesn’t really happen in practice!
Re: Can a Rust binary use incompatible versions of the same library?
#9Earlier quoted context omitted.
> (The more I work in Python, the more I think that Python's approach is actually a good one ...) I've come to the opposite conclusion. I've "git cloned" several programs in both python and ruby (which has the same behaviour) only to discover that I can't actually install the project's dependencies. The larger your gemfile / requirements.txt is, the more likely this is to happen. All it takes is a couple packages in…
I'm not saying this hasn't happened to you, but I'm curious: are you working with scientific Python codebases or similar? I've done Python development off and on for the last ~10 years, and I think I can count the number of times I've had transitive conflicts on a single hand. But I almost never touch scientific/statistical/etc. Python codebases, so I'm curious is this is a discipline/practice concern in different su…
I've encountered the same problem with Python codebases in the LLM / machine learning space. The requirements.txt files for those projects are full of unversioned dependencies, including Git repositories at some floating ref (such as master/HEAD).
In the easy cases, digging through the PyPI version history to identify the latest version as of some date is enough to get a working install (as far as I can tell -- maybe it's half-broken and I only use the working half?). In the hard cases, it may take an entire day to locate a CI log or contemporary bug report or something that lists out all the installed package versions.
It doesn't help that every Python-based project seems to have its own bespoke packaging system. It's never just pip + requirements.txt, it'll have a Dockerfile with `apt update`, or some weird meta-packaging thing like Conda that adds it own layers of non-determinism. Overall the feeling is that it was only barely holding together on the author's original machine, and getting it to build anywhere else is pure luck.
For example: https://github.com/AUTOMATIC1111/stable-diffusion-webui/blob... (with some discussion at https://github.com/AUTOMATIC1111/stable-diffusion-webui/disc...)
Re: Can a Rust binary use incompatible versions of the same library?
#10For fun, you could add this to Python and I think it would it cover a lot of edge cases?
You would need:
A function v_tree_install(spec) which installs a versioned pypi package like “foo=3.2” and all its dependencies in its own tree, rather than in site-packages.
Another pair of functions v_import and v_from_import to wrap importlib with a name, version, and symbols. These functions know how to find the versioned package in its special tree and push that tree to sys.path before starting the import.
To cover the case for when the imported code has dynamic imports you could also wrap any callable code (functions, classes) with a wrapper that also does the sys.push/pop before/after each call.
You then replace third party imports in your code with calls assigning to symbols in your module:
# import foo
foo = v_import(“foo==3.2”)
# from foo import bar, baz as q
bar, q = v_from_import(
“foo>=3.3”,
“bar”,
“baz”,
)
Edit: …and you’d need to edit the sys.modules cache too, or purge it after each “clever” import?