Earlier quoted context omitted.
My ideal situation is that the system should maintain authoritative versions of every package and version that is ever requested, and they should not need to be shipped. Multiple versions of a package should coexist. /usr/lib/python3.12/torch/2.1.0/ /usr/lib/python3.12/torch/2.1.1/ /usr/lib/python3.12/torch/2.1.2/ When a package requests 2.1.1 it fetches it right out of there, installing from PyPI if it doesn't. The…
I'm assuming by system you mean OS, which is a terrible, terrible idea. Dev stack and system libs should not coexist, especially because system libs should be vetted by the OS vendor, but you can't ask them to do that for dev libs. > I have to create a new conda environment for almost every ML paper that comes out That's how it's supposed to work: one env per project. As for the rest, it's more telling about the C/C+…
That causes 50 copies of the exact same version of a 1GB library to exist on my system that are all obtained from the same authority (PyPI). I have literally 50 copies of the entire set of CUDA libraries because every conda environment installs PyTorch and PyTorch includes its own CUDA.
I'm not asking the OS to maintain this, but rather the package manager ("npm" or "pip" or similar) should do so on a system-wide basis. "python" and "pip" should allow for 1 copy per officially-released version of each package to live on the system, and multiple officially-released version numbers to coexist in /usr/lib. If a dev version is being used or any version that deviates from what is on PyPI, then that should live within the project.