Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Conda: A package management disaster?

pyherald.com

Post reply on HN
Print view View original

61–70 of 234 posts

Re: Conda: A package management disaster?

#61

Post by fluorinerocket »

alsodumb wrote: post #4

As someone with admittedly no formal CS education, I've been using conda for all of my grad school and never managed to break it. I create a virtual environment for every project. I install almost all packages with pip, except for any binaries or CUDA related things from conda. I always exported the conda yaml file and managed to reproduce the code/environment including the Python version. I've seen a lot of posts ov…

Same but I try to use conda to install everything first, and only use pip as a last resort. If pip only installs the package and no dependency it's fine

Re: Conda: A package management disaster?

#62

Post by Balinares »

StableAlkyne wrote: post #33
SamPatt wrote: post #28

Conda is the only package manager I've used on Ubuntu that intermittently and inexplicably gets stuck when installing or uninstalling. It will sometimes resolve itself if left alone for hours, but often won't. I avoid it as much as possible.

It's because of the SAT solver for dependencies. Unlike Pip, it keeps track of every package you installed and goes out of its way to avoid installing incompatible packages. Why go through all this trouble? Because originally it was meant to be a basic "scientific Python" distribution, and needed to be strict around what's installed for reproducibility reasons. It's IMO overkill for most users, and I suspect most sci…

> Personally, I use Conda not for reproducibility, but so all of my projects have independent environments without having to mess with containers

A perfectly reasonable goal, yup! Thankfully not one that, in fact, requires conda. Automated per-project environments are increasingly the default way of doing things in Python, thank goodness. It's been a long time coming.

Re: Conda: A package management disaster?

#63

Post by dxuh »

The_Colonel wrote: post #32

It's rare to see something as systematically broken as Python package/dependencies ecosystem. What I don't understand - what makes this so difficult to solve in Python? It seems that many other platforms solved this a long time ago - maven 2.0 was released almost 20 years ago. While it wasn't / isn't by no means perfect, its fundamentals were decent already back then. One thing which I think messed this up from the b…

I think at least part of it is that there are so many solutions for Python packaging, which are often intermixed or only half-supported by developers. It's a tough ask to provide dedicated support for pip, conda, poetry and what else is there plus a couple different ways to create virtual environments. Of course if you do everything right, you set it up once (if even that) and it just keeps working forever, but it is never like that. Someone will use a tool you haven't and it will not work correctly and they will find a workaround and the mess starts.

Also I think that Python packages are sometimes distributed as shared libraries is a problem. When I think about conan or vcpkg (package managers for C and C++), they usually suck because some dependencies are available on some platforms and not on others or even in one version on one platform and in another version on another and you get messes all around if you need to support multiple platforms.

I think generally binary package managers are almost always bad* and source based package managers almost always work well (I think those are essentially easy mode).

*: unless they maintain a source package of their own that they actually support and have a fixed set of well-supported platforms (like system package managers on most Linux distros do).

Re: Conda: A package management disaster?

#64

Post by neonsunset »

The_Colonel wrote: post #43
reactordev wrote: post #40

Earlier quoted context omitted.

The answer is not Yet Another Tool In The Chain. Python community itself needs to address this. Because if they don’t then you’ll have requirements.txt, setuptools, pyproject, pip, pipx, pipenv, pyenv, venv, nix.

Agreed. Often there's a quite tight coupling between the core platform devs and package management - node.js has its npm, rust cargo, go has one as well and for the most part it seems to have worked out fine for them. Java and .NET (and I think PHP) are different in the sense that the package management systems have no relation to the platform developers, but industry standards (maven, gradle, NuGET, Composer) still…

FWIW Nuget to .NET is what Cargo crates are to Rust instead of what Maven and Gradle are to Java. The package manager is just a part of the SDK.

Even the CLI workflow is identical: dotnet add package / cargo add (.NET had it earlier too, it's nice that Cargo now also has it).

Re: Conda: A package management disaster?

#65

Post by eviks »

reactordev wrote: post #40
pmarreck wrote: post #36

Earlier quoted context omitted.

Nix solves it for me. Takes a bit more effort upfront, but the payoff is "Python dependency determinism," which is pretty much unachievable in any other way, so...

The answer is not Yet Another Tool In The Chain. Python community itself needs to address this. Because if they don’t then you’ll have requirements.txt, setuptools, pyproject, pip, pipx, pipenv, pyenv, venv, nix.

Well, there is no way to address it then, no magic will eliminate everything from the list.

So another tool isn't meaningfully different (and it can be the answer): if "the community" migrates to the new tool it wouldn't matter that there's a dozen of other unused tools.

Same thing if "the community" fixes an existing tool and migrates to it: other unused tools will still exist

Re: Conda: A package management disaster?

#66

Post by fluorinerocket »

theamk wrote: post #7
alsodumb wrote: post #4

As someone with admittedly no formal CS education, I've been using conda for all of my grad school and never managed to break it. I create a virtual environment for every project. I install almost all packages with pip, except for any binaries or CUDA related things from conda. I always exported the conda yaml file and managed to reproduce the code/environment including the Python version. I've seen a lot of posts ov…

You are doing something right, author does some pretty unusual things: - Setup custom kernels in Jupyter Notebook - Hardlink the environments, then install same packages via pip in one and conda in others - install conda inside conda (!!!) and enter nested environment - Use tox within conda I believe as long as you treat the environments as "cattle" (if it goes bad, remove it and re-create from yaml file), you should…

Yep nuke the bad env and start over. Conda is great only problem are when a package is not available on conda forge or you have to compile and install with setup.py. But then you can blow the env away and start over.

Re: Conda: A package management disaster?

#67

Post by These335 »

Can somebody please eli5 why it is so unanimously accepted that Python's package management is terrible? For personal projects venv + requirements.txt has never caused problems for me. For work projects we use poetry because of an assumption that we would need something better but I remain unconvinced (nothing was causing a problem for that decision to be made).

Re: Conda: A package management disaster?

#68

Post by Balinares »

LarsDu88 wrote: post #49
akdor1154 wrote: post #31

Earlier quoted context omitted.

Isn't uv to conda what uv is to pip?

`uv` is not a drop-in replacement for `conda` in the sense that `conda` also handles non-python dependencies, has its own distinct api server for packages, and has its own packaging yaml standard. `pixi` basically covers `conda` while using the same solver as `uv` and is written in Rust like `uv`. Now is it a good idea to have python's package management tool handle non-python packages? I think that's debateable. I p…

Bookmarking. Thanks for sharing the link, looks like a great overview of that particular tragic landscape. :)

Also crossing fingers that uv ends up being the last one standing when the comprehensive amounts of dust here settle. But until then, I'll look into pixi, on the off chance it minimizes some of my workplace sorrows.

Re: Conda: A package management disaster?

#69

Post by japanuspus »

The problem conda solved that nothing had solved before was installing binary dependencies on MS Windows.

Before conda, getting a usable scipy install up and running on MS Windows was a harrowing experience. And having two independent installations was basically impossible. The real hard work that went into conda was reverse engineering all the nooks and crannies of the DLL loading heuristics, to allow it to ensure that you loaded what you intended.

If you are working on macOS and deploying to some *nix in the cloud, you are unlikely to find any value in this. But in ten years as lead on a large tool that was deployed to personal (Windows) laptops in a corporate environment, I did not find anything that beat conda.

Re: Conda: A package management disaster?

#70

Post by oivey »

Besides the horrendous formatting, some stuff in this article seem incorrect or irrelevant. Like, is this even possible?

> A single Anaconda distribution may have multiple NumPy versions installed at the same time, although only one will be available to the Python process (note that this means that sub-processes created in this Python process won’t necessarily have the same version of NumPy!).

I’m pretty sure there’s not, but maybe there is some insane way to cause subprocesses to do this. Besides that, under the authors definition, different Python virtualenvs also install multiple copies of libraries in the same way conda does.

The comments about Jupyter also seem very confused. It’s hard to make heads or tails of exactly what the author is saying. There might be some misunderstandings of how Jupyter kernels select environments.

> Final warning: no matter how ridiculous this is: the current directory in Python is added to the module lookup path, and it precedes every other lookup location. If, accidentally, you placed a numpy.py in the current directory of your Python process – that is going to be the numpy module you import.

This has nothing to do with conda.

Post reply on HN