Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Conda: A package management disaster?

pyherald.com

Post reply on HN
Print view View original

31–40 of 234 posts

Re: Conda: A package management disaster?

#31

Post by akdor1154 »

maurosilber wrote: post #17
alsodumb wrote: post #4

As someone with admittedly no formal CS education, I've been using conda for all of my grad school and never managed to break it. I create a virtual environment for every project. I install almost all packages with pip, except for any binaries or CUDA related things from conda. I always exported the conda yaml file and managed to reproduce the code/environment including the Python version. I've seen a lot of posts ov…

I had the same experience. But you should try pixi, which is to conda what uv is to pip.

Isn't uv to conda what uv is to pip?

Re: Conda: A package management disaster?

#32

Post by The_Colonel »

It's rare to see something as systematically broken as Python package/dependencies ecosystem.

What I don't understand - what makes this so difficult to solve in Python? It seems that many other platforms solved this a long time ago - maven 2.0 was released almost 20 years ago. While it wasn't / isn't by no means perfect, its fundamentals were decent already back then.

One thing which I think messed this up from the beginning was applying the Unix philosophy with several/many individual tools as opposed to one cohesive system - requirements.txt, setuptools, pip, pipx, pipenv, venv... were always woefully inadequate, but produced a myriad of possible combinations to support. It seems like simplicity was the main motivation for such design, but these certainly seems like examples of being too simplistic for the job.

I recently tried to run a Python app (after having a couple of years break from Python) which used conda and I got lost there quickly. Project README described using conda, mamba, anaconda, conda-forge, mini-forge, mini-conda ... In the end, nothing I tried worked.

Re: Conda: A package management disaster?

#33

Post by StableAlkyne »

SamPatt wrote: post #28

Conda is the only package manager I've used on Ubuntu that intermittently and inexplicably gets stuck when installing or uninstalling. It will sometimes resolve itself if left alone for hours, but often won't. I avoid it as much as possible.

It's because of the SAT solver for dependencies. Unlike Pip, it keeps track of every package you installed and goes out of its way to avoid installing incompatible packages.

Why go through all this trouble? Because originally it was meant to be a basic "scientific Python" distribution, and needed to be strict around what's installed for reproducibility reasons.

It's IMO overkill for most users, and I suspect most scientific users don't care either - most of the time I see grads and researchers just say "fuck it" and use Pip whenever Conda refuses to get done in a timely fashion.

And the ones who do care about reproducibility are using R anyway, since there's a perception those libraries are "more correct" (read: more faithful to the original publication) than Pythonland. And TBH I can't blame them when the poster child of it is Sklearn's RandomForestRegressor not even being correctly named - it's bagged trees under the default settings, and you don't get any indication of this unless you look at that specific kwarg in the docs.

Personally, I use Conda not for reproducibility, but so all of my projects have independent environments without having to mess with containers

Re: Conda: A package management disaster?

#34

Post by shwouchk »

throwawaymaths wrote: post #24
lmz wrote: post #22

Earlier quoted context omitted.

Wasn't node the only programming language that used a subdirectory for deps by default? Ruby and Perl certainly didn't have it - although Ruby did subsequently add Bundler to gems and gems supported multiversioning.

Rust, julia, elixir

julia just store the analogue of a requirements.txt (Project.toml) and the lock file (Manifest.toml). And has its own package issues including packages regularly breaking for every minor release (although i enjoy the language and will keep using it)

Re: Conda: A package management disaster?

#35

Post by pmarreck »

I feel like a major selling point of Nix is "solving the Python dependency-hell problem" (as well as that of pretty much every other stack)

I've seen so many issues with different Python venvs from different Python project directories stepping on each others' dependencies somehow (probably because there are some global ones) that the fact that I can now just stick a basic and barely-modified-per-project Python flake.nix file in each one and be always guaranteed to have the entirely of the same dependencies available when I run it 6 months later is a win.

Re: Conda: A package management disaster?

#36

Post by pmarreck »

The_Colonel wrote: post #32

It's rare to see something as systematically broken as Python package/dependencies ecosystem. What I don't understand - what makes this so difficult to solve in Python? It seems that many other platforms solved this a long time ago - maven 2.0 was released almost 20 years ago. While it wasn't / isn't by no means perfect, its fundamentals were decent already back then. One thing which I think messed this up from the b…

Nix solves it for me. Takes a bit more effort upfront, but the payoff is "Python dependency determinism," which is pretty much unachievable in any other way, so...

Re: Conda: A package management disaster?

#37

Post by drawnwren »

pmarreck wrote: post #35

I feel like a major selling point of Nix is "solving the Python dependency-hell problem" (as well as that of pretty much every other stack) I've seen so many issues with different Python venvs from different Python project directories stepping on each others' dependencies somehow (probably because there are some global ones) that the fact that I can now just stick a basic and barely-modified-per-project Python flake.…

Do you have a publicly available copy of your flake?

Re: Conda: A package management disaster?

#38

Post by MobiusHorizons »

duped wrote: post #26
throwawaymaths wrote: post #12

Earlier quoted context omitted.

imagine being a beginner to programming and being told "use venvs" or worse, imagine being a longtime user of shells but not python and then being presented a venv as a solution to the problem that for some reason python doesn't stash deps in a subdirectory of your project

I have imagined this, because I've worked on products where our first time user had never used a CLI tool or REPL before. It's a nightmare. That said, it's no less a nightmare than every other CLI tool, because even our most basic conventions are tribal knowledge that are not taught outside of our communities and it's always an uphill battle teaching ones that may be unfamiliar to someone from a different tribe.

It is true that every field (honestly every corner of most fields) has certain specific knowledge that is both incredibly necessary to get anything done, and completely arbitrary. These are usually historical reactions to problems solved in a very particular way usually without a lot of thought, simply because it was an expedient option at the time.

I feel like venv is one such solution. A workaround that doesn’t solve the problem at its root, so much as make the symptoms manageable. But there is (at least for me) a big difference between things like that and the cool ideas that underlie shell tooling like Unix pipes. Things like jq or fzf are awesome examples of new tooling that fit beautifully in the existing paradigm but make it more powerful and useful.

Re: Conda: A package management disaster?

#39

Post by snicker7 »

alkh wrote: post #14
duped wrote: post #10

Earlier quoted context omitted.

My experience with conda is that its fine if you're the original author of whatever you're using it for and never share it with anyone else. But as a professional I usually have to pull in someone else's work and make it function on a completely different machine/environment. I've only had negative experiences with conda for that reason. IME the hard job of package management is not getting software to work in one lo…

+1. On top of that, even with the new resolver it still takes ages to resolve a dependency for me, so somethimes I end up just using pip directly. Not sure if I am doing something wrong(mb you have to manually tweak something in the configs?) but it's pretty common for me to experience this

I believe the problem is the lack of proper dependency indexing at PyPI. The SAT solvers used by poetry or pdm or uv often have to download multiple versions of the same dependencies to find a solution.

Re: Conda: A package management disaster?

#40

Post by reactordev »

pmarreck wrote: post #36
The_Colonel wrote: post #32

It's rare to see something as systematically broken as Python package/dependencies ecosystem. What I don't understand - what makes this so difficult to solve in Python? It seems that many other platforms solved this a long time ago - maven 2.0 was released almost 20 years ago. While it wasn't / isn't by no means perfect, its fundamentals were decent already back then. One thing which I think messed this up from the b…

Nix solves it for me. Takes a bit more effort upfront, but the payoff is "Python dependency determinism," which is pretty much unachievable in any other way, so...

The answer is not Yet Another Tool In The Chain. Python community itself needs to address this. Because if they don’t then you’ll have requirements.txt, setuptools, pyproject, pip, pipx, pipenv, pyenv, venv, nix.
Post reply on HN