Conda: A package management disaster? • Hacker News Board

Re: Conda: A package management disaster?

#191

Post by AnthonBerg » Sat, Dec 21, 2024, 10:35 AM UTC

macinjosh wrote: post #181

AnthonBerg wrote: post #164

Earlier quoted context omitted.

For what it’s worth – A small technical fact: It is entirely possible to use poetry to determine the precise set of packages to install and write a requirements.txt, and then shotgun install those packages in parallel. I used a stupidly simple fish shell for loop that ran every requirements line as a pip install with an “&” to background the job and a “wait” after the loop. (iirc) Could use xargs or parallel too. Thi…

That poor package server getting 39 simultaneous pulls at the same time from one user.

This is indeed something to consider!

Not as an excuse for bad behavior but rather to consider infrastructure and expectations:

The packages might be cached locally.

There might be many servers – a CDN and/or mirrors.

Each server might have connection limits.

(The machine downloading the packages miiiiiight be able to serve as a mirror for others.)

If these are true, then it’s altruistically self-interested for everyone that the downloader gets all the packages as quickly as possible to be able to get stuff done.

I don’t know if they are true. I’d hope that local caching, CDNs and mirrors as well as reasonable connection limits were a self-evident and obviously minimal requirement for package distribution in something as arguably nation-sized as Python.

And… just… everywhere, really.

Re: Conda: A package management disaster?

#192

Post by sznio » Sat, Dec 21, 2024, 12:02 PM UTC

The domain got parked.

Re: Conda: A package management disaster?

#193

Post by _Wintermute » Sat, Dec 21, 2024, 12:38 PM UTC

StableAlkyne wrote: post #33

SamPatt wrote: post #28

Conda is the only package manager I've used on Ubuntu that intermittently and inexplicably gets stuck when installing or uninstalling. It will sometimes resolve itself if left alone for hours, but often won't. I avoid it as much as possible.

It's because of the SAT solver for dependencies. Unlike Pip, it keeps track of every package you installed and goes out of its way to avoid installing incompatible packages. Why go through all this trouble? Because originally it was meant to be a basic "scientific Python" distribution, and needed to be strict around what's installed for reproducibility reasons. It's IMO overkill for most users, and I suspect most sci…

> And the ones who do care about reproducibility are using R anyway

I worked in a pharma company with lots of R code and this comment is bringing up some PTSD. One time we spent weeks trying to recreate an "environment" to reproduce a set of results. Try installing a specific version of a package, and all the dependencies it pulls in are the latest version, whether or not they are compatible. Nobody actually records the package versions they used.

The R community are only now realising that reproducible environments are a good thing, and not everybody simply wants the latest version of a package. Packrat was a disaster, renv is slightly better.

Re: Conda: A package management disaster?

#194

Post by disgruntledphd2 » Sat, Dec 21, 2024, 1:11 PM UTC

wruza wrote: post #189

dist-epoch wrote: post #127

Earlier quoted context omitted.

This was mostly because most scientific packages didn't provide Windows binary builds for many years. Today you can just "pip install scipy" on Windows at it will just work.

Oh right, recently I started learning classic ML and “just” tried to install tensorflow, which, itself or through one of dependencies, stopped providing windows binaries since x.y.z and so my python has to be downgraded to 3.a and then other dependencies stop installing. Eventually I managed to find a proper version intersection for everything together with some shady repo, but it felt like one more requirement and I…

Tensorflow is the worst. Basically every time my python env was borked (with multiple incompatible versions of Numpy) it was down to tensorflow.

Re: Conda: A package management disaster?

#195

Post by disgruntledphd2 » Sat, Dec 21, 2024, 1:15 PM UTC

throwawaymaths wrote: post #9

alsodumb wrote: post #4

As someone with admittedly no formal CS education, I've been using conda for all of my grad school and never managed to break it. I create a virtual environment for every project. I install almost all packages with pip, except for any binaries or CUDA related things from conda. I always exported the conda yaml file and managed to reproduce the code/environment including the Python version. I've seen a lot of posts ov…

i think you got lucky and fell into best practices on your first go > except for any binaries or CUDA related things from conda doing the default thing with cuda related python packages used to often result in "fuck it, reinstall linux". admittedly, i dont know how it is now. i have one machine that runs python with a gpu and it runs only one python program.

> doing the default thing with cuda related python packages used to often result in "fuck it, reinstall linux"

From about 2014-17 you are correct, but it appears (on ubuntu at least), that it mostly works now. Maybe I've just gotten better at dealing with the pain though...

Re: Conda: A package management disaster?

#196

Post by disgruntledphd2 » Sat, Dec 21, 2024, 1:18 PM UTC

kristianp wrote: post #169

marcosdumay wrote: post #160

Earlier quoted context omitted.

> For personal projects venv + requirements.txt has never caused problems for me. That means you don't use Windows. What is great. Keep not using it. But most people will have a different experience.

Wait, are most of the people having problems in this thread using windows? It's been mentioned a couple of times, but not by most.

There's a bunch of people who tend to have problems with Python dep management: - Windows users

- DS/compiled libs users (mostly Fortran/Cuda/C++)

- Anyone with dependencies on native/non python libraries.

Conda definitely helps with 2 and 3 above, and uv is at least a nice, fast API over pip (which is better since it started doing dependency checking and binary wheels).

More generally, lots of the issues come from the nature of python as a glue language over compiled libraries, which is a relatively harder problem in general.

Re: Conda: A package management disaster?

#197

Post by lmz » Sat, Dec 21, 2024, 1:19 PM UTC

chuckadams wrote: post #175

lmz wrote: post #22

Earlier quoted context omitted.

Wasn't node the only programming language that used a subdirectory for deps by default? Ruby and Perl certainly didn't have it - although Ruby did subsequently add Bundler to gems and gems supported multiversioning.

It’s fairly common for Perl apps to use Carton (more or less a Perl clone of Bundler) to install vendored dependencies.

Oh that's nice. When I last looked (quite a long time ago), local::lib seemed to be the recommended way, and that seemed a bit more fiddly than python's virtualenv.

Re: Conda: A package management disaster?

#198

Post by disgruntledphd2 » Sat, Dec 21, 2024, 1:21 PM UTC

dagw wrote: post #87

invaliduser wrote: post #82

Earlier quoted context omitted.

venv + requirements.txt has worked for every single python project I made for the last 2 years (I'm new to python). Only issue I had was when using a newish python version and not having a specific library released yet for this new version, but downgrading python solved this. Being new to the ecosystem I have no clue why people would use Conda and why it matters. I tried it, but was left bewildered, not understanding…

I have no clue why people would use Conda The big thing to realise is that when Conda first was released it was the only packaging solution that truly treated Windows as a first class citizen and for a long time was really the only way to easily install python packages on Windows. This got it a huge following in the scientific community where many people don't have a solid programming/computer background and generall…

> it manages your entire dependency chain down to the C level in a cross platform way.

I love conda, but this isn't true. You need to opt-in to a bunch of optional compiler flags to get a portable yml file, and then it can often fail on different OS's/versions anyway.

I haven't done too much of this since 2021 (gave up and used containers instead) but it was a nightmare getting windows/mac builds to work correctly with conda back then.

Re: Conda: A package management disaster?

#199

Post by chuckadams » Sat, Dec 21, 2024, 1:35 PM UTC

lmz wrote: post #197

chuckadams wrote: post #175

Earlier quoted context omitted.

It’s fairly common for Perl apps to use Carton (more or less a Perl clone of Bundler) to install vendored dependencies.

Oh that's nice. When I last looked (quite a long time ago), local::lib seemed to be the recommended way, and that seemed a bit more fiddly than python's virtualenv.

Carton uses local::lib under the covers. I found local::lib far less fiddly than virtualenv myself, but it just doesn't try to do as much as virtualenv. These days I do PHP for a living, and for all the awfulness in php, they did nail it with composer.

Re: Conda: A package management disaster?

#200

Post by setopt » Sat, Dec 21, 2024, 2:31 PM UTC

fire_lake wrote: post #190

dannyz wrote: post #108

While `uv` works amazingly well I think a lot of people don't realize that installing packages through conda (or let's say the conda-forge ecosystem) has technical advantages compared to wheels/pypi. When you install the numpy wheel through `uv` you are likely installing a pre-compiled binary that bundles openblas inside of it. When you install numpy through conda-forge, it dynamically links against a dummy blas pack…

Is there a reason this behaviour couldn’t be implemented in uv? Is this beyond what the pyproject.toml spec supports?

My guess is that the difference is more that PyPI intends to be a Python package repository, and thus I don’t think you can just upload say a binary copy of MKL without accompanying Python code. It’s originally a source-based repository with binary wheels being an afterthought. (I still remember the pre-wheel nightmare `pip install numpy` used to give, when it required compiling the C/C++/Fortran pieces which often failed and was often hard to debug…)

But Anaconda and CondaForge are general package repository, they are not Python-specific but are happy to be used for R, Julia, C/C++/Fortran binaries, etc. it’s primarily a binary-based repository. For example, you can `conda install python` but you can’t `pip install python`.

I don’t know if there is any technical barrier or just a philosophical barrier. Clearly, Pip handles binary blobs inside of Python packages fine, so I would guess the latter but am happy to be corrected :).