Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Conda: A package management disaster?

pyherald.com

Post reply on HN
Print view View original

171–180 of 234 posts

Re: Conda: A package management disaster?

#171

Post by shoo »

These335 wrote: post #67

Can somebody please eli5 why it is so unanimously accepted that Python's package management is terrible? For personal projects venv + requirements.txt has never caused problems for me. For work projects we use poetry because of an assumption that we would need something better but I remain unconvinced (nothing was causing a problem for that decision to be made).

i think there might be merit to gdiamos's point that python is a popular language with a large number of users, and this might mean that python package management isn't unusually bad, but more users implies more complaints.

i think there was a significant step change improvement in python packaging around 2012, when the wheel format was introduced, which standardised distributing prebuilt platform-specific binary packages. for packages with gnarly native library dependencies / build toolchains (e.g. typical C/fortran numeric or scientific library wrapped in a layer of python bindings), once someone sets up a build server to bake wheels for target platforms, it becomes very easy to pip install them without dragging in that project's native build-from-source toolchain.

venv + pip (+ perhaps maintaining a stack of pre-built wheels for your target platform, for a commercial project where you want to be able to reproduce builds) gets most of the job done, and those ingredients have been in place for over 10 years.

around the time wheel was introduced, i was working at a company that shipped desktop software to windows machines, we used python for some of the application components. between venv + pip + wheels, it was OK.

where there were rough edges were things like: we have a dep on python wrapper library pywhatever, which requires a native library libwhatever.dll built from the c++ whatever project to be installed -- but libwhatever.dll has nothing to do with python, maybe its maintainers kindly provide an msi installer, so if you install it into a machine, it gets installed into the windows system folder, so venv isn't able to manage it & offer isolation if you need to install multiple versions for different projects / product lines, as venv only manages python packages, not arbitrary library dependencies from other ecosystems

but it's a bit much blame python for such difficulties: if you have a python library that has a native dependency on something that isnt a python package, you need to do something else to manage that dep. that's life. if you're trying to do it on windows, which doesn't have an O/S level package manager.. well, that's life.

Re: Conda: A package management disaster?

#172

Post by KolenCh »

dannyz wrote: post #108

While `uv` works amazingly well I think a lot of people don't realize that installing packages through conda (or let's say the conda-forge ecosystem) has technical advantages compared to wheels/pypi. When you install the numpy wheel through `uv` you are likely installing a pre-compiled binary that bundles openblas inside of it. When you install numpy through conda-forge, it dynamically links against a dummy blas pack…

In this sense I personally prefer pixi because of this. It is pixi like but resolves using conda channels like conda, and similar to conda it supports PyPI packages via uv.

With a background in scientific computing where many of the dependencies I managed are compiled, conda packages gives me much more control.

P.S. I’d like to point out to others to differentiate between package index and package managers. PyPI is an index (that hosts packages in a predefined format) while pip, poetry, uv are package managers that resolve and build your environments using the index.

Similarly but a bit more confusingly, conda can be understood as the index, hosted by anaconda but can also be hosted elsewhere, with different “channels” (kinda like a GitHub organization) where conda-forge is a popular one built by communities. Conda is also a reference implementation of a package manager that uses anaconda channels to resolve. Mamba is an independent, performant, drop in replacement of conda. And pixi is a different one with a different interface by the author of mamba.

Even more confusingly, there are distributions. Distributions come with a set of predefined packages together with the package manager such that you just start running things immediately (sort of like a TeXLive distribution in relation to the package manager tlmgr.) there are anaconda distributions (if you installed anaconda instead of installing conda, that’s what you get), but also Intels distribution for Python, mini forge, mambaforge, etc.

Re: Conda: A package management disaster?

#173

Post by teekert »

GuestFAUniverse wrote: post #133
teekert wrote: post #123

People here focus on Python, but to me, a bioinformatician, conda is much more, it provides 99.99% of the tools I need. Like bwa, samtools, rsem, salmon, fastqc, R. And many, many obscure tools.

And then somebody tries to install mamba via conda and that house of cards reveals itself.

I install Snakemake via miniforge which uses mambaforge to make its own envs. Biology is messy ;)

Re: Conda: A package management disaster?

#174

Post by robertlagrant »

LarsDu88 wrote: post #27
duped wrote: post #10

Earlier quoted context omitted.

My experience with conda is that its fine if you're the original author of whatever you're using it for and never share it with anyone else. But as a professional I usually have to pull in someone else's work and make it function on a completely different machine/environment. I've only had negative experiences with conda for that reason. IME the hard job of package management is not getting software to work in one lo…

Poetry is pretty slow. I think `uv` will ultimately displace it on that basis alone.

How is uv so much faster? My understanding is Poetry is slow sometimes because PyPi doesn't have all the metadata required to solve things, so it needs to download packages and then figure it out.

Re: Conda: A package management disaster?

#175

Post by chuckadams »

lmz wrote: post #22
throwawaymaths wrote: post #12

Earlier quoted context omitted.

imagine being a beginner to programming and being told "use venvs" or worse, imagine being a longtime user of shells but not python and then being presented a venv as a solution to the problem that for some reason python doesn't stash deps in a subdirectory of your project

Wasn't node the only programming language that used a subdirectory for deps by default? Ruby and Perl certainly didn't have it - although Ruby did subsequently add Bundler to gems and gems supported multiversioning.

It’s fairly common for Perl apps to use Carton (more or less a Perl clone of Bundler) to install vendored dependencies.

Re: Conda: A package management disaster?

#176

Post by LarsDu88 »

lrog wrote: post #76
LarsDu88 wrote: post #27

Earlier quoted context omitted.

Poetry is pretty slow. I think `uv` will ultimately displace it on that basis alone.

Can you recommend any good article / comparison of uv vs poetry vs conda? We've used different combinations of pipx+lockfiles or poetry, which has been so far OK'ish. But recently discovered uv and are wondering about existing experience so far across the industry.

I wrote an overview, but didn't post benchmarks https://dublog.net/blog/so-many-python-package-managers/

Re: Conda: A package management disaster?

#177

Post by traversaro »

LarsDu88 wrote: post #49
akdor1154 wrote: post #31

Earlier quoted context omitted.

Isn't uv to conda what uv is to pip?

`uv` is not a drop-in replacement for `conda` in the sense that `conda` also handles non-python dependencies, has its own distinct api server for packages, and has its own packaging yaml standard. `pixi` basically covers `conda` while using the same solver as `uv` and is written in Rust like `uv`. Now is it a good idea to have python's package management tool handle non-python packages? I think that's debateable. I p…

I am not sure pixi uses the same solver of uv, at least in general. pixi uses resolvo (https://github.com/mamba-org/resolvo) for conda packages, while uv (that in turns uses pubgrub https://github.com/pubgrub-rs/pubgrub) for pip packages.

Re: Conda: A package management disaster?

#178

Post by michaelmrose »

wakawaka28 wrote: post #134
michaelmrose wrote: post #120

Earlier quoted context omitted.

Intelligent systems simply cache and re-use versions and do stash deps for every toy project without consuming space. Also installing everything with pip is a great way to enjoy unexplainable breakage when a Doesn't work with v1 and b doesn't work with v2. It also leads to breaking Linux systems where a large part of the system is python code. Especially where user upgrades system python for no reason.

If you install a package in a fresh environment then it does actually get installed. It can be inherited from the global environment but I don't think disparate venvs that separately install a package actually share the package files. If they did, then a command executed in one tree could destroy the files in another tree. I have not done an investigation to look into this today but I think I'm right about this.

In better designed systems than python they do. To share them with python you need something with dedup. Eg BTRFS ZFS

Re: Conda: A package management disaster?

#179

Post by throwawaymaths »

shwouchk wrote: post #34
throwawaymaths wrote: post #24

Earlier quoted context omitted.

Rust, julia, elixir

julia just store the analogue of a requirements.txt (Project.toml) and the lock file (Manifest.toml). And has its own package issues including packages regularly breaking for every minor release (although i enjoy the language and will keep using it)

yep, i was wrong about julia.

Re: Conda: A package management disaster?

#180

Post by macinjosh »

whywhywhywhy wrote: post #167

Really the issue is python itself, it shouldn't be treating it's installs and packages as something that's linked and intertwined to the base operating system. People like to complain about node packages but never seen people have the trouble with them that they have with python.

This is spot on. Running some python projects on nixos is a nightmare because of this model. Especially if it’s ML related.
Post reply on HN