Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Conda: A package management disaster?

pyherald.com

Post reply on HN
Print view View original

81–90 of 234 posts

Re: Conda: A package management disaster?

#81

Post by maxnoe »

Balinares wrote: post #58
BrenBarn wrote: post #52

Nothing in the "article" seems to support the title. A lot of it is just about Python packaging in general, or about problems when mixing conda- and pip-installed packages. In my experience conda is enormously superior to the standard Python packaging tools.

If we're doing anecdotal evidence, then mine is that conda is by far the worst of the main Python packaging solutions in use. The absurd slowness and incompatibility with the entire rest of the Python world are only the visible tip of that iceberg. To the best of my ability to tell, conda largely exists to make up for endemic deficiencies in Windows software distribution toolchains (not Python specific) and sadly it'…

The "absurd slowness" is gone since more than a year when it switched to using the libmamba solver.

Re: Conda: A package management disaster?

#82

Post by invaliduser »

These335 wrote: post #67

Can somebody please eli5 why it is so unanimously accepted that Python's package management is terrible? For personal projects venv + requirements.txt has never caused problems for me. For work projects we use poetry because of an assumption that we would need something better but I remain unconvinced (nothing was causing a problem for that decision to be made).

venv + requirements.txt has worked for every single python project I made for the last 2 years (I'm new to python). Only issue I had was when using a newish python version and not having a specific library released yet for this new version, but downgrading python solved this.

Being new to the ecosystem I have no clue why people would use Conda and why it matters. I tried it, but was left bewildered, not understanding the benefits.

Re: Conda: A package management disaster?

#83

Post by mook »

TheAceOfHearts wrote: post #73
The_Colonel wrote: post #32

It's rare to see something as systematically broken as Python package/dependencies ecosystem. What I don't understand - what makes this so difficult to solve in Python? It seems that many other platforms solved this a long time ago - maven 2.0 was released almost 20 years ago. While it wasn't / isn't by no means perfect, its fundamentals were decent already back then. One thing which I think messed this up from the b…

The problem is that Python refuses to take responsibility for the whole ecosystem. One of the biggest success stories in programming language development has been Rust's realization that all of it matters: language, version management, package management, and build tools. To have a truly outstanding experience you need to take responsibility for the whole ecosystem. Python and many other older languages just focus on…

You can kind of see this in golang. Originally it came with stuff to download dependencies, but it had major issues with more complex projects and some community-made tools became popular instead. But it meant that multiple tools were used in different places and it was kind of a mess. Later on a new system was done in the default toolchain and even though it has problems it's good enough that it's now surprising for somebody to use non-default tools.

Re: Conda: A package management disaster?

#84

Post by tasuki »

These335 wrote: post #67

Can somebody please eli5 why it is so unanimously accepted that Python's package management is terrible? For personal projects venv + requirements.txt has never caused problems for me. For work projects we use poetry because of an assumption that we would need something better but I remain unconvinced (nothing was causing a problem for that decision to be made).

In your requirements.txt, do you pin the concrete versions or leave some leeway?

If you aren't precise, you're gonna get different versions of your dependencies on different machines. Oops.

Pinning concrete versions is of course better, but then there isn't a clear and easy way to upgrade all dependencies and check whether ci still passes.

Re: Conda: A package management disaster?

#85

Post by BiteCode_dev »

The_Colonel wrote: post #32

It's rare to see something as systematically broken as Python package/dependencies ecosystem. What I don't understand - what makes this so difficult to solve in Python? It seems that many other platforms solved this a long time ago - maven 2.0 was released almost 20 years ago. While it wasn't / isn't by no means perfect, its fundamentals were decent already back then. One thing which I think messed this up from the b…

Python packaging is broken mostly because bootstrapping is broken, and it cascades to packaging but people don't know the bootstrapping is responsible and blame packaging.

Not saying packaging doesn't have faults, but on it's own, on a good Python setup, it's actually better than average. But few people have a good setup. In fact most people don't know what a good setup looks like.

And here is why bootstrapping is broken: https://www.bitecode.dev/p/why-is-the-python-installation-pr...

Re: Conda: A package management disaster?

#86

Post by BiteCode_dev »

Conda used to be a life saver when years and years ago, compiled extensions were hard to install because you had to compile them yourself.

Nowadays, thanks to wheels being numerous and robust, the appeal of anaconda is disappearing for most users except for some exotic mixes.

conda itself now causes more trouble than it solves as it's slow, and lives in its own incompatible world.

But anaconda solves a different problem now that nobody else solves, and that's managing Python for big corporation. This is worth a lot of money to big structures that need to control packages origin, permissions, updates, and so on, at scale.

So it thrives there.

Re: Conda: A package management disaster?

#87

Post by dagw »

invaliduser wrote: post #82
These335 wrote: post #67

Can somebody please eli5 why it is so unanimously accepted that Python's package management is terrible? For personal projects venv + requirements.txt has never caused problems for me. For work projects we use poetry because of an assumption that we would need something better but I remain unconvinced (nothing was causing a problem for that decision to be made).

venv + requirements.txt has worked for every single python project I made for the last 2 years (I'm new to python). Only issue I had was when using a newish python version and not having a specific library released yet for this new version, but downgrading python solved this. Being new to the ecosystem I have no clue why people would use Conda and why it matters. I tried it, but was left bewildered, not understanding…

I have no clue why people would use Conda

The big thing to realise is that when Conda first was released it was the only packaging solution that truly treated Windows as a first class citizen and for a long time was really the only way to easily install python packages on Windows. This got it a huge following in the scientific community where many people don't have a solid programming/computer background and generally still ran Windows on their desktops.

Conda also not only manages your python interpreter and python libraries, it manages your entire dependency chain down to the C level in a cross platform way. If a python library is a wrapper around a C library then pip generally won't also install the C library, Conda (often) will. If you have two different projects that need two different versions of GDAL or one needs OpenBLAS and one that needs MKL, or two different versions of CUDA then Conda (attempts to) solve that in a way that transparently works on Windows, Linux and MacOS. Using venv + requirements.txt you're out of luck and will have to fall back on doing everything in its own docker container.

Conda lets you mix private and public repos as well as mirroring public packages on-perm in a transparent way much smoother than pip, and has tools for things like audit logging, find grained access control, package signing and centralised controls and policy management.

Conda also has support for managing multi-language projects. Does your python project need nodejs installed to build the front-end? Conda can also manage your nodejs install. Using R for some statistical analysis in some part of your data pipeline? Conda will mange your R install. Using a Java library for something? Conda will make sure everybody has the right version of Java installed.

Also, it at least used to be common for people writing numeric and scientific libraries to release Conda packages first and then only eventually publish on PyPi once the library was 'done' (which could very well be never). So if you wanted the latest cutting edge packages in many fields you needed Conda.

Now there are obviously a huge class a projects where none of these features are needed and mean nothing. If you don't need Conda, then Conda is no longer the best answer. But there are still a lot of niche things Conda still does better than any other tool.

Re: Conda: A package management disaster?

#88

Post by drwu »

invaliduser wrote: post #82
These335 wrote: post #67

Can somebody please eli5 why it is so unanimously accepted that Python's package management is terrible? For personal projects venv + requirements.txt has never caused problems for me. For work projects we use poetry because of an assumption that we would need something better but I remain unconvinced (nothing was causing a problem for that decision to be made).

venv + requirements.txt has worked for every single python project I made for the last 2 years (I'm new to python). Only issue I had was when using a newish python version and not having a specific library released yet for this new version, but downgrading python solved this. Being new to the ecosystem I have no clue why people would use Conda and why it matters. I tried it, but was left bewildered, not understanding…

I am using Conda to build binary modules for different Python versions.

As user of the modules, venv is sufficient.

Re: Conda: A package management disaster?

#89

Post by fransje26 »

jszymborski wrote: post #53
alsodumb wrote: post #4

As someone with admittedly no formal CS education, I've been using conda for all of my grad school and never managed to break it. I create a virtual environment for every project. I install almost all packages with pip, except for any binaries or CUDA related things from conda. I always exported the conda yaml file and managed to reproduce the code/environment including the Python version. I've seen a lot of posts ov…

God forbid you should require conda-forge and more than three packages lest the dependency resolver take longer than the heat death of the planet to complete.

Install mamba first?

Re: Conda: A package management disaster?

#90

Post by ElectricalUnion »

neonsunset wrote: post #64
The_Colonel wrote: post #43

Earlier quoted context omitted.

Agreed. Often there's a quite tight coupling between the core platform devs and package management - node.js has its npm, rust cargo, go has one as well and for the most part it seems to have worked out fine for them. Java and .NET (and I think PHP) are different in the sense that the package management systems have no relation to the platform developers, but industry standards (maven, gradle, NuGET, Composer) still…

FWIW Nuget to .NET is what Cargo crates are to Rust instead of what Maven and Gradle are to Java. The package manager is just a part of the SDK. Even the CLI workflow is identical: dotnet add package / cargo add (.NET had it earlier too, it's nice that Cargo now also has it).

Wait, newer versions of thr JDK Java SDK now bundle maven and gradle? Why does everyone use mvnw/gradlew for?
Post reply on HN