Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Conda: A package management disaster?

pyherald.com

Post reply on HN
Print view View original

131–140 of 234 posts

Re: Conda: A package management disaster?

#131

Post by jokethrowaway »

The_Colonel wrote: post #32

It's rare to see something as systematically broken as Python package/dependencies ecosystem. What I don't understand - what makes this so difficult to solve in Python? It seems that many other platforms solved this a long time ago - maven 2.0 was released almost 20 years ago. While it wasn't / isn't by no means perfect, its fundamentals were decent already back then. One thing which I think messed this up from the b…

my approach is to ignore all the *conda stuff and:

yay -S python-virtualenv # I'm on arch, do not confuse with 12 similarly named alternatives pyenv virtualenv 3.10 random-python-crap pyenv local 3.10.6/envs/random-python-crap pip install -r requirements.txt

and it works (sometimes deps are in some other places, or you have to pass -c constraints.txt or there is no file and you need to create it in various ways)

At least by not using local .env directories, I always know where to find them.

I install a lot of AI projecst so I have around 1TB just for the same python dependencies installed over and over again.

Sometimes I can get away with trying to use the same venv for two different projects but 8/10 deps get broken.

Re: Conda: A package management disaster?

#132

Post by HideousKojima »

skeledrew wrote: post #106
fire_lake wrote: post #79

Earlier quoted context omitted.

I don’t think the Python community has a culture of thinking about software engineering in a principled and systematic way like you would see in places like Haskell, Rust or Clojure communities. Pythons strength (and weakness) is an emphasis on quick scripts, data science and statistics. There’s simply not the right people with the right mindset.

Not "right" or "wrong" mindset. Just different.

No it's wrong because of the mess it makes. Which makes even the things that that crowd of people wants to focus on, like wuick scripts or data science, harder.

Re: Conda: A package management disaster?

#133

Post by GuestFAUniverse »

teekert wrote: post #123

People here focus on Python, but to me, a bioinformatician, conda is much more, it provides 99.99% of the tools I need. Like bwa, samtools, rsem, salmon, fastqc, R. And many, many obscure tools.

And then somebody tries to install mamba via conda and that house of cards reveals itself.

Re: Conda: A package management disaster?

#134

Post by wakawaka28 »

michaelmrose wrote: post #120
wakawaka28 wrote: post #18

Earlier quoted context omitted.

Beginners in Python typically don't need venvs. They can just install a few libraries (or no libraries even) to get started. If you truly need venvs then you're either past the initial learning phase or you're learning how to run Python apps instead of learning Python itself. For some libraries, it is not acceptable to stash the dependencies for every single toy app you use. I don't know how much space TensorFlow or…

Intelligent systems simply cache and re-use versions and do stash deps for every toy project without consuming space. Also installing everything with pip is a great way to enjoy unexplainable breakage when a Doesn't work with v1 and b doesn't work with v2. It also leads to breaking Linux systems where a large part of the system is python code. Especially where user upgrades system python for no reason.

If you install a package in a fresh environment then it does actually get installed. It can be inherited from the global environment but I don't think disparate venvs that separately install a package actually share the package files. If they did, then a command executed in one tree could destroy the files in another tree. I have not done an investigation to look into this today but I think I'm right about this.

Re: Conda: A package management disaster?

#135

Post by zmakrr »

The_Colonel wrote: post #32

It's rare to see something as systematically broken as Python package/dependencies ecosystem. What I don't understand - what makes this so difficult to solve in Python? It seems that many other platforms solved this a long time ago - maven 2.0 was released almost 20 years ago. While it wasn't / isn't by no means perfect, its fundamentals were decent already back then. One thing which I think messed this up from the b…

PyPI was always broken due to weird ideas for problems that were long solved in other languages or distributions. They had/have the backing of fastly.net, which created an arrogant and incompetent environment where people listed to no one.

Conda suffers from the virtual environment syndrome. Virtual environments are always imperfect and confusing. System libraries sometime leak through. The "scientific" Python stack has horrible mixtures of C/C++/Cython etc., all poorly written ad difficult to build.

Projects deteriorated in their ability to build from source due to the availability of binary wheels and the explosion of build systems. In 2010 there was a good chance that building a C project worked. Now you fight with meson versions, meson-python, cython versions, libc versions and so forth.

There is no longer any culture of correctness and code cleanliness in the Python ecosystem. A lot of good developers have left. Some current developers work for the companies who sell solutions for the chaos in the ecosystem.

Re: Conda: A package management disaster?

#136

Post by wakawaka28 »

jazzyjackson wrote: post #51
throwawaymaths wrote: post #23

Earlier quoted context omitted.

i remember reading somewhere (on twitter iirc) an amateur sex survey statistician who decided she needed to use python to analyze her dataset, being guided toward setting up venvs pretty early on by her programmer friends and getting extremely frustrated.

Was it aella? I don't know of any other sex survey statisticians so I'm assuming you mean aella. She has a pretty funny thread here but no mention of venvs: (non-musk-link https://xcancel.com/Aella_Girl/status/1522633160483385345 ) Every google for help I do is useless. Each page is full of terms I don't understand at *all*. They're like "Oh solving that error is simple, just take the library and shove it into the je…

Good grief there seems to be no getting away from that woman. One of my ex girlfriends was fascinated by her but to me she is quite boring. If she wasn't fairly attractive, nobody would care about her banal ramblings.

Re: Conda: A package management disaster?

#137

Post by Kwpolska »

hobofan wrote: post #44
The_Colonel wrote: post #32

It's rare to see something as systematically broken as Python package/dependencies ecosystem. What I don't understand - what makes this so difficult to solve in Python? It seems that many other platforms solved this a long time ago - maven 2.0 was released almost 20 years ago. While it wasn't / isn't by no means perfect, its fundamentals were decent already back then. One thing which I think messed this up from the b…

> What I don't understand - what makes this so difficult to solve in Python? I think there are many answers to this, and there are many factors contributing to it, but if I had to pick one: The setup.py file. It needs to be executed to determine the dependencies of a project. Since it's a script, that allows any maintainer of any package you are using to do arbitrarily complex/dumb stuff in it like e.g. conditionally…

setup.py allowed arbitrary things, but at least it always went through setuptools (or closely related predecessors, such as distribute or distlib). There is now pyproject.toml, but at the same time, there are tons of build backends that can do different things. And one of the most popular modern packaging tools, poetry, uses a non-standard section for the package data.

Re: Conda: A package management disaster?

#138

Post by vel0city »

mardifoufs wrote: post #101
These335 wrote: post #67

Can somebody please eli5 why it is so unanimously accepted that Python's package management is terrible? For personal projects venv + requirements.txt has never caused problems for me. For work projects we use poetry because of an assumption that we would need something better but I remain unconvinced (nothing was causing a problem for that decision to be made).

Try building a package and you will get hundreds of little paper cuts. Need a different index for some packages? It will work with a cli "pip install -from-index", but pip will not let you add an index in a requirement.txt for... security reasons. That means, good luck trying to "enforce" the CUDA version of pytorch without using third party tooling. So you either hard code a direct link (breaks platform portability)…

You can include command line parameters like index-url and include-extra-index-url and find-links and what not in a requirements file.

Re: Conda: A package management disaster?

#139

Post by biztos »

braza wrote: post #122
perrygeo wrote: post #104

Earlier quoted context omitted.

> what makes this so difficult to solve in Python? Python creates the perfect storm for package management hell: - Most the valuable libraries are natively compiled (so you get all the fun of distributing binaries for every platform without any of the traditional benefits of native compilation) - The dynamic nature makes it challenging to understand the non-local impacts of changes without a full integration test sui…

> Too many fractured packaging solutions, not a single one well designed. And they all conflict. 100% this. Last 4 years, one of the most frustrating parts of SWE that I need to deal with on a daily basis is packaging data science & machine learning applications and APIs in Python. Maybe this is a very mid-solution, but one solution that I found was to use dockerized local environments with all dependencies pinned vi…

I'm trying to do the same but with uv instead of poetry. So far so good, and it helps that for me delivering as a docker container is a requirement, but I have no idea what's going to happen if I need to run "real" ML stuff. (Just doing a lot of plotting so far.)

Re: Conda: A package management disaster?

#140

Post by Kwpolska »

cogman10 wrote: post #119
The_Colonel wrote: post #32

It's rare to see something as systematically broken as Python package/dependencies ecosystem. What I don't understand - what makes this so difficult to solve in Python? It seems that many other platforms solved this a long time ago - maven 2.0 was released almost 20 years ago. While it wasn't / isn't by no means perfect, its fundamentals were decent already back then. One thing which I think messed this up from the b…

> what makes this so difficult to solve in Python? I think the answer is the same thing that makes it difficult to make a good package manager for C++. When a language doesn't start with decent package management, it becomes really hard to retrofit a good one later in the lifespan of that language. Everyone can see "this sucks" but there's simply no good route to change the status quo. I think Java is the one languag…

Java, C#, JavaScript (node) all disagree. If the Python core team wanted good packaging, they could have done it ages ago. Sure, a good solution might not be applicable for past Python versions, but they aren’t doing anything to make it any better.
Post reply on HN