Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Python Has Too Many Package Managers

dublog.net

Post reply on HN
Print view View original

161–170 of 170 posts

Re: Python Has Too Many Package Managers

#161

Post by gillesjacobs »

p1esk wrote: post #115
gillesjacobs wrote: post #85

Earlier quoted context omitted.

ML engineer, I am up to 3 irreversibly broken environments after simply adding a package. Conda is inexcusably slow and but even mamba can save it. Every conda project I have ever was later switched to poetry. Here's a fun challenge: try to determine channel priority in your conda env, go ahead try. Conda is the single worst packaging tool I have ever used.

OK, would you say poetry is the best for ML engineers?

Poetry worked best in several Python-first companies and clients I have worked for.

It's not perfect but the best bet for complex and collaboration-heavy projects.

Re: Python Has Too Many Package Managers

#162

Post by rangerelf »

sealedservant wrote: post #97
rangerelf wrote: post #41

Earlier quoted context omitted.

I've read "they're too/so/very fragile" so many times, but nobody has given me an example of how or why they consider them fragile, what steps do I need to do to break them. Myself? I consider them ephemeral: I create my Makefile with a 'venv' target to delete/rebuild the virtual environment in case of changes. Some do take longer to rebuild, but with a global pip cache it takes much less time to rebuild once it's be…

> And I've done the last one, converted all packages to wheels Did this require just downloading the packages again from the requirements.txt, or can pip do this? That could help me out quite a bit...

Use "pip wheel":

https://pip.pypa.io/en/stable/cli/pip_wheel/

The package should be installed first, then create wheels from all of them, finally upload them all to a repository for storage.

Re: Python Has Too Many Package Managers

#163

Post by rangerelf »

slt2021 wrote: post #73
rangerelf wrote: post #66

Earlier quoted context omitted.

Vastly different use cases. Publish a module as a wheel if you want to be able to use pip to install it in your servers, or anywhere really. Once you've built out your virtual environment you can build your docker image and push it to a registry. Your helm chart should include references to docker images that are getting deployed to a kubernetes cluster, among other entities that need to be built out for your app to…

By publishing module you refer to situation when a single module is reused across different applications? So if you dont have modules that used across apps, then you dont need to package and publish wheel? or if your modules are already vendored in your main application module (monorepo) then no need to package app as a wheel?

The wheel, kept on an Sudafed repository, will let you lock down your supply chain, plus no need to rebuild every time it gets used.

I would build it once for whatever required architectures are needed, build wheels for them, amd use those for packaging up the final apps that need deploying (through docker, k8s, whatever).

Re: Python Has Too Many Package Managers

#164

Post by abdullahkhalids »

kiney wrote: post #155
abdullahkhalids wrote: post #68

Earlier quoted context omitted.

I use the same workflow, and works for me very well. It would be cool, though, to have a wrapper around pip that adds any packages I install to the requirements file. That way I don't have to do it manually.

pip freeze Is what you're looking for

pip freeze lists all possible installed packages with exact version.

I want only the manually installed packages in exactly the way I installed them.

So pip install numpy>=3.2 should put numpy>=3.2 in requirements.txt. and pip install numpy should put numpy in requirements.txt.

Re: Python Has Too Many Package Managers

#165

Post by marcosdumay »

sitkack wrote: post #55
marcosdumay wrote: post #28

Earlier quoted context omitted.

It's certainly a joke, because the Python original motto was "there's only one way to do it", and the current one is "there's only one (obvious) way to do it". The Python's motto was created as an obvious reference to the Perl's one, purposefully negating it.

The Python motto has always been satire.

Kinda. They basically say "Perl is such a bad language that I bet if I inverted every thing on it I'll make a great language".

So, not completely serious, but not just a joke either. Python isn't exactly a "reverse Perl" but it's close to it, and it's a really good language.

Re: Python Has Too Many Package Managers

#166

Post by theshrike79 »

JackSlateur wrote: post #152
lexicality wrote: post #37

Earlier quoted context omitted.

I do a lot of maintenance work and every time I've encountered a (complex) project set up in this way that's older than ~6 months, it's bitrotted from breaking changes in the dependencies. The python ecosystem does not stand still and seems quite happy to introduce breaking changes in non-major versions. To my mind, there are 3 ways to make sure your python project of today will work in 2 years time: 1. Have no depen…

What sucks is when you imply that your project will be unmaintained for 2 years, without any kind of update (not even basic security upgrades). This is a not a project, this is abandonware from day 1.

It's not like that simple 50 line Python script taking stuff from API A and stuffing them to API B is a full ass project with a JIRA project ID and a Product Manager.

It works, people forget it exists.

Then something breaks and I need to start fixing it. And the amount of work it takes to just update the packages is more than it'd take me to rewrite the whole thing in a language with a sane distribution mechanism (Go).

Re: Python Has Too Many Package Managers

#167

Post by dogaar »

One thing this blog does not question is whether package managers are needed at all. Look at how Deno does it: packages are imported from URLs and downloaded to a local cache at run-time. This allows CDN-based package distribution, but you still need a package search engine.

Re: Python Has Too Many Package Managers

#168

Post by yjftsjthsd-h »

colemannerd wrote: post #106
yjftsjthsd-h wrote: post #101

Earlier quoted context omitted.

The problem mentioned with pyenv is that people accidentally develop/test on the wrong version of python itself. But that's specific to pyenv, and I don't actually see where the article discusses problems with venv. So again: What exact steps would a team take using just pip+virtualenv or pip+conda (the comment you responded to didn't mention pyenv or venv) that would lead to production outages?

It feels like you've determined there's nothing wrong with pyenv, pip, and virtualenv so any issues brought up, you will reject. If that's not the case, here's the issue - someone used pyenv and did not exactly specify the python type - I believe we were on 3.9 and prod was 3.9.11 and the current python version was 3.9.12. There was a downstream package that had an OS dependency - I believe it was pandas - that confl…

> It feels like you've determined there's nothing wrong with pyenv, pip, and virtualenv so any issues brought up, you will reject.

Alternatively, I'm rejecting your claims because you keep making them and then not providing evidence. Now that you've actually described the problem, I can agree that that's a footgun, and pyenv should start to strongly discourage setting a global version in much the same way that pip has started to protect against people using `sudo pip install` to trash their systems.

Re: Python Has Too Many Package Managers

#169

Post by jacobgorm »

big-green-man wrote: post #160
jacobgorm wrote: post #156

Earlier quoted context omitted.

It seems to me one would have to solve the Halting Problem for that to be universally true.

Not quite, it is pretty trivial to check what external dependencies your program calls, and what those dependencies call and so forth. I'm sure you could write a program that has a dependency chain that requires the entire database of packages in cargo (and that would he an interesting experiment). A lot of modern programming languages evaluate what code is referenced in a working program and only include used code,…

I program that is Turing complete cannot be reasoned about unless you can solve the Halting problem. So you cannot decide at compile time if a dependency that is imported is actually needed or not. In rust even the build-scripts are Turing-complete rust-programs, which makes this even harder.

Re: Python Has Too Many Package Managers

#170

Post by BerislavLopac »

slt2021 wrote: post #23

I see a lot of package managers I never head of, am a happy venv & pip user. One of the key faults of pip is what happens when you decide to remove a dependency. Removing a dependency does not actually remove the sub-dependencies that were brought in by the original dependency, leaving a lot of potential cruft. This is not really an issue if your virtual environments are disposable. Just nuke and recreate venv from s…

> Can people explain why venv&pip is a bad solution that doesnt work for them they have to resort to other package managers?

It's very simple:

1. As you said yourself, you don't need venv at all. 2. The format used by `pip -r` (colloquially known as "requirements file") is not a defined standard. 3. pip is not a package manager, but a package version resolver and installer.

Post reply on HN