Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Python Has Too Many Package Managers

dublog.net

Post reply on HN
Print view View original

101–110 of 170 posts

Re: Python Has Too Many Package Managers

#101

Post by yjftsjthsd-h »

colemannerd wrote: post #98
yjftsjthsd-h wrote: post #96

Earlier quoted context omitted.

> For me, this setup has caused production outages multiple times across multiple teams How does that happen?

I've personally seen the issues the article mentions with pyenv and virtual envs first hand when relying on pip.

The problem mentioned with pyenv is that people accidentally develop/test on the wrong version of python itself. But that's specific to pyenv, and I don't actually see where the article discusses problems with venv. So again: What exact steps would a team take using just pip+virtualenv or pip+conda (the comment you responded to didn't mention pyenv or venv) that would lead to production outages?

Re: Python Has Too Many Package Managers

#102

Post by tandav »

There is also virtualenvwrapper. It’s quite handy to create, list, remove virtual environment. I prefer to store all venvs in ~/.cache/virtualenvs instead of .venv in project directory, makes it more clean, no need to exclude for backups or git repository.

https://virtualenvwrapper.readthedocs.io/en/latest/

Re: Python Has Too Many Package Managers

#103

Post by PufPufPuf »

aborsy wrote: post #43

I use Python but I’m not a professional Python programmer. I use pip and venv, with requirements.txt. It copies Python binary and some of the shared standard libraries from the system, installs the required packages in sites-packages, and it always works fine. Am I missing anything major by not using conda and Poetry?

That's generally fine until the build breaks due to a transitive dependency several levels down releasing a breaking change. While requirements.txt may hold enough information to build the project now, it is not guaranteed to work in the future -- you would need lockfiles for that guarantee.

I've seen projects sort of emulating this by using "requirements.in" to list direct program dependencies, and auto-generating "requirements.txt" using "pip freeze". But when you find yourself doing that, it's probably time to switch to Poetry.

Re: Python Has Too Many Package Managers

#104

Post by nrclark »

sealedservant wrote: post #100
nrclark wrote: post #91

I ship a lot of Python in a CI/CD or devops context, and also deploy it to embedded targets. I've never needed anything more than pip, a venv, and pip-tools (to provide pip-compile). Venvs are treated as disposable. The basic workflow I use is: One-time (or as-needed for manual upgrades): 1. Make a venv with setuptools, wheel, and pip-tools (to get pip-compile) installed. 2. Use venv's pip-compile to generate a fully…

My problem with pip-tools is this: # pip-tools dependencies dependencies = [ # direct dependencies "build >= 1.0.0", "click >= 8", "pip >= 22.2", "pyproject_hooks", "tomli; python_version ] pip is nice since it comes out of the box with Python. setuptools used to but now it's gone. The dependency explosion is a huge problem for a lot of people trying to lockdown the security and maintainability of their codebases. I…

The nice thing about my approach is that pip-tools is only needed when modifying your requirements, and it also lives in a separate venv/requirements.txt from the dependencies you actually care about.

So yes: you need those dependencies in a development context, if you're actively modifying your Python requirements. But they don't make their way into your production requirements.txt, and don't need to be installed anywhere other than a short-lived venv.

Re: Python Has Too Many Package Managers

#105

Post by LarsDu88 »

simonw wrote: post #74

On Rye: "This project was ultimately abandoned by its author in 2023 and given to Astral.sh in favor of supporting uv instead" I don't think that's quite the right way to frame this. Handing Rye over to a company that could maintain it full time isn't the same thing as "abandoning" it - and the new maintainers are active on that project: https://github.com/astral-sh/rye/commits/main/

Made a mistake here and have since corrected it

Re: Python Has Too Many Package Managers

#106

Post by colemannerd »

yjftsjthsd-h wrote: post #101
colemannerd wrote: post #98

Earlier quoted context omitted.

I've personally seen the issues the article mentions with pyenv and virtual envs first hand when relying on pip.

The problem mentioned with pyenv is that people accidentally develop/test on the wrong version of python itself. But that's specific to pyenv, and I don't actually see where the article discusses problems with venv. So again: What exact steps would a team take using just pip+virtualenv or pip+conda (the comment you responded to didn't mention pyenv or venv) that would lead to production outages?

It feels like you've determined there's nothing wrong with pyenv, pip, and virtualenv so any issues brought up, you will reject.

If that's not the case, here's the issue - someone used pyenv and did not exactly specify the python type - I believe we were on 3.9 and prod was 3.9.11 and the current python version was 3.9.12. There was a downstream package that had an OS dependency - I believe it was pandas - that conflicted and needed to be updated locally to work with 3.9.12. This broke and raised an error in production that was not reproducible locally - and when you deploy on AWS, reproducing can be a pain in the butt. I'm sure if the data scientist had used perfect pyenv, virtualenv, and pip commands; we would have caught this. However, they're very complicated - especially for people who focus on math - so requiring full knowledge of these tools is unrealistic for most data scientists.

Re: Python Has Too Many Package Managers

#107

Post by mih »

sealedservant wrote: post #18

I concur and I also think that there are too many build backends. pdm is my current favorite package manager. It is fully PEP-compliant and the lockfile generation is nice. I wouldn't call hatch a package manager because I don't think it can make lockfiles. uv is on my radar but it doesn't look ready for primetime yet. I saw they are building out a package management API with commands such as `uv add` and `uv remove`…

Another user of pdm here for professional projects. It sure is more standards compliant than poetry. Support for in-project venvs and integration of configs for packages such as pytest is quite useful.

When evaluating package managers, poetry for sure was a contender. However listening to others experiences regarding poetry developers introducing breaking changes that could potentially cause the CI pipeline made it a no go [1]. uv seems to be coming along rather nicely, but wasn't anywhere near the level of stability compared to pdm during the evaluation phase.

[1] https://www.youtube.com/watch?v=Gr9o8MW_pb0

Re: Python Has Too Many Package Managers

#108

Post by drewcoo »

rilindo wrote: post #12
actionfromafar wrote: post #11

After all, the motto is "there's more than one way to do it"!

That is actually Perl's motto. [0] [0] https://wiki.c2.com/?ThereIsMoreThanOneWayToDoIt

But Perl started this whole one package manager to bind us thing with CPAN.

Re: Python Has Too Many Package Managers

#110

Post by VagabundoP »

LarsDu88 wrote: post #76
VagabundoP wrote: post #30

Use Rye. It wasn't abandoned it ownership was transferred. Rye uses other pretty standard stuff under the hood, tools that follow PEPs, its just a front end that is sane. uv is fast as well. It downloads the pinned version of standalone Python, it keeps everything in its own venv and theres very little messing/tweaking of the environment. It is messy, although its getting better. I doubt everything will ever standard…

I made a mistake about Rye when I posted the first draft of this article. That has since been corrected. Rye is incredibly young, and when I first encountered it, I looked at its source code and found it was simply calling piptools under the hood. I did not realize that it had changed and now uses uv (or perhaps just the same solver as uv) under the hood.

It is young, and it does use established tools under the hood, but I think you should take another look:

- It manages the portable python version downloads depending on what you've pinned in your pyproject or the standard version: https://github.com/indygreg/python-build-standalone

- It uses Hatch, uv (or piptools), venv etc.

- It has a sane set of commands and should work even if they swap out some of the underlying tools.

- Its fast and has a great coverage of PEPs.

I've used it now for a few versions and I think its the best meta-packaging/python management tool out there. I

Post reply on HN