Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Python Has Too Many Package Managers

dublog.net

Post reply on HN
Print view View original

91–100 of 170 posts

Re: Python Has Too Many Package Managers

#91

Post by nrclark »

I ship a lot of Python in a CI/CD or devops context, and also deploy it to embedded targets. I've never needed anything more than pip, a venv, and pip-tools (to provide pip-compile). Venvs are treated as disposable. The basic workflow I use is:

One-time (or as-needed for manual upgrades):

   1. Make a venv with setuptools, wheel, and pip-tools (to get pip-compile) installed.
   2. Use venv's pip-compile to generate a fully-pinned piptools-requirements.txt for the venv.
   3. Check piptools-requirements.txt into my repo. This is used to get a stable, versioned `pip-compile` for use on my payload requirements.
During normal development:

    1. Add high-level dependencies to a `requirements.in` file. Usually unversioned, unless there's a good reason to specify something more exact.
    2. On changes to `requirements.in`, make a venv from `piptools-requirements.txt` and its `pip-compile` to solve `requirements.in` into a fully-pinned `requirements.txt`.
    3. Check requirements.in and requirements.txt into the repo. 
    4. Install packages from requirements.txt when making the venv that I need for production.
This approach is very easy to automate, CI/CD friendly, and completely repeatable. It doesn't require any nonstandard tools to deploy (and only needs pip-compile when recompiling requirements.txt). It also makes a clear distinction between "what packages do the developers actually want?" and "what is the fully-versioned set of all dependencies".

It's worked great for me over the years, and I'd highly recommend it as a reliable way to use the standard Python package tooling.

Re: Python Has Too Many Package Managers

#92

Post by kelseyfrog »

LarsDu88 wrote: post #72
kelseyfrog wrote: post #61

Earlier quoted context omitted.

3 is exactly what you're supposed to do. A lot of people will assume that specifying major version upper bounds on dependencies is what you're supposed to do, but I've seen this fail more often than freezing dependencies. The problem with major version upper bounds is that if it's possible to write a test case for a bug, it's possible to depend on broken behavior. Changing behavior in a way that breaks users should b…

I mention this ceiling pinning footgun in the article. It's an enormous pain in the ass to explain to folks, and some software engineers I've met are totally incredulous that that's "not the right thing to do" Poetry makes it 10x worse with its `^` operator

Definitely! And the article you link to in this section is great too

Re: Python Has Too Many Package Managers

#93

Post by artexy »

The people who say "just use pip and venv" don't understand the issue.

Distutils has been ripped out of Python core, setuptools is somewhat deprecated but not really. Just don't call setup.py directly. Or use flit. Or perhaps pyproject.toml? If the latter, flit, poetry and the 100 other frontends all have a different syntax.

Would you like to copy external data into the staging area while using flit? You are out of luck, use poetry. Poetry versions Should you use pip? Well, the latest hotness are the search-unfriendly named modules "build" and "install", which create a completely isolated environment. Which is the officially supported tool that will stay supported and not ripped out like distutils?

Questions over questions. And all that for creating a simple zip archive of a directory, which for some reason demands gigantic frameworks.

Re: Python Has Too Many Package Managers

#94

Post by yjftsjthsd-h »

lexicality wrote: post #37
slt2021 wrote: post #23

I see a lot of package managers I never head of, am a happy venv & pip user. One of the key faults of pip is what happens when you decide to remove a dependency. Removing a dependency does not actually remove the sub-dependencies that were brought in by the original dependency, leaving a lot of potential cruft. This is not really an issue if your virtual environments are disposable. Just nuke and recreate venv from s…

I do a lot of maintenance work and every time I've encountered a (complex) project set up in this way that's older than ~6 months, it's bitrotted from breaking changes in the dependencies. The python ecosystem does not stand still and seems quite happy to introduce breaking changes in non-major versions. To my mind, there are 3 ways to make sure your python project of today will work in 2 years time: 1. Have no depen…

IMHO the problem with python package management isn't python package management. The biggest problem I've hit is when python depends on non-python - `pip install` failing because I need to `apt install libsomething-dev` is the big one, but also managing multiple versions of python itself (I count this as a dependency on non-python because the python3 binary isn't written in python - if it was, then pip+virtualenv could manage it like everything else). And also as you note there's just ecosystem churn.

Re: Python Has Too Many Package Managers

#95

Post by alganet »

g8oz wrote: post #59

I just want to point out that Composer for PHP is a great experience.

Composer did the right thing: it doesn't handle binaries, SAT for solving a _single_ version for the entire project tree and overall focus on simplicity.

You need to handle PHP extensions outside of it. Composer can only warn you if your PHP lacks some compiled part, but won't act on it. It's good separation of concerns (and phpize/pecl already handles extensions very well).

You also don't have multiple copies of the same package at different versions. Composer requires that a single version must be compatible. This sounds like a nightmare that would break easy, but it actually works by forcing package providers to be more backwards-compatible.

That being said, we often don't use Composer to install tools in a machine, like we do with Python. Composer is there mostly to build standalone projects to be deployed elsewhere.

You can `composer global install` stuff, but you often don't need to. I absolutely see this as a win. Tools for my machine should be installed by my OS package manager, not some language specific thing.

Re: Python Has Too Many Package Managers

#96

Post by yjftsjthsd-h »

colemannerd wrote: post #88
wantsanagent wrote: post #22

I use pip. I plan to continue using pip. If I need an isolated environment, I use conda, but then I install everything with pip. If I need to guarantee versions I pip freeze. There's a lot of cruft and desire for a one-size-fits-all solution but the base tools are probably good enough. My setup is not the one-size-fits-all solution but it works for me, and my team, and lots of other teams. Beware anyone who tells you…

BUT importantly... it REALLY does not work well for lots of teams. For me, this setup has caused production outages multiple times across multiple teams. Maybe the root python ecosystem should learn and adopt from other ecosystems that have figured out complex deployment in a much easier way.

> For me, this setup has caused production outages multiple times across multiple teams

How does that happen?

Re: Python Has Too Many Package Managers

#97

Post by sealedservant »

rangerelf wrote: post #41
sealedservant wrote: post #25

Earlier quoted context omitted.

Virtual environments are cool, and necessary, but at the same time, they are incredibly limited and I always get frustrated at the lack of features they should have. They are too fragile. Nuking a whole venv when you mess up isn't really efficient. They aren't portable. You need to package up your editable project for an offline system? Too bad, virtual environments use hardcoded paths and symlinks that will be broke…

I've read "they're too/so/very fragile" so many times, but nobody has given me an example of how or why they consider them fragile, what steps do I need to do to break them. Myself? I consider them ephemeral: I create my Makefile with a 'venv' target to delete/rebuild the virtual environment in case of changes. Some do take longer to rebuild, but with a global pip cache it takes much less time to rebuild once it's be…

> And I've done the last one, converted all packages to wheels

Did this require just downloading the packages again from the requirements.txt, or can pip do this? That could help me out quite a bit...

Re: Python Has Too Many Package Managers

#98

Post by colemannerd »

yjftsjthsd-h wrote: post #96
colemannerd wrote: post #88

Earlier quoted context omitted.

BUT importantly... it REALLY does not work well for lots of teams. For me, this setup has caused production outages multiple times across multiple teams. Maybe the root python ecosystem should learn and adopt from other ecosystems that have figured out complex deployment in a much easier way.

> For me, this setup has caused production outages multiple times across multiple teams How does that happen?

I've personally seen the issues the article mentions with pyenv and virtual envs first hand when relying on pip.

Re: Python Has Too Many Package Managers

#99

Post by mistahchris »

chpatrick wrote: post #36

I really like Nix for Python, as long as the packages I need are already packaged. Otherwise I'll use pip and venv to try stuff out. Can't stand conda.

If you haven't yet, check out https://devenv.sh (super powered nix shell and more). It's pretty nice for python packages and installs your requirements to a project local venv for you via whatever tool you want (pip, poetry, uv etc).

I've been using it for a couple of years and it's super nice to be able to manage both python and "native" dependencies, and other non-python development tools all together.

I used just nix and whatever python packages are already in nixpkgs for several projects. And that works really really well until you run into an issue with compatibility like I did. It seems to mostly happen when some extremely common tool like `awscli2` depends on a specific version of some package and so it's pinned.

Re: Python Has Too Many Package Managers

#100

Post by sealedservant »

nrclark wrote: post #91

I ship a lot of Python in a CI/CD or devops context, and also deploy it to embedded targets. I've never needed anything more than pip, a venv, and pip-tools (to provide pip-compile). Venvs are treated as disposable. The basic workflow I use is: One-time (or as-needed for manual upgrades): 1. Make a venv with setuptools, wheel, and pip-tools (to get pip-compile) installed. 2. Use venv's pip-compile to generate a fully…

My problem with pip-tools is this:

# pip-tools dependencies

dependencies = [

  # direct dependencies

  "build >= 1.0.0",

  "click >= 8",

  "pip >= 22.2",

  "pyproject_hooks",

  "tomli; python_version
]

pip is nice since it comes out of the box with Python. setuptools used to but now it's gone.

The dependency explosion is a huge problem for a lot of people trying to lockdown the security and maintainability of their codebases. I think that's why a lot of people are rallying around Astral's projects like uv and ruff... they do so many things and they do them well.

Post reply on HN