Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Python Has Too Many Package Managers

dublog.net

Post reply on HN
Print view View original

61–70 of 170 posts

Re: Python Has Too Many Package Managers

#61

Post by kelseyfrog »

turndown wrote: post #44
lexicality wrote: post #37

Earlier quoted context omitted.

I do a lot of maintenance work and every time I've encountered a (complex) project set up in this way that's older than ~6 months, it's bitrotted from breaking changes in the dependencies. The python ecosystem does not stand still and seems quite happy to introduce breaking changes in non-major versions. To my mind, there are 3 ways to make sure your python project of today will work in 2 years time: 1. Have no depen…

Is option 3 exactly what you’re supposed to do? Freezing your dependency graph and/or explicitly denoting what version of the dependency you want are your best bets for avoiding problems like this

3 is exactly what you're supposed to do.

A lot of people will assume that specifying major version upper bounds on dependencies is what you're supposed to do, but I've seen this fail more often than freezing dependencies.

The problem with major version upper bounds is that if it's possible to write a test case for a bug, it's possible to depend on broken behavior. Changing behavior in a way that breaks users should be a major version bump, but that's not actually how people use semver and semver isn't really described that way either. It's described in a way that makes people think that changes in type signatures are the predominant impetus to bump major versions.

Re: Python Has Too Many Package Managers

#62

Post by mschuster91 »

slt2021 wrote: post #23

I see a lot of package managers I never head of, am a happy venv & pip user. One of the key faults of pip is what happens when you decide to remove a dependency. Removing a dependency does not actually remove the sub-dependencies that were brought in by the original dependency, leaving a lot of potential cruft. This is not really an issue if your virtual environments are disposable. Just nuke and recreate venv from s…

> This is not really an issue if your virtual environments are disposable. Just nuke and recreate venv from scratch using only what you need.

There was (or still is, I wouldn't be surprised!) a time when the AWS CLI and the AWS Elastic Beanstalk CLI had incompatible versions of some dependency and couldn't be installed in the same venv. Not fun.

Re: Python Has Too Many Package Managers

#63

Post by verst »

SethMLarson wrote: post #24

Happily using pip, venv, and pip-tools for every project and still finding them more than suitable. They might not have the marketing budget or pizazz of others, but if you're looking for effective and boring tools that get the job done so you can solve more interesting problems they work just fine.

Same.

Additionally I use pyenv and pyenv-virtualenv to manage multiple Python versions. Different venvs pointing to different Python versions etc, but the core tools are still pip, venv and pip-tools. (I do not use conda)

I publish a Python SDK with about 85k monthly downloads according to PyPI. I make sure to run tests (unit tests, type checking, integration tests) against all currently supported minor versions of Python (3.8 - 3.12)

Re: Python Has Too Many Package Managers

#65

Post by [deleted] »

rangerelf wrote: post #41
sealedservant wrote: post #25

Earlier quoted context omitted.

Virtual environments are cool, and necessary, but at the same time, they are incredibly limited and I always get frustrated at the lack of features they should have. They are too fragile. Nuking a whole venv when you mess up isn't really efficient. They aren't portable. You need to package up your editable project for an offline system? Too bad, virtual environments use hardcoded paths and symlinks that will be broke…

I've read "they're too/so/very fragile" so many times, but nobody has given me an example of how or why they consider them fragile, what steps do I need to do to break them. Myself? I consider them ephemeral: I create my Makefile with a 'venv' target to delete/rebuild the virtual environment in case of changes. Some do take longer to rebuild, but with a global pip cache it takes much less time to rebuild once it's be…

[deleted]

Re: Python Has Too Many Package Managers

#66

Post by rangerelf »

slt2021 wrote: post #51
rangerelf wrote: post #41

Earlier quoted context omitted.

I've read "they're too/so/very fragile" so many times, but nobody has given me an example of how or why they consider them fragile, what steps do I need to do to break them. Myself? I consider them ephemeral: I create my Makefile with a 'venv' target to delete/rebuild the virtual environment in case of changes. Some do take longer to rebuild, but with a global pip cache it takes much less time to rebuild once it's be…

Can I ask you a question, what do you think is better approach: 1) publish packages as wheels or 2) publish apps as docker images (or docker-compose files or helm charts)? Why some People prefer 1 to 2? I think 2 is more “production friendly” and universal across other languages and stacks (same approach used for java js ruby etc)

Vastly different use cases.

Publish a module as a wheel if you want to be able to use pip to install it in your servers, or anywhere really.

Once you've built out your virtual environment you can build your docker image and push it to a registry.

Your helm chart should include references to docker images that are getting deployed to a kubernetes cluster, among other entities that need to be built out for your app to work.

Those are all different layers of the deployment, independent of each other.

If someone prefers 1 over 2 (or viceversa), they don't understand what they think they know.

Re: Python Has Too Many Package Managers

#67

Post by daemonologist »

turndown wrote: post #44
lexicality wrote: post #37

Earlier quoted context omitted.

I do a lot of maintenance work and every time I've encountered a (complex) project set up in this way that's older than ~6 months, it's bitrotted from breaking changes in the dependencies. The python ecosystem does not stand still and seems quite happy to introduce breaking changes in non-major versions. To my mind, there are 3 ways to make sure your python project of today will work in 2 years time: 1. Have no depen…

Is option 3 exactly what you’re supposed to do? Freezing your dependency graph and/or explicitly denoting what version of the dependency you want are your best bets for avoiding problems like this

It is, but Python software tends towards large dependency graphs which will quickly accumulate CVEs, and so this strategy greatly upsets your security/audit people.

(Obviously patching vulnerabilities is good and proper, but automatically flagged CVEs are only ever _potential_ at best; in many contexts most of them are not actual vulnerabilities.)

Re: Python Has Too Many Package Managers

#68

Post by abdullahkhalids »

slt2021 wrote: post #23

I see a lot of package managers I never head of, am a happy venv & pip user. One of the key faults of pip is what happens when you decide to remove a dependency. Removing a dependency does not actually remove the sub-dependencies that were brought in by the original dependency, leaving a lot of potential cruft. This is not really an issue if your virtual environments are disposable. Just nuke and recreate venv from s…

I use the same workflow, and works for me very well.

It would be cool, though, to have a wrapper around pip that adds any packages I install to the requirements file. That way I don't have to do it manually.

Re: Python Has Too Many Package Managers

#69

Post by yegle »

Why vendoring is not a common practice baffles me, especially since the leftpad incident happened over 8 years ago.

For Python, you can use `pip wheel` (https://pip.pypa.io/en/stable/cli/pip_wheel/) to download .whl files of your dependencies in a folder, add that folder to your version control, and update `sys.path` to include your .whl.

For updating packages, you run `pip wheel` again and check in the new .whl files after carefully review the changes.

Re: Python Has Too Many Package Managers

#70

Post by slt2021 »

lexicality wrote: post #58
slt2021 wrote: post #51

Earlier quoted context omitted.

Can I ask you a question, what do you think is better approach: 1) publish packages as wheels or 2) publish apps as docker images (or docker-compose files or helm charts)? Why some People prefer 1 to 2? I think 2 is more “production friendly” and universal across other languages and stacks (same approach used for java js ruby etc)

How would you use the python `requests` library if it was packaged as a docker image?

I meant you package your whole app into container with all required libs, and dont publish your app as a package
Post reply on HN