Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How to improve Python packaging

chriswarrick.com

Post reply on HN
Print view View original

51–60 of 204 posts

Re: How to improve Python packaging

#51

Post by DangitBobby »

DangitBobby wrote: post #31

I have a single, simple script (not a package!) that has dependencies. Actually, I have a few of these, just sitting in /usr/local/bin so I can execute them whenever. How should I be managing environments for these scripts? Do I install dependencies in shared system python? Should I create a shared venv? Where should I store it? Any tools out there that make this decision for you and manage it? Just the fact that hom…

In an amazing coincidence, I just found another front-page post exploring this same topic! For Python, it seems to suggest using nix-shell.

1. https://dbohdan.com/scripts-with-dependencies

Re: How to improve Python packaging

#52

Post by phailhaus »

kdazzle wrote: post #19
rektide wrote: post #6

Cant resist digging at Node.js even when writing up how infinitely better Node.js is at dealing with packages than python, haha: > Let’s try removing is-odd to demonstrate how badly designed this package is: You literally just deleted is-even 's dependency on is-odd then have the audacity to be shocked that it broke? There's a lot of hatred for the small package philosophy of node.js, but it's also a huge win, stands…

I’m biased, but I do like Python’s package management much better than Node. Even just regular old virtualenvs. Can’t tell you how many times deleting the node packages dir and reinstalling fixes a weird issue, but that’s happened very rarely for me on the python side. Also, having to comb through a bunch of node packages of dubious quality to find some sort of standard approach happens way too often. Like take pytho…

I'd say that Node's package management has run laps around Python's, to the point where it's pretty embarrassing. Since it's relatively new, it was able to hit the ground running with best practices:

1. Declarative package manifests. Python's ecosystem is still a mess of various approaches, and the fact that you have to run the setup.py script to determine dependencies is a nightmare. Because of this, running dependency resolution and installing is an order of magnitude faster in Node than in Python.

2. Drop-dead simple isolated environments: everything's in `node_modules`. You literally can't make a mistake by blindly running `npm install` in the project dir. With Python it's on you to manage your virtualenv, which boils down to PATH manipulation and symlinks that you'll have to remember to undo when switching around. There's no default for what to call your venv either, so it's on you to settle on a standard and gitignore it. Every time you run `pip install`, you have to hesitate and make sure you're in the right environment, or else risk borking the wrong env (or your global!)

3. Out-of-the-box comprehensive lockfile support. Debugging Python dependency issues is a nightmare. There's literally no way to figure out why a dependency was installed without using 3rd party tools (like pipdeptree). In Node, simply running `npm install` will automatically generate a proper lockfile.

I work full stack, and the difference is like night and day. I barely think about dependency management in Node.

Re: How to improve Python packaging

#53

Post by qbasic_forever »

woodruffw wrote: post #22
qbasic_forever wrote: post #15

Earlier quoted context omitted.

Nowhere in pypa documentation is your simple workflow described or mentioned. Instead it's a jumble of links to a myriad of tools including hatchling, flit, pdm, etc. and basically just a shoulder shrug, 'I don't know, figure it all out yourself' message. This article makes a great point that the current pypa 'guidance' is too confusing and vague for actual end users (i.e. people that don't work directly on pypa or h…

I agree that it's confusing. That being said, there is an official PyPA tutorial that goes through the exact steps need to produce the commands I suggested, so I don't think it's accurate to say that it's nowhere to be found[1]. Edit: Unfortunately it's easy to confuse the above tutorial with this one[2], which is specifically for setuptools. So I can appreciate end user confusion around the documentation, particular…

No I'm looking at link 1 and specifically have an issue with the huge table of options for how to setup your pyproject.toml--it has tabs for hatchling, flit, pdm, etc. but _zero_ description of why I would want to use one of those tools. It's just more confusing, like you're being thrown a bag of parts and no description of what to build or how to do it.

To be honest that entire pypa doc should be like two paragraphs long instead of 1000+ words. It should be basically, "ok to package your python app just run , that's it you're done!". Every decision should be made for me, like it is with npm, cargo, etc. I shouldn't have to think beyond running one command.

That's what python end users need. We don't need a million tools and huge docs with seemingly no vision or goal. We need to get shit done and the less time we faff about with packaging and tooling the better.

Re: How to improve Python packaging

#54

Post by Izkata »

woodruffw wrote: post #50
Izkata wrote: post #42

Earlier quoted context omitted.

Or if you don't want to install something else and are willing to just use version numbers (instead of also hashes like pip-compile in that link), "pip freeze" is built in.

The tricky thing with `pip freeze` is that it dumps your environment , not your resolved set : your environment also contains things like your `pip` and `setuptools` versions, any developing tooling you have, and potentially your global environmental state (if the environment has global access and you forget to pass `--local` to `pip freeze`). In other words, it's generally a superset of the resolutions collected by…

By default (at least in python3 nowadays) it also excludes pip, setuptools, distribute, and wheel; you need "--all" to include them in the output.

Re: How to improve Python packaging

#55

Post by woodruffw »

Izkata wrote: post #54
woodruffw wrote: post #50

Earlier quoted context omitted.

The tricky thing with `pip freeze` is that it dumps your environment , not your resolved set : your environment also contains things like your `pip` and `setuptools` versions, any developing tooling you have, and potentially your global environmental state (if the environment has global access and you forget to pass `--local` to `pip freeze`). In other words, it's generally a superset of the resolutions collected by…

By default (at least in python3 nowadays) it also excludes pip, setuptools, distribute, and wheel; you need "--all" to include them in the output.

Oh, that's news to me! I stand corrected.

(The point about other development tooling is, I believe, still accurate -- if you e.g. have `black` installed, `pip freeze` will show it.)

Re: How to improve Python packaging

#56

Post by LtWorf »

matsemann wrote: post #7

Because of the insanity of python I run everything in Docker (through compose). No more issues with it not working on some dev's computer because of a missing wheel, or that they need to have X and Y c++ toolchain packages installed locally etc. No more trying to fix a broken setup after upgrading python or poetry versions, just "docker compose build" and you're up and running. No spending days getting a freshly clon…

.deb packages…

Re: How to improve Python packaging

#58

Post by carapace »

samwillis wrote: post #13

This post is prompted by the survey of Python users, their feedback, and a current thread [0] on the Python forms discussing a way forward. I read the thread the other day, it's long, and there are a lot of opinions. Towards the end of this post there is an interesting observation: > Discourse, the platform that the discussion was held on, shows the number of times a link was clicked. Granted, this count might not be…

I think you're right. At risk of making a "lowbrow dismissal" the very length of TFA is a clear symptom of the problem: Design-by-committee (The PyPA (Python Package Authority) reminds me of the movie Brazil. Don't get me started.)

- - - -

Someone should do for packaging what pathlib did for file I/O: make a Pythonic model/API for it. Then deployment becomes simple, scriptable, testable, repeatable, etc...

Re: How to improve Python packaging

#59

Post by charliermarsh »

> There are two tools for dealing with packages in the Node world, namely npm and Yarn.

Not to mention pnpm. Or Bun. Or even Deno. All of which deal with packages, and make different choices.

This isn't to be a pedant, but I think it's a reasonable example of why it's less important that there's a single way to do a thing, or even a single tool vs. multiple alternatives; instead, what I care about is that I don't have to string together tens of tools to do that thing, and that there's some degree of compatibility and standardization in the API between those different alternatives.

Re: How to improve Python packaging

#60

Post by tempest_ »

imglorp wrote: post #18
tempest_ wrote: post #12

I've pretty much settled on poetry for most things at this point. It still has a ton of rough edges like error messages still closer to a stack trace than something actionable and it's slow as hell resolving but it does the job 90 percent of the time without issue. Also running everything in docker now anyway which side steps some other problems.

That's where we are as well, but yes the problems you didn't sidestep with poetry+docker are still there: the pypi fecosystem out there does not stop quantum fluctuations just because you want it to. If you pin too much you're not getting security fixes and if you don't pin enough, you get bitten every time a tertiary dep changes that you never asked for. Oh yeah and there are now trojans upstream almost daily.

Yeah but that trade off is entirely separate.

We trade security for speed(or "velocity" if you want to be jargon about it).

I just pin everything and go through my projects every couple weeks and bump the deps (unless some really big CVE hits the news).

Post reply on HN