Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How to improve Python packaging

chriswarrick.com

Post reply on HN
Print view View original

191–200 of 204 posts

Re: How to improve Python packaging

#191

Post by vonseel »

BerislavLopac wrote: post #190
BerislavLopac wrote: post #189

Despite being misguided in a few points, the best bit about this article is the detailed table comparing the functionalities of different Python packaging tools. One missing detail is the ability of PDM to support different build backends, which allows for some interesting capabilities: for example, using hatchling as the backend it is possible to utilise hatch's support for dynamic versioning, whcih does not exist i…

One of the misguided points is this: > It is also notable that PEP 20, the Zen of Python, states this: > There should be one-- and preferably only one --obvious way to do it. > Python packaging definitely does not follow it. There are 14 ways, and none of them is obvious or the only good one. All in all, this is an unsalvageable mess. Why can’t Python pick one tool? So a few comments on that: 1. There is a reason PEP…

> Why can’t Python pick one tool?

Sometimes I wonder if developers are just making new tools out of self-interest and they want their name on some well-known project, and that's why they don't just work with one of the existing projects to implement their ideas and move the overall community towards a common working model.

> what is the harm in having multiple competing solutions?

Developer confusion and community fragmentation.

Re: How to improve Python packaging

#192

Post by vonseel »

woodruffw wrote: post #11

It's not very well documented, but the PyPA tools do provide a unified experience when used correctly. Here's a PyPA project (FD: one I work on) that uses a single pyproject.toml to handle all aspects of packaging (and most non-packaging tool configuration, to boot)[1]. With a single file like that, the only thing you need to do to start a local development environment is: python -m venv env && . env/bin/activate pyt…

So, the project you linked has a pyproject.toml file uses flit:

    requires = ["flit_core >=3.2,
Python.org defaults to using hatchling in the docs[1].

PyPA has a sample repository using setuptools[2]. This project is linked from Python.org on this page.[3]

Python.org recommends using setuptools under packaging recommendations as well, with no mention of hatchling or flit even on that page.[4]

No wonder so many devs are frustrated about this. I've been writing mostly Python for work and fun the past 11 years and I'm f*cking confused what tools to use. When I start a new side project I don't know whether a tool is going to exist and still work in a few years, because there's 10 tools trying to do the same things and it's not clear what the community is actually getting behind.

[1]: https://packaging.python.org/en/latest/tutorials/packaging-p...

[2]: https://github.com/pypa/sampleproject/blob/main/pyproject.to...

[3]: https://packaging.python.org/en/latest/guides/distributing-p...

[4]: https://packaging.python.org/en/latest/guides/tool-recommend...

Re: How to improve Python packaging

#193

Post by vonseel »

Izkata wrote: post #40
qbasic_forever wrote: post #15

Earlier quoted context omitted.

Nowhere in pypa documentation is your simple workflow described or mentioned. Instead it's a jumble of links to a myriad of tools including hatchling, flit, pdm, etc. and basically just a shoulder shrug, 'I don't know, figure it all out yourself' message. This article makes a great point that the current pypa 'guidance' is too confusing and vague for actual end users (i.e. people that don't work directly on pypa or h…

AFAIK in the python 2 era it was the de-facto standard (just swap "python -m venv" with "virtualenv"). All these new tools have just made it more complicated and it's not clear to me what they gain, seems to me like they've simply succeeded in convincing people it's complicated by obscuring what's actually going on.

Correct. And most people back then were just using `pip install -r requirements.txt`, including different files for dev/test/prod as needed.

I worked at one company that used buildout back then, which seemed relatively rare, but buildout was more flexible and allowed you to do things like installing system packages or running your own scripts. We used it to pull down config files that weren't checked into the repo but still wanted to be shared, helping setup the local developer environment, and a few other things.

Re: How to improve Python packaging

#194

Post by nezirus »

uranusjr wrote: post #179
nezirus wrote: post #92

Reading this is really depressing. I just want Cargo for Python. Poetry it is for now, but it has quirks, and it is dog slow...

A lot of people said they want Cargo for Python but immediately backed out when you require them to `cargo run` their program. They want `python myscript.py` to still magically work, but that’s exactly where a lot of the magic comes from. Running `python` directly is like manually invoking `rustc` (or maybe slightly more automated like a make script); it works, but is on an entirely different layer of abstraction and…

That is funny since poetry modus operandi is 'poetry run python ...'

On topic, "Cargo for Python" should be official, singular (no competing standards), opinionated and fast (tm).

Re: How to improve Python packaging

#195

Post by thinker5555 »

TigerTeamX wrote: post #157
theacodes wrote: post #140

Earlier quoted context omitted.

As the person who implemented the -t flag and knows the horrendously sharp edge cases that it fails on, all I can say is bless your heart.

Please do elaborate before digging my own grave, thanks :D

I did some looking around, and it sounds like it fails to work correctly with namespace packages. Something about the way you have to add them to the path leads to them being later in the path, so there could be times where they don't get found because something earlier in the path gets used instead.

Re: How to improve Python packaging

#196

Post by int_19h »

u801e wrote: post #186
poulpy123 wrote: post #182

Earlier quoted context omitted.

because where you have to build 1 or 2 packages for windows and macOS (97% of the computers used by end users), you have to build tens of packages for the main linux distribs (and not even all)

> you have to build tens of packages for the main linux distribs (and not even all) This is something that's typically handled by the package maintainers for each linux distro, rather than the ones who developed the application. Some developers maintain their own public repositories for packages they built and instruct end users to add their repositories to their package manager config, but they typically will also i…

Last thing the Linux package maintainers need is the entirety of PyPI dumped onto their lap to package and maintain.

Re: How to improve Python packaging

#197

Post by int_19h »

uranusjr wrote: post #179
nezirus wrote: post #92

Reading this is really depressing. I just want Cargo for Python. Poetry it is for now, but it has quirks, and it is dog slow...

A lot of people said they want Cargo for Python but immediately backed out when you require them to `cargo run` their program. They want `python myscript.py` to still magically work, but that’s exactly where a lot of the magic comes from. Running `python` directly is like manually invoking `rustc` (or maybe slightly more automated like a make script); it works, but is on an entirely different layer of abstraction and…

Is there any reason why `python foo.py` couldn't do the equivalent of what `cargo run` does, if that's the expectation of most users?

Re: How to improve Python packaging

#198

Post by int_19h »

TheChaplain wrote: post #28
Lyngbakr wrote: post #21

It seems most people agree that Python's packaging isn't great, but, conversely, is there a language where most people agree that the approach to packaging is awesome? I mean, what's the gold standard to aspire to?

Don't know if it counts but Java has done pretty well in my opinion. A single JAR-file can combine everything you need to run an application or be distributed as a library, and as long as the java command is installed it just works. Likewise WAR's make it dead simple to add/remove/update applications during runtime. Of course there are always special cases, but packaging and distribution in the Java world have always…

Python has similar facilities for distribution, since it can load code directly from .zip files, treating them as if they were directories - very similar to JAR. The problem is on the stage where you have to acquire and manage those dependencies.

Re: How to improve Python packaging

#199

Post by uranusjr »

int_19h wrote: post #197
uranusjr wrote: post #179

Earlier quoted context omitted.

A lot of people said they want Cargo for Python but immediately backed out when you require them to `cargo run` their program. They want `python myscript.py` to still magically work, but that’s exactly where a lot of the magic comes from. Running `python` directly is like manually invoking `rustc` (or maybe slightly more automated like a make script); it works, but is on an entirely different layer of abstraction and…

Is there any reason why `python foo.py` couldn't do the equivalent of what `cargo run` does, if that's the expectation of most users?

Most of those workflow tools support at least two execution contexts, one for development and one for deployment (`cargo run` vs executing the compiled binary directly, `npm run` vs `node foo.js`, etc.). There are various reasons for the separation including hidden build steps or to set up the runtime environment, generally you want certain things automated during development but not when the program is deployed. A more accurate way to say this `python foo.py` can’t replace `cargo run` while still doing what it currently does; obviously it could be changed to run the program in the development context. The problem is it’s currently used in multiple contexts, and nobody really wants their workflow to break.

Re: How to improve Python packaging

#200

Post by damnever »

I am the author of pigar[1], and I am using Go a lot, Go has its problems too, but I am a fan of `import "url"` style import statement, developers can write code first, and sync the dependency later with `go mod tidy`.

To fix problems in Python's world, Python's community should simplify the tools and cultivate a habit to declare the dependency first(maybe this should be mandatory).

[1]: https://github.com/damnever/pigar

Post reply on HN