Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Underappreciated challenges with Python packaging

pypackaging-native.github.io

Post reply on HN
Print view View original

51–60 of 75 posts

Re: Underappreciated challenges with Python packaging

#51

Post by teekert »

kevin_thibedeau wrote: post #37
woodruffw wrote: post #11

Earlier quoted context omitted.

If it's pure Python, the only packaging file you need is `pyproject.toml`. You can fill that file with packaging metadata per PEP 518 and PEP 621, including using modern build tooling like flit[1] for the build backend and build[2] for the frontend. With that, you entire package build (for all distribution types) should be reducible to `python -m build`. Here's an example of a full project doing everything with just…

Good luck finding clear documentation on how to write the pyproject file. For something that is supposed to be the way forward there hasn't been much effort to make it easy to implement. There's also annoyances like the inability to install a script in the search path without implementing it as a module. Something setup.py doesn't require.

For me that is all handled by Poetry. I really like Poetry. But now I've been struggling (off and on) to install my private package from a private repo with PyPI dependencies for a week now...

Re: Underappreciated challenges with Python packaging

#52

Post by idoubtit »

krick wrote: post #36

Is there any consensus on how to deal with packaging and environments in Python by now? Can you suggest me some tutorial for that? I've been out of the loop for a long time, and would like to get an update on how things are in Python in 2023, but I'm not sure if there even is a consensus — what I can find by googling seems to be several kinda competing approaches. This seems surprising, because most "modern" language…

From the sibling thread about packaging and deploying a single script, there was no consensus. There was disagreement on the best way to package, and doubts about the mid term future of some suggested solutions. The following alternatives were suggested:

- package with a `pyproject.toml` file configured to use modern tooling

- package with a `pyproject.toml` file configured to use traditional `setup.py` tooling

- package with traditional `setup.py` tooling

- package with poetry

- package with whatever, deploy with nuikta or pipx

- skip the packaging and deploy with Pyinstaller

- skip the packaging and deploy with nikta

Note that, unless the Python world has radically changed while I was looking away, the packaging does not ensure a simple way to deploy the package and its single script. I remember vividly `pipenv` crashing on me, so switching to venv+pip (or was it virtualenv+pip?) then setting up a bash wrapper to call the Python script with the right venv...

Re: Underappreciated challenges with Python packaging

#53

Post by akx »

kevin_thibedeau wrote: post #37
woodruffw wrote: post #11

Earlier quoted context omitted.

If it's pure Python, the only packaging file you need is `pyproject.toml`. You can fill that file with packaging metadata per PEP 518 and PEP 621, including using modern build tooling like flit[1] for the build backend and build[2] for the frontend. With that, you entire package build (for all distribution types) should be reducible to `python -m build`. Here's an example of a full project doing everything with just…

Good luck finding clear documentation on how to write the pyproject file. For something that is supposed to be the way forward there hasn't been much effort to make it easy to implement. There's also annoyances like the inability to install a script in the search path without implementing it as a module. Something setup.py doesn't require.

The hatch project (PyPA's own tool) has a good guide. https://hatch.pypa.io/latest/config/metadata/#project-metada...

You can also use `hatch new --init` to convert a `setup.py` (whether imperative or setup.cfg-backed) to a `pyproject.toml`.

Re: Underappreciated challenges with Python packaging

#54

Post by akx »

eesmith wrote: post #48
talideon wrote: post #35

Earlier quoted context omitted.

I wouldn't recommend that at all these days. setup.py definitely is not future-proof: it's undergoing deprecation. Better would be a basic pyproject.toml file along the lines of the following: [build-system] requires = ["setuptools"] build-backend = "setuptools.build_meta" [project] name = "foobar" version = "0.0.1" dependencies = [ "...", ] [project.scripts] foo = "foobar:main" See: https://setuptools.pypa.io/en/lat…

> it's undergoing deprecation In the "we would rather people not use this but it's going to stay around for a long time" sense. I strongly doubt it will disappear within the next decade or two. There's a long tail of setup.py-based tools. Last I checked pyproject.toml only supports the simplest of Python/C extensions. Anything fancy, like --with/--without compilation flags to enable/disable optional support, compiler…

The new style to do that is a PEP 517 build backend.

Re: Underappreciated challenges with Python packaging

#55

Post by _frkl »

blackcat201 wrote: post #40
optimalsolver wrote: post #22

Python packaging is a solved problem: https://python-poetry.org/

Recently just migrate a project from pypoetry away to the traditional setup method. Poetry works great for simple package, but once you started to add in complexities, it just falls apart due to everything was abstract away and simplified into config files and command line.

I had the same experience. I think setuptools nowadays is quite good, esp. In combination with setup tools_scm.

Re: Underappreciated challenges with Python packaging

#57

Post by tlocke »

krick wrote: post #36

Is there any consensus on how to deal with packaging and environments in Python by now? Can you suggest me some tutorial for that? I've been out of the loop for a long time, and would like to get an update on how things are in Python in 2023, but I'm not sure if there even is a consensus — what I can find by googling seems to be several kinda competing approaches. This seems surprising, because most "modern" language…

As I see it, the standard way to package Python projects is:

https://packaging.python.org/en/latest/tutorials/packaging-p...

and the longer story is that this method has the flexibility to allow other implementations of packaging tools to be used, and so it fosters choice and competition in the ecosystem. In contrast, the old method of packaging was tied to a particular implementation.

Re: Underappreciated challenges with Python packaging

#58

Post by eesmith »

akx wrote: post #54
eesmith wrote: post #48

Earlier quoted context omitted.

> it's undergoing deprecation In the "we would rather people not use this but it's going to stay around for a long time" sense. I strongly doubt it will disappear within the next decade or two. There's a long tail of setup.py-based tools. Last I checked pyproject.toml only supports the simplest of Python/C extensions. Anything fancy, like --with/--without compilation flags to enable/disable optional support, compiler…

The new style to do that is a PEP 517 build backend.

Yes. That PEP comments:

> The difficulty of interfacing with distutils means that there aren’t many such systems right now, but to give a sense of what we’re thinking about see flit or bento.

Bento is dead. Flit doesn't handle extensions, and points instead to Enscons, which in turn depends on SCons - a build system I have no experience with.

Plus, I sell a source code license. My customers make wheels for their internal PyPI mirrors. I would need to consider how any change might affect them, without the experience to make that judgement.

It seems far easier for me to stay with setup.py than explore a PEP 517 alternative.

So far what I've seen is either people using something like Enscons, or a very complex build system like SciPy's where setuptools just doesn't work. I haven't seen much migration for smaller setup.py systems like mine .. but I also haven't been tracking that well enough.

Any pointers for how that would work?

Re: Underappreciated challenges with Python packaging

#59

Post by deniska »

With modern tooling packaging pure python code to be used by other python developers is a relatively painless process.

The main problem with python packaging is that it's often C/C++ packaging in disguise, among multiple OSes and CPU architectures, and that's far from being solved. Building such python wheel is essentially like building a "portable" (aka one you don't need to properly install into the system) linux/windows/macos application. That comes with a variety of caveats and requires some specialized knowledge one wouldn't pick up playing around with just python alone.

Re: Underappreciated challenges with Python packaging

#60

Post by quietbritishjim »

woodruffw wrote: post #18
quietbritishjim wrote: post #17

Earlier quoted context omitted.

> including using modern build tooling like flit[1] for the build backend and build[2] for the frontend. Or you can use setuptools, which is the package that enables old setup.py builds, as the backend with pyproject.toml. This has the advantage of being mature, unlikely to be abandoned, and possibly some familiarity if you've used it before. Even then, you can use build as the front end build tool.

Yes, setuptools is also perfectly fine. It had some rough edges around PEP 621 support for a while, but those were mostly smoothed out in 2021. (I'll note that maturity is not strong evidence here: distutils is very mature, but is in the process of being deprecated and removed from Python entirely. I don't think that's likely to happen to setuptools, but the fact that behavioral PEPs now exist for all of these tools…

I suppose I meant maturity as in: actively maintained and recommended for use for a long period, which doesn't apply to distutils.

And I should've been more upfront about the real reason for suggesting setuptools: there seem to be a number of build tools that support pyproject.toml, including flit, poetry and setuptools (and I'm sure I've seen at least one other). For me, at least, when I was making a small library recently, it was an overwhelming choice for a part of my project that feels like just admin rather than core business logic. I came close to giving up and just using setup.py with `setup()`. At least setuptools with pyproject.toml is a choice that feels safe; it may not be the best, but it will certainly be good enough that I'm unlikely to regret it later, so I didn't need to spend a lot of time looking at the detailed pros and cons of all the choices.

Post reply on HN