Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Underappreciated challenges with Python packaging

pypackaging-native.github.io

Post reply on HN
Print view View original

11–20 of 75 posts

Re: Underappreciated challenges with Python packaging

#11

Post by woodruffw »

CJefferson wrote: post #10

I've been planning on packaging a python package recently, and the internet is annoyingly full of guides which are, I think, out of date. They at least suggest quite different things. I just have a single python file, meant to be treated as an executable (no package at present). There are a whole bunch of tests, but that's obviously separate. Any suggestions on modern best practices welcome!

If it's pure Python, the only packaging file you need is `pyproject.toml`. You can fill that file with packaging metadata per PEP 518 and PEP 621, including using modern build tooling like flit[1] for the build backend and build[2] for the frontend.

With that, you entire package build (for all distribution types) should be reducible to `python -m build`. Here's an example of a full project doing everything with just `pyproject.toml`[3] (FD: my project).

[1]: https://github.com/pypa/flit

[2]: https://github.com/pypa/build

[3]: https://github.com/pypa/pip-audit

Re: Underappreciated challenges with Python packaging

#12

Post by ensignavenger »

CJefferson wrote: post #10

I've been planning on packaging a python package recently, and the internet is annoyingly full of guides which are, I think, out of date. They at least suggest quite different things. I just have a single python file, meant to be treated as an executable (no package at present). There are a whole bunch of tests, but that's obviously separate. Any suggestions on modern best practices welcome!

If you are wanting to release it to pypi as a python package, I would personally use Poetry. But your case- a single pure Python package, is a simple case that won't have many problems like are brought up in the article, whatever tool you use.

If you want a stand alone executable, I haven't found a good, single, cross platform tool for that yet... seems like there is a separate tool for each platform.

Re: Underappreciated challenges with Python packaging

#13

Post by woodruffw »

0x008 wrote: post #2

Python packaging’s greatest challenge is 10 competing tools and standards.

At this point, there are 10 competing tools but no longer so many competing standards: the standards for Python packaging from 2015 onwards are PEP 517 (build system standardization), PEP 518 (using pyproject.toml to configure the build system), and PEP 621 (storing project metadata, previously standardized, in pyproject.toml). These standards build on top of each other, meaning that they don't offer conflicting advice.

The TL;DR for Python packaging in 2022 is that, unless you're building CPython extensions, you can do everything through pyproject.toml with PyPA-maintained tooling.

Re: Underappreciated challenges with Python packaging

#15

Post by quietbritishjim »

ensignavenger wrote: post #12
CJefferson wrote: post #10

I've been planning on packaging a python package recently, and the internet is annoyingly full of guides which are, I think, out of date. They at least suggest quite different things. I just have a single python file, meant to be treated as an executable (no package at present). There are a whole bunch of tests, but that's obviously separate. Any suggestions on modern best practices welcome!

If you are wanting to release it to pypi as a python package, I would personally use Poetry. But your case- a single pure Python package, is a simple case that won't have many problems like are brought up in the article, whatever tool you use. If you want a stand alone executable, I haven't found a good, single, cross platform tool for that yet... seems like there is a separate tool for each platform.

Nuitka works on Windows, Linux and Mac

https://nuitka.net/

Re: Underappreciated challenges with Python packaging

#16

Post by irskep »

This is an incredible example of organizing information well and making a case to a wide audience. It's difficult enough to shave all the yaks necessary to get a high-level view of all issues related to a problem, and to express all those problems in good writing is an additional tough challenge. These folks have done an amazing job at both.

Shoutout to Material for MkDocs enabling the swanky theme and Markdown extensions. https://squidfunk.github.io/mkdocs-material/

Re: Underappreciated challenges with Python packaging

#17

Post by quietbritishjim »

woodruffw wrote: post #11
CJefferson wrote: post #10

I've been planning on packaging a python package recently, and the internet is annoyingly full of guides which are, I think, out of date. They at least suggest quite different things. I just have a single python file, meant to be treated as an executable (no package at present). There are a whole bunch of tests, but that's obviously separate. Any suggestions on modern best practices welcome!

If it's pure Python, the only packaging file you need is `pyproject.toml`. You can fill that file with packaging metadata per PEP 518 and PEP 621, including using modern build tooling like flit[1] for the build backend and build[2] for the frontend. With that, you entire package build (for all distribution types) should be reducible to `python -m build`. Here's an example of a full project doing everything with just…

> including using modern build tooling like flit[1] for the build backend and build[2] for the frontend.

Or you can use setuptools, which is the package that enables old setup.py builds, as the backend with pyproject.toml. This has the advantage of being mature, unlikely to be abandoned, and possibly some familiarity if you've used it before. Even then, you can use build as the front end build tool.

Re: Underappreciated challenges with Python packaging

#18

Post by woodruffw »

quietbritishjim wrote: post #17
woodruffw wrote: post #11

Earlier quoted context omitted.

If it's pure Python, the only packaging file you need is `pyproject.toml`. You can fill that file with packaging metadata per PEP 518 and PEP 621, including using modern build tooling like flit[1] for the build backend and build[2] for the frontend. With that, you entire package build (for all distribution types) should be reducible to `python -m build`. Here's an example of a full project doing everything with just…

> including using modern build tooling like flit[1] for the build backend and build[2] for the frontend. Or you can use setuptools, which is the package that enables old setup.py builds, as the backend with pyproject.toml. This has the advantage of being mature, unlikely to be abandoned, and possibly some familiarity if you've used it before. Even then, you can use build as the front end build tool.

Yes, setuptools is also perfectly fine. It had some rough edges around PEP 621 support for a while, but those were mostly smoothed out in 2021.

(I'll note that maturity is not strong evidence here: distutils is very mature, but is in the process of being deprecated and removed from Python entirely. I don't think that's likely to happen to setuptools, but the fact that behavioral PEPs now exist for all of these tools means that the decline/abandonment of any poses much less of an ecosystem risk.)

Re: Underappreciated challenges with Python packaging

#19

Post by mixmastamyk »

CJefferson wrote: post #10

I've been planning on packaging a python package recently, and the internet is annoyingly full of guides which are, I think, out of date. They at least suggest quite different things. I just have a single python file, meant to be treated as an executable (no package at present). There are a whole bunch of tests, but that's obviously separate. Any suggestions on modern best practices welcome!

Keep it simple and fashion-proof. Been using setup.py for a one-script package for one or two? decades:

    from setuptools import setup

    setup(
        name          = 'foobar',
        scripts       = ['foo'],  # install a script from current fldr
        # ...
    )
A few years ago I had to start using twine to register and upload it to pypi.

Re: Underappreciated challenges with Python packaging

#20

Post by david2ndaccount »

wiredfool wrote: post #7

Pillow has most of the issues that are listed in the article. (oddly enough for a graphics library, the GPU part is the only part that I don't think we've stumbled over at one point or another.) From a quality of life issue -- having the sdist install behind an opt-in flag by default for our package would be great. Unless you're a developer with a lot of -dev packages for imaging libraries already on your system, you…

Could you release the sdist as a separate package and only upload binary wheels for a normal install?
Post reply on HN