Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Underappreciated challenges with Python packaging

pypackaging-native.github.io

Post reply on HN
Print view View original

41–50 of 75 posts

Re: Underappreciated challenges with Python packaging

#41

Post by woodruffw »

kevin_thibedeau wrote: post #39
woodruffw wrote: post #38

Earlier quoted context omitted.

> Good luck finding clear documentation on how to write the pyproject file. For something that is supposed to be the way forward there hasn't been much effort to make it easy to implement. PEP 621, which I mentioned, covers the format of `pyproject.toml` in detail. I also linked an example which, to the best of my knowledge, covers all current best practices for that file. > There's also annoyances like the inability…

The PEP in no way explains how to write a usable pyproject for ordinary projects. It's basically just targeted at people developing installers. I meant package. A directory with an __init__.py. You can't install standalone script.py (or a generated wrapper) as /usr/local/bin/script with a pyproject.

> The PEP in no way explains how to write a usable pyproject for ordinary projects. It's basically just targeted at people developing installers.

Did you look at it[1]?

> I meant package. A directory with an __init__.py. You can't install standalone script.py (or a generated wrapper) as /usr/local/bin/script with a pyproject.

I still don't think I understand what your expectation is here: a `pyproject.toml` is just a metadata specification. The only difference between it and `setup.py` is that the latter is arbitrary code.

There's an old, long deprecated way to use `setup.py`, namely `setup.py install`. But that's been discouraged in favor of `pip install` for years, which behaves precisely the same way with `pyproject.toml`. If you want to install a script into `/usr/local/bin`, `pip install` with a package specified in `pyproject.toml` will work just fine.

[1]: https://peps.python.org/pep-0621/#example

Re: Underappreciated challenges with Python packaging

#42

Post by saila »

krick wrote: post #36

Is there any consensus on how to deal with packaging and environments in Python by now? Can you suggest me some tutorial for that? I've been out of the loop for a long time, and would like to get an update on how things are in Python in 2023, but I'm not sure if there even is a consensus — what I can find by googling seems to be several kinda competing approaches. This seems surprising, because most "modern" language…

I don't think there's a consensus, but there are some good modern options. I think poetry is a good choice, and it seems to be fairly popular. I use it for all my Python projects and haven't found a compelling reason to switch to another option in the past few years.

As to the question of whether you should ever use pip to install packages globally, the answer is almost always no. For command line tools, the best option IMO is pipx. The second best option is pip install --user.

If you're developing a library or application, you should always isolate it in a virtualenv, which is something poetry will handle for you when you run `poetry install`.

Re: Underappreciated challenges with Python packaging

#43

Post by draw_down »

blackcat201 wrote: post #40
optimalsolver wrote: post #22

Python packaging is a solved problem: https://python-poetry.org/

Recently just migrate a project from pypoetry away to the traditional setup method. Poetry works great for simple package, but once you started to add in complexities, it just falls apart due to everything was abstract away and simplified into config files and command line.

[dead]

Re: Underappreciated challenges with Python packaging

#44

Post by egberts1 »

woodruffw wrote: post #41
kevin_thibedeau wrote: post #39

Earlier quoted context omitted.

The PEP in no way explains how to write a usable pyproject for ordinary projects. It's basically just targeted at people developing installers. I meant package. A directory with an __init__.py. You can't install standalone script.py (or a generated wrapper) as /usr/local/bin/script with a pyproject.

> The PEP in no way explains how to write a usable pyproject for ordinary projects. It's basically just targeted at people developing installers. Did you look at it[1]? > I meant package. A directory with an __init__.py. You can't install standalone script.py (or a generated wrapper) as /usr/local/bin/script with a pyproject. I still don't think I understand what your expectation is here: a `pyproject.toml` is just a…

Not doing that either … until there are ways for only the person (me) to hold the key to the revision submission into pip.

Re: Underappreciated challenges with Python packaging

#46

Post by woodruffw »

egberts1 wrote: post #44
woodruffw wrote: post #41

Earlier quoted context omitted.

> The PEP in no way explains how to write a usable pyproject for ordinary projects. It's basically just targeted at people developing installers. Did you look at it[1]? > I meant package. A directory with an __init__.py. You can't install standalone script.py (or a generated wrapper) as /usr/local/bin/script with a pyproject. I still don't think I understand what your expectation is here: a `pyproject.toml` is just a…

Not doing that either … until there are ways for only the person (me) to hold the key to the revision submission into pip.

You're confusing `pip` with PyPI. `pip` is a package installer; you can use it to install local packages, or packages that are hosted on an index. In this case, we're solely talking about local packages.

Re: Underappreciated challenges with Python packaging

#47

Post by physPop »

ris wrote: post #26

This is exactly why I avoid pip et al whenever I can and just use Nix.

Do you know of any up to date blogs/howtos/guides on nix+python where the python project contains modules that need to be compiled (eg Cython, pybind, etc)? I've found the basic info at https://nixos.wiki/wiki/Packaging/Python but it doesn't really go in depth for more complex use cases than having a setup.py...

Re: Underappreciated challenges with Python packaging

#48

Post by eesmith »

talideon wrote: post #35
mixmastamyk wrote: post #19

Earlier quoted context omitted.

Keep it simple and fashion-proof. Been using setup.py for a one-script package for one or two? decades: from setuptools import setup setup( name = 'foobar', scripts = ['foo'], # install a script from current fldr # ... ) A few years ago I had to start using twine to register and upload it to pypi.

I wouldn't recommend that at all these days. setup.py definitely is not future-proof: it's undergoing deprecation. Better would be a basic pyproject.toml file along the lines of the following: [build-system] requires = ["setuptools"] build-backend = "setuptools.build_meta" [project] name = "foobar" version = "0.0.1" dependencies = [ "...", ] [project.scripts] foo = "foobar:main" See: https://setuptools.pypa.io/en/lat…

> it's undergoing deprecation

In the "we would rather people not use this but it's going to stay around for a long time" sense. I strongly doubt it will disappear within the next decade or two. There's a long tail of setup.py-based tools.

Last I checked pyproject.toml only supports the simplest of Python/C extensions. Anything fancy, like --with/--without compilation flags to enable/disable optional support, compiler-specific flags (in my case, to add OpenMP), compile-time code generation (like using yacc/lex), etc. requires a setup.py and a bunch of hacking.

Re: Underappreciated challenges with Python packaging

#49

Post by japanuspus »

CJefferson wrote: post #10

I've been planning on packaging a python package recently, and the internet is annoyingly full of guides which are, I think, out of date. They at least suggest quite different things. I just have a single python file, meant to be treated as an executable (no package at present). There are a whole bunch of tests, but that's obviously separate. Any suggestions on modern best practices welcome!

As detailed in the other answers, there are two parts to this: 1) Creating a python package from your project (and possibly share this on pypi), and 2) Making this package available as an end-user application.

For step 2 you can use nuitka or similar, but if your audience is somewhat developer-oriented, you can also propose for them to use pipx: https://github.com/pypa/pipx.

Re: Underappreciated challenges with Python packaging

#50

Post by williamvds »

physPop wrote: post #47
ris wrote: post #26

This is exactly why I avoid pip et al whenever I can and just use Nix.

Do you know of any up to date blogs/howtos/guides on nix+python where the python project contains modules that need to be compiled (eg Cython, pybind, etc)? I've found the basic info at https://nixos.wiki/wiki/Packaging/Python but it doesn't really go in depth for more complex use cases than having a setup.py...

Have you checked out the manual?

https://nixos.org/manual/nixpkgs/stable/#python

Particularly section 17.27.1.2. Developing with Python. Combine that with a generic guide on using Flakes and that should get you started

Post reply on HN