Underappreciated challenges with Python packaging • Hacker News Board

Re: Underappreciated challenges with Python packaging

#31

Post by JAlexoid » Wed, Jan 04, 2023, 1:52 AM UTC

atoav wrote: post #9

groodt wrote: post #3

Earlier quoted context omitted.

Python packaging gets a lot of criticism. It's a meme. The thing is, it's actually improved dramatically over the years and continues to improve. The problems it solves are very complex if one looks a little below the surface. It is solving different problems to the ecosystems that it's often compared to: golang, rust, java, js.

This is true, still: Everyday things that should be straightforward, or pythonic if you will, are way to convoluted and complicated. As a python dev with experience since 2.6 I agree it has gotten better, but it is also rotten at it's core. The problems python has to solve there are hard , but this is why it should be priority number one to solve them elegantly and in a fashion so that 99% of python users never have…

Well... The setuptools package was around for way too long.

I never managed to get data files packaged into a dist elegantly... annoys me to this day.

Re: Underappreciated challenges with Python packaging

#32

Post by uranusjr » Wed, Jan 04, 2023, 2:34 AM UTC

david2ndaccount wrote: post #20

wiredfool wrote: post #7

Pillow has most of the issues that are listed in the article. (oddly enough for a graphics library, the GPU part is the only part that I don't think we've stumbled over at one point or another.) From a quality of life issue -- having the sdist install behind an opt-in flag by default for our package would be great. Unless you're a developer with a lot of -dev packages for imaging libraries already on your system, you…

Could you release the sdist as a separate package and only upload binary wheels for a normal install?

You could, and that’s what psycopg2 did for a while (for a different reason but effectively they separated source and binary releases as different projects). This solves an immediate problem but introduces new challenges, probably the most significant being Python does not allow OR dependencies (I want at least one of those) and splitting the package made dependant projects’ lives miserable.

Re: Underappreciated challenges with Python packaging

#33

Post by uranusjr » Wed, Jan 04, 2023, 2:45 AM UTC

hot_gril wrote: post #24

groodt wrote: post #3

Earlier quoted context omitted.

Python packaging gets a lot of criticism. It's a meme. The thing is, it's actually improved dramatically over the years and continues to improve. The problems it solves are very complex if one looks a little below the surface. It is solving different problems to the ecosystems that it's often compared to: golang, rust, java, js.

How is it very different from NodeJS? Cause I find npm way easier to deal with than Python packaging, and it's also dealing with native code. I used Python heavily for 6 years and still have no idea how the packages work as a user trying to install libs; I used to just thrash around till it works. I don't use it anymore at my new job. The one thing I understand is npm installs everything locally by default (unless yo…

Fundamentally the problems are the same, but the communities have very different priorities. The vast majority of Node is still web dev and the community focuses on dependencies that are deployed as JavaScript files (including transpiled code); this allows NPM to mostly ignore issues related to linking to native binaries and put more focus on those user-facing features you see and like. Python on the other hand has a lot of stakes on interacting with native binaries (scientific computing, automation, etc.) and with systems older than Node’s existence. This consumes up a ton of the community’s resource, and every new UX improvement also needs to take account of those use cases. If you’re ever tasked with keeping projects with native dependencies working over a moderate period of time in both languages, you’d gain a lot of respect on how well Python packaging works.

Re: Underappreciated challenges with Python packaging

#34

Post by pxc » Wed, Jan 04, 2023, 2:47 AM UTC

hot_gril wrote: post #24

groodt wrote: post #3

Earlier quoted context omitted.

Python packaging gets a lot of criticism. It's a meme. The thing is, it's actually improved dramatically over the years and continues to improve. The problems it solves are very complex if one looks a little below the surface. It is solving different problems to the ecosystems that it's often compared to: golang, rust, java, js.

How is it very different from NodeJS? Cause I find npm way easier to deal with than Python packaging, and it's also dealing with native code. I used Python heavily for 6 years and still have no idea how the packages work as a user trying to install libs; I used to just thrash around till it works. I don't use it anymore at my new job. The one thing I understand is npm installs everything locally by default (unless yo…

> How is [Python] very different from NodeJS?

Unlike Node, Python is essentially older than modern package management. When Python developers first decided to tackle distributing their code, `apt-get` did not yet exist.

Early approaches which stuck around way too long let any package do more or less anything at install time, and didn't bother with static metadata (can't figure out what your deps are except by attempting an install!). Subsequent solutions have struggled to build consensus in an already vast and mature ecosystem. Backwards compatibility means compatibility with bad conventions.

Re: Underappreciated challenges with Python packaging

#35

Post by talideon » Wed, Jan 04, 2023, 3:29 AM UTC

mixmastamyk wrote: post #19

CJefferson wrote: post #10

I've been planning on packaging a python package recently, and the internet is annoyingly full of guides which are, I think, out of date. They at least suggest quite different things. I just have a single python file, meant to be treated as an executable (no package at present). There are a whole bunch of tests, but that's obviously separate. Any suggestions on modern best practices welcome!

Keep it simple and fashion-proof. Been using setup.py for a one-script package for one or two? decades: from setuptools import setup setup( name = 'foobar', scripts = ['foo'], # install a script from current fldr # ... ) A few years ago I had to start using twine to register and upload it to pypi.

I wouldn't recommend that at all these days. setup.py definitely is not future-proof: it's undergoing deprecation.

Better would be a basic pyproject.toml file along the lines of the following:

    [build-system]
    requires = ["setuptools"]
    build-backend = "setuptools.build_meta"

    [project]
    name = "foobar"
    version = "0.0.1"
    dependencies = [
        "...",
    ]

    [project.scripts]
    foo = "foobar:main"

See: https://setuptools.pypa.io/en/latest/userguide/quickstart.ht...

Re: Underappreciated challenges with Python packaging

#36

Post by krick » Wed, Jan 04, 2023, 3:51 AM UTC

Is there any consensus on how to deal with packaging and environments in Python by now? Can you suggest me some tutorial for that?

I've been out of the loop for a long time, and would like to get an update on how things are in Python in 2023, but I'm not sure if there even is a consensus — what I can find by googling seems to be several kinda competing approaches. This seems surprising, because most "modern" languages seem to have a very well defined set of practices to deal with all of that stuff. Some languages already come with their built-in stuff (Go, Rust), others simply have well-known solutions (like, technically there still exist PEAR and PECL for PHP, but everyone just knows how to use composer, which solves both packaging and dependency-management problems, and it's also pretty clear what problems it doesn't solve).

For Python there seems to be like a dozen of tools and I'm not sure which are outdated and not used by anyone, which are useless fancy wrappers (not used by anyone) and what is the actual go-to tool (if there is any) for all common cases. Dependency-management, version locking, shipping an executable, environment separation for some local scripts, should I even ever use pip install globally, etc.

Re: Underappreciated challenges with Python packaging

#37

Post by kevin_thibedeau » Wed, Jan 04, 2023, 3:53 AM UTC

woodruffw wrote: post #11

CJefferson wrote: post #10

I've been planning on packaging a python package recently, and the internet is annoyingly full of guides which are, I think, out of date. They at least suggest quite different things. I just have a single python file, meant to be treated as an executable (no package at present). There are a whole bunch of tests, but that's obviously separate. Any suggestions on modern best practices welcome!

If it's pure Python, the only packaging file you need is `pyproject.toml`. You can fill that file with packaging metadata per PEP 518 and PEP 621, including using modern build tooling like flit[1] for the build backend and build[2] for the frontend. With that, you entire package build (for all distribution types) should be reducible to `python -m build`. Here's an example of a full project doing everything with just…

Good luck finding clear documentation on how to write the pyproject file. For something that is supposed to be the way forward there hasn't been much effort to make it easy to implement.

There's also annoyances like the inability to install a script in the search path without implementing it as a module. Something setup.py doesn't require.

Re: Underappreciated challenges with Python packaging

#38

Post by woodruffw » Wed, Jan 04, 2023, 4:12 AM UTC

kevin_thibedeau wrote: post #37

woodruffw wrote: post #11

Earlier quoted context omitted.

If it's pure Python, the only packaging file you need is `pyproject.toml`. You can fill that file with packaging metadata per PEP 518 and PEP 621, including using modern build tooling like flit[1] for the build backend and build[2] for the frontend. With that, you entire package build (for all distribution types) should be reducible to `python -m build`. Here's an example of a full project doing everything with just…

Good luck finding clear documentation on how to write the pyproject file. For something that is supposed to be the way forward there hasn't been much effort to make it easy to implement. There's also annoyances like the inability to install a script in the search path without implementing it as a module. Something setup.py doesn't require.

> Good luck finding clear documentation on how to write the pyproject file. For something that is supposed to be the way forward there hasn't been much effort to make it easy to implement.

PEP 621, which I mentioned, covers the format of `pyproject.toml` in detail. I also linked an example which, to the best of my knowledge, covers all current best practices for that file.

> There's also annoyances like the inability to install a script in the search path without implementing it as a module. Something setup.py doesn't require.

I'm not sure I'm following. A module in Python is just a Python file, so your script is a module. Are you saying that you can't distribute single-module packages with pyproject.toml? Because I don't think that's true.

Re: Underappreciated challenges with Python packaging

#39

Post by kevin_thibedeau » Wed, Jan 04, 2023, 4:16 AM UTC

woodruffw wrote: post #38

kevin_thibedeau wrote: post #37

Earlier quoted context omitted.

Good luck finding clear documentation on how to write the pyproject file. For something that is supposed to be the way forward there hasn't been much effort to make it easy to implement. There's also annoyances like the inability to install a script in the search path without implementing it as a module. Something setup.py doesn't require.

> Good luck finding clear documentation on how to write the pyproject file. For something that is supposed to be the way forward there hasn't been much effort to make it easy to implement. PEP 621, which I mentioned, covers the format of `pyproject.toml` in detail. I also linked an example which, to the best of my knowledge, covers all current best practices for that file. > There's also annoyances like the inability…

The PEP in no way explains how to write a usable pyproject for ordinary projects. It's basically just targeted at people developing installers.

I meant package. A directory with an __init__.py. You can't install standalone script.py (or a generated wrapper) as /usr/local/bin/script with a pyproject.

Re: Underappreciated challenges with Python packaging

#40

Post by blackcat201 » Wed, Jan 04, 2023, 4:19 AM UTC

optimalsolver wrote: post #22

Python packaging is a solved problem: https://python-poetry.org/

Recently just migrate a project from pypoetry away to the traditional setup method. Poetry works great for simple package, but once you started to add in complexities, it just falls apart due to everything was abstract away and simplified into config files and command line.