Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Underappreciated challenges with Python packaging

pypackaging-native.github.io

Post reply on HN
Print view View original

61–70 of 75 posts

Re: Underappreciated challenges with Python packaging

#61

Post by jborean93 »

eesmith wrote: post #58
akx wrote: post #54

Earlier quoted context omitted.

The new style to do that is a PEP 517 build backend.

Yes. That PEP comments: > The difficulty of interfacing with distutils means that there aren’t many such systems right now, but to give a sense of what we’re thinking about see flit or bento. Bento is dead. Flit doesn't handle extensions, and points instead to Enscons, which in turn depends on SCons - a build system I have no experience with. Plus, I sell a source code license. My customers make wheels for their inte…

You should continue to use the setup.py to define the extensions but put all the remaining metadata in the pyproject.toml. The pyproject.toml will reference setuptools as your build backend like https://github.com/jborean93/pyspnego/blob/main/pyproject.to... and the setup.py will reference anything that cannot be expressed in the pyproject.toml like C extensions https://github.com/jborean93/pyspnego/blob/main/setup.py.

The benefits of this is now your project has metadata that tools like pip/poetry/etc can use to figure out what is required (Python project wise) to build your project. For example pip will create an isolated venv with setuptools and Cython for the project I listed when installing from the sdist. You can now also take advantage of `python -m build` to build this project rather than a setuptools specific incantation. This is universal across all build providers so if you want to change to poetry in the future you can will hopefully no build script changes.

Re: Underappreciated challenges with Python packaging

#62

Post by tpoacher »

CJefferson wrote: post #10

I've been planning on packaging a python package recently, and the internet is annoyingly full of guides which are, I think, out of date. They at least suggest quite different things. I just have a single python file, meant to be treated as an executable (no package at present). There are a whole bunch of tests, but that's obviously separate. Any suggestions on modern best practices welcome!

The approach I prefer is to not mess with setuptools etc at all in the first place, and simply make a nice executable package.

e.g. https://github.com/tpapastylianou/self-contained-runnable-py...

Re: Underappreciated challenges with Python packaging

#63

Post by eesmith »

jborean93 wrote: post #61
eesmith wrote: post #58

Earlier quoted context omitted.

Yes. That PEP comments: > The difficulty of interfacing with distutils means that there aren’t many such systems right now, but to give a sense of what we’re thinking about see flit or bento. Bento is dead. Flit doesn't handle extensions, and points instead to Enscons, which in turn depends on SCons - a build system I have no experience with. Plus, I sell a source code license. My customers make wheels for their inte…

You should continue to use the setup.py to define the extensions but put all the remaining metadata in the pyproject.toml. The pyproject.toml will reference setuptools as your build backend like https://github.com/jborean93/pyspnego/blob/main/pyproject.to... and the setup.py will reference anything that cannot be expressed in the pyproject.toml like C extensions https://github.com/jborean93/pyspnego/blob/main/setup.p…

I know my project is an oddball. So far I have no required external dependencies, and my optional dependencies are for what the linked-to pages refer to as "native dependencies", which can't be specified by pyproject.toml.

My "setuptools specific incantation" is "pip install" or "pip install -e". I do have a setup.cfg.

The recommendation last year was "If you're building an application, use Poetry. If you're building a library, use Flit", and since my package is a library, I've never really considered poetry.

But! I'm switching from argparse to click - my first required dependency! - so within a month or so I'll be putting my toes into the pyproject.toml waters.

Thank you for your pointers. Isn't there also a way to specify the requirements for building the documentation? I didn't see it in your example.

Re: Underappreciated challenges with Python packaging

#64

Post by xorcist »

pxc wrote: post #34
hot_gril wrote: post #24

Earlier quoted context omitted.

How is it very different from NodeJS? Cause I find npm way easier to deal with than Python packaging, and it's also dealing with native code. I used Python heavily for 6 years and still have no idea how the packages work as a user trying to install libs; I used to just thrash around till it works. I don't use it anymore at my new job. The one thing I understand is npm installs everything locally by default (unless yo…

> How is [Python] very different from NodeJS? Unlike Node, Python is essentially older than modern package management . When Python developers first decided to tackle distributing their code, `apt-get` did not yet exist. Early approaches which stuck around way too long let any package do more or less anything at install time, and didn't bother with static metadata (can't figure out what your deps are except by attemp…

Not really. CPAN was already a thing when Python itself was released, long before there was any Python packages to share.

Re: Underappreciated challenges with Python packaging

#65

Post by hot_gril »

uranusjr wrote: post #33
hot_gril wrote: post #24

Earlier quoted context omitted.

How is it very different from NodeJS? Cause I find npm way easier to deal with than Python packaging, and it's also dealing with native code. I used Python heavily for 6 years and still have no idea how the packages work as a user trying to install libs; I used to just thrash around till it works. I don't use it anymore at my new job. The one thing I understand is npm installs everything locally by default (unless yo…

Fundamentally the problems are the same, but the communities have very different priorities. The vast majority of Node is still web dev and the community focuses on dependencies that are deployed as JavaScript files (including transpiled code); this allows NPM to mostly ignore issues related to linking to native binaries and put more focus on those user-facing features you see and like. Python on the other hand has a…

I don't know how common this is, but some NPM packages involve native binaries, so it must be doable. pg-native for example.

Re: Underappreciated challenges with Python packaging

#66

Post by hot_gril »

pxc wrote: post #34
hot_gril wrote: post #24

Earlier quoted context omitted.

How is it very different from NodeJS? Cause I find npm way easier to deal with than Python packaging, and it's also dealing with native code. I used Python heavily for 6 years and still have no idea how the packages work as a user trying to install libs; I used to just thrash around till it works. I don't use it anymore at my new job. The one thing I understand is npm installs everything locally by default (unless yo…

> How is [Python] very different from NodeJS? Unlike Node, Python is essentially older than modern package management . When Python developers first decided to tackle distributing their code, `apt-get` did not yet exist. Early approaches which stuck around way too long let any package do more or less anything at install time, and didn't bother with static metadata (can't figure out what your deps are except by attemp…

That makes sense. Backwards compatibility can be a large burden. I'm still sad that they haven't managed to make a new thing that just works. NodeJS is repurposing a language and runtime originally meant for hacky web scripting, but it ended up ok in the end.

In general, if a community doesn't agree on how to fix a problem, someone else will provide a solution at a higher layer. Now it's common to install a whole Docker image to run some Python code instead of a few Python deps.

Re: Underappreciated challenges with Python packaging

#67

Post by mixmastamyk »

talideon wrote: post #35
mixmastamyk wrote: post #19

Earlier quoted context omitted.

Keep it simple and fashion-proof. Been using setup.py for a one-script package for one or two? decades: from setuptools import setup setup( name = 'foobar', scripts = ['foo'], # install a script from current fldr # ... ) A few years ago I had to start using twine to register and upload it to pypi.

I wouldn't recommend that at all these days. setup.py definitely is not future-proof: it's undergoing deprecation. Better would be a basic pyproject.toml file along the lines of the following: [build-system] requires = ["setuptools"] build-backend = "setuptools.build_meta" [project] name = "foobar" version = "0.0.1" dependencies = [ "...", ] [project.scripts] foo = "foobar:main" See: https://setuptools.pypa.io/en/lat…

“Undergoing deprecation” as in not deprecated yet. Great majority of pkgs using it, and Python takes a decade+ to deprecate things.

Also this is a few lines of well understood Python, not exactly a huge investment, right? Does several lines even need to be future proof?

My bet is you’ll need to modify the toml solution more often than the setup.py in the next decade.

Re: Underappreciated challenges with Python packaging

#68

Post by mixmastamyk »

tomrod wrote: post #23
mixmastamyk wrote: post #19

Earlier quoted context omitted.

Keep it simple and fashion-proof. Been using setup.py for a one-script package for one or two? decades: from setuptools import setup setup( name = 'foobar', scripts = ['foo'], # install a script from current fldr # ... ) A few years ago I had to start using twine to register and upload it to pypi.

Is there a place you recommend where I can learn more about this?

It’s documented, try google, stackoverflow, and reading other packages’ setup.py for tricks.

However the scripts=[], keyword is the key to the case above.

Re: Underappreciated challenges with Python packaging

#69

Post by woodruffw »

quietbritishjim wrote: post #60
woodruffw wrote: post #18

Earlier quoted context omitted.

Yes, setuptools is also perfectly fine. It had some rough edges around PEP 621 support for a while, but those were mostly smoothed out in 2021. (I'll note that maturity is not strong evidence here: distutils is very mature, but is in the process of being deprecated and removed from Python entirely. I don't think that's likely to happen to setuptools, but the fact that behavioral PEPs now exist for all of these tools…

I suppose I meant maturity as in: actively maintained and recommended for use for a long period, which doesn't apply to distutils. And I should've been more upfront about the real reason for suggesting setuptools: there seem to be a number of build tools that support pyproject.toml, including flit, poetry and setuptools (and I'm sure I've seen at least one other). For me, at least, when I was making a small library r…

That's very reasonable! I don't mean to disparage that decision at all: setuptools is rock solid and a very safe choice.

Re: Underappreciated challenges with Python packaging

#70

Post by uranusjr »

hot_gril wrote: post #65
uranusjr wrote: post #33

Earlier quoted context omitted.

Fundamentally the problems are the same, but the communities have very different priorities. The vast majority of Node is still web dev and the community focuses on dependencies that are deployed as JavaScript files (including transpiled code); this allows NPM to mostly ignore issues related to linking to native binaries and put more focus on those user-facing features you see and like. Python on the other hand has a…

I don't know how common this is, but some NPM packages involve native binaries, so it must be doable. pg-native for example.

pg-native is a good example actually. Its readme lines out how you need to first get a compiler, libpq, and have some certain commands in your PATH. With psycopg2 (Python’s equivalent), the most common scenario is ‘pip install paycopg2-binary’ and you’re good to go.
Post reply on HN