Live data from Hacker News

Underappreciated challenges with Python packaging

pypackaging-native.github.io

71–75 of 75 posts

Re: Underappreciated challenges with Python packaging

#71

Earlier quoted context omitted.

I don't know how common this is, but some NPM packages involve native binaries, so it must be doable. pg-native for example.

pg-native is a good example actually. Its readme lines out how you need to first get a compiler, libpq, and have some certain commands in your PATH. With psycopg2 (Python’s equivalent), the most common scenario is ‘pip install paycopg2-binary’ and you’re good to go.

[deleted]

Re: Underappreciated challenges with Python packaging

#72

Earlier quoted context omitted.

I don't know how common this is, but some NPM packages involve native binaries, so it must be doable. pg-native for example.

pg-native is a good example actually. Its readme lines out how you need to first get a compiler, libpq, and have some certain commands in your PATH. With psycopg2 (Python’s equivalent), the most common scenario is ‘pip install paycopg2-binary’ and you’re good to go.

So the difference is psycopg2-binary* bundles the libpq native code and pg-native doesn't? I'm no expert, but I think npm packages can include native code if they want thanks to node-gyp, only the node-libpq (which pg-native relies on) author seemingly decided not to package in libpq itself.

* Back when I used this, there was just psycopg2 which had the bin included.

Re: Underappreciated challenges with Python packaging

#73
post #64
post #34

Earlier quoted context omitted.

> How is [Python] very different from NodeJS? Unlike Node, Python is essentially older than modern package management . When Python developers first decided to tackle distributing their code, `apt-get` did not yet exist. Early approaches which stuck around way too long let any package do more or less anything at install time, and didn't bother with static metadata (can't figure out what your deps are except by attemp…

Not really. CPAN was already a thing when Python itself was released, long before there was any Python packages to share.

What did cpan.pm actually look like in 1993? Did packages ('distributions') have static metadata? Could the installer perform recursive dependency resolution? Was the resolver complete?

My impression is that none of that resembled contemporary CPAN, and isn't really what I have in mind with the admittedly ambiguous phrase 'modern package management'.

But I'd love to hear more! The history of package management is very interesting to me. Tales of ancient but sophisticated package management systems are very welcome. :)

Re: Underappreciated challenges with Python packaging

#74

Earlier quoted context omitted.

pg-native is a good example actually. Its readme lines out how you need to first get a compiler, libpq, and have some certain commands in your PATH. With psycopg2 (Python’s equivalent), the most common scenario is ‘pip install paycopg2-binary’ and you’re good to go.

So the difference is psycopg2-binary* bundles the libpq native code and pg-native doesn't? I'm no expert, but I think npm packages can include native code if they want thanks to node-gyp, only the node-libpq (which pg-native relies on) author seemingly decided not to package in libpq itself. * Back when I used this, there was just psycopg2 which had the bin included.

Correct, but the challenges are to compile the dynamic library correctly so it runs on another machine, and on a given target machine choose to load a correct compiled artifact. Python has extremely good support for those compared to other similar ecosystems. The fact that most Python packages decide to do bundling on a comparably broard set of platforms, while packages on some other language are not, is a window to understand how the operation is made much easier by the ecosystem.

Re: Underappreciated challenges with Python packaging

#75
post #2

Python packaging’s greatest challenge is 10 competing tools and standards.

At this point, there are 10 competing tools but no longer so many competing standards: the standards for Python packaging from 2015 onwards are PEP 517 (build system standardization), PEP 518 (using pyproject.toml to configure the build system), and PEP 621 (storing project metadata, previously standardized, in pyproject.toml). These standards build on top of each other, meaning that they don't offer conflicting advi…

The problem is, as long as old python versions continue to exist, the competing standards also will continue to exist. From a user experience stand point it is horrible. Depending on the python, pip, or Setup tools version on a system the Install command might do something drastically different in each case. Often it’s not even clear what’s happening under the hood.
Post reply on HN