Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Thoughts on the Python packaging ecosystem

pradyunsg.me

Post reply on HN
Print view View original

21–30 of 121 posts

Re: Thoughts on the Python packaging ecosystem

#21

Post by wheelerof4te »

There really needs to be one tool that combines installing, building, testing and running Python projects. This tool should be officially endorsed by core developers of Python and gradually added to the standard distribution model. The good starting point for this would be Pip.

It is de facto standard tool for installing packages. It can be extended to do more.

Also, the standard should favor pure-Python packages, in order to untangle Python from it's legacy as a glue language for modules written in C(++). Otherwise, no progress will be made at the language level and Python will die out once C becomes legacy language, replaced by safer systems programming languages.

We need to learn from Java's and Javascript's ecosystems. Those languages are now used mostly in their pure form. They are portable, optimized and more stable than most languages. With enough care, Python can become the same.

Re: Thoughts on the Python packaging ecosystem

#23

Post by kelsolaar »

jvolkman wrote: post #9
kelsolaar wrote: post #7

Earlier quoted context omitted.

Poetry certainly made our life much easier and was a productivity improvement. It the the tool that brought us back from Conda because it was able to solve environments than nothing in Pip land was able to do.

Was that before or after pip's new resolver was released at the end of 2020?

Before!

Re: Thoughts on the Python packaging ecosystem

#24

Post by wheelerof4te »

chaxor wrote: post #5

Packaging and Nvidia are pretty much the main reasons python has degraded in status for the past 7 years. It started with the added and unnecessary confusion from conda (and silly things like pythonxy), but has really escalated through extra stupidities like poetry. Much of the features that may be enticing for these extra packages should have pushed the developers to improve the standard tools in python (pip). The o…

Python's demise would be it's undeserving fate to be the world's best "glue" language.

The language needs to stand on it's own to be competitive in the long run.

Pure Python modules are much easier to install and keep updated.

Re: Thoughts on the Python packaging ecosystem

#25

Post by mrtweetyhack »

atomicnumber3 wrote: post #10

Why does Ruby not have similar problems? (Or if it does, why does nobody seem to care?) Like why does Ruby have gem and bundler, each doing one thing, whereas python has fifty bazillion tools that all do nearly the same thing if you squint but all have their own weird problems? I've personally just ended up using poetry and that has mostly stopped me from having to care overmuch about the tooling. I feel like I'm spo…

[dead]

Re: Thoughts on the Python packaging ecosystem

#26

Post by divbzero »

woodruffw wrote: post #6

This is a great writeup by a central figure in Python packaging, and gets to the core of one of Python packaging's biggest strengths (and weaknesses): the PyPA is primarily an "open tent," with mostly independent (but somewhat standards-driven) development within it. Pradyun's point about unnecessary competition rings especially true to me, and points to (IMO) a hard reality about where the ecosystem needs to go: at…

To me the natural choice would be pip which, as OP points out, has the advantage of shipping with Python by default. We could remedy the shortcomings of pip by incorporating the most successful features from the N alternatives that exist today.

The npm/yarn fork that formed in 2016 is the closest analogy I can think of. My impression is that npm improved on its biggest weaknesses (e.g. a lock file) and has remained the default choice in that ecosystem. I suspect the same would happen with pip if it too made significant improvements.

Re: Thoughts on the Python packaging ecosystem

#27

Post by KyeRussell »

jessaustin wrote: post #17
philsnow wrote: post #16

> A class of users expect a packaging tool that provides a cohesive experience (like npm (NodeJS), [...], etc) – a single tool that provides a build system, dependency manager, publishing, running project-specific tasks/scripts, etc. I’ve referred to this as “workflow tool” in this post. I don't know enough about node, but aren't there at least two or three package managers (npm, yarn, maybe pnpm)? Then there are hal…

...aren't there at least two or three package managers (npm, yarn, maybe pnpm)? I'm not sure about yarn, but pnpm has exactly the same API as npm has. It simply has a different disk organization and caching strategy. If npm maintainers so chose, pnpm's behavior would be an option within npm. Since they haven't chosen that, it seems completely reasonable to "compete" in the way that pnpm does.

Yes, but one has to come to that conclusion oneself, which increases friction.

I realise you’re talking about pnpm, and not yarn, but when I think of issues like this, yarn is what comes to mind. A bunch of projects prefer yarn over npm, or at the very least place them side by side, in the project documentation. As someone reasonably familiar with JavaScript, but that only works with it sporadically, I always find myself going back and trying to reverse-engineer the reasoning, and I’m never entirely confident.

So when I circle back to my 6-monthly journey into writing some JavaScript, and see another thing called “pnpm” making its way into package installation docs, I cannot be blamed for sighing and being a little irked. Especially since my daily driver is Python, so I’ve got loads of packaging trauma.

Re: Thoughts on the Python packaging ecosystem

#28

Post by Kwpolska »

wiredfool wrote: post #18
Kwpolska wrote: post #13

Earlier quoted context omitted.

Python packaging has too many people inventing their own tools and their own little fiefdoms instead of picking one good tool and pushing it forward. There’s also the PyPA, which is an organization (or a group of semi-related volunteers, depending which interpretation suits them best) that maintains many of the tools (although none of the most feature-complete, namely poetry and PDM) and that produces standards that…

I’ve just looked through the docs for poetry and I don’t see anything there about compiled extensions.

The compiled extension story is pretty bad in most tools. Poetry can do it with a custom build.py file, which calls distutils/setuptools: https://stackoverflow.com/questions/60073711/how-to-build-c-...

Re: Thoughts on the Python packaging ecosystem

#29

Post by wheelerof4te »

revskill wrote: post #22

As i see, most of issues is related to native extensions ? So it belongs to another issue instead of just packaging tool ?

Indeed.

Packaging pure Python often boils down to where to place the packages, the hierarchy between them and where to find the starting point for an application (if it is one and not a library).

Insert native extensions into the mix and you get all kinds of issues.

Like, which library to link to? Which version? Which foreign function library to use? Will my Python version be compatible with that library? Et cetera.

Re: Thoughts on the Python packaging ecosystem

#30

Post by KyeRussell »

ram_rar wrote: post #15

I get a lot of flak for saying this, but I do hope python committers and psf members take a hard look at perl and its demise. Perl had a very similar problem of having N equivalent choices for doing the same thing and python for better or worse is heading in the same direction. In the last few yrs of my professional career many projects I worked, were on are migrating python -> golang cuz frankly many people were fed…

These days I don’t think I’m being too brash in saying that anyone dealing with Python 2 to Python 3 transition stuff is so deep in the realm of enterprise abandonware that it’s not worth paying it much attention. Not necessarily including you here!

But as someone that went through the transition, I’m certainly not going to deny how much of a shit show it was. And I get a feeling that the two situations were/are caused by the same political / organisational / philosophical factors.

I’d be miffed if the Python / PyPA mob got so distracted with their internal politics, certain terse personalities, or the hands-off competition-is-good ideology, that the packaging story makes Python an increasingly unappealing choice. Even worse, we could end up with a packaging ecosystem run by effing Microsoft like the JS people do, because “competition”.

And yes I’m totally conflating aspects of packaging here. At least we all settled on PyPi, except for all those that haven’t… :)

Post reply on HN