Thoughts on the Python packaging ecosystem • Hacker News Board

Re: Thoughts on the Python packaging ecosystem

#11

Post by woodruffw » Sat, Jan 21, 2023, 10:07 PM UTC

atomicnumber3 wrote: post #10

Why does Ruby not have similar problems? (Or if it does, why does nobody seem to care?) Like why does Ruby have gem and bundler, each doing one thing, whereas python has fifty bazillion tools that all do nearly the same thing if you squint but all have their own weird problems? I've personally just ended up using poetry and that has mostly stopped me from having to care overmuch about the tooling. I feel like I'm spo…

In some ways, it's actually pretty weird that Ruby hasn't had similar problems (I say wearing both my Ruby fanboy and Python packaging contributor hats).

In others, it's less weird: Ruby's upswing happened with Rails and a handful of other "killer" frameworks, which helped to solidify developer workflows (and expectations) around packaging. Python, by contrast, has had multiple generations of "killer" usecases, each with their own baggage (and many predating any real packaging standards).

The history of Python packaging is also much older, and much more devolved than anybody in 2023 would consider reasonable for a packaging ecosystem: the earliest generation of PyPI, for example, was just an index that pointed to other webhosts for downloads, rather than a full package host. This helped ossify manual workflows that developers at the time were content with, and some of that cost is still being paid forwards.

Re: Thoughts on the Python packaging ecosystem

#12

Post by [deleted] » Sat, Jan 21, 2023, 10:07 PM UTC

atomicnumber3 wrote: post #10

Why does Ruby not have similar problems? (Or if it does, why does nobody seem to care?) Like why does Ruby have gem and bundler, each doing one thing, whereas python has fifty bazillion tools that all do nearly the same thing if you squint but all have their own weird problems? I've personally just ended up using poetry and that has mostly stopped me from having to care overmuch about the tooling. I feel like I'm spo…

[deleted]

Re: Thoughts on the Python packaging ecosystem

#13

Post by Kwpolska » Sat, Jan 21, 2023, 10:08 PM UTC

atomicnumber3 wrote: post #10

Why does Ruby not have similar problems? (Or if it does, why does nobody seem to care?) Like why does Ruby have gem and bundler, each doing one thing, whereas python has fifty bazillion tools that all do nearly the same thing if you squint but all have their own weird problems? I've personally just ended up using poetry and that has mostly stopped me from having to care overmuch about the tooling. I feel like I'm spo…

Python packaging has too many people inventing their own tools and their own little fiefdoms instead of picking one good tool and pushing it forward. There’s also the PyPA, which is an organization (or a group of semi-related volunteers, depending which interpretation suits them best) that maintains many of the tools (although none of the most feature-complete, namely poetry and PDM) and that produces standards that promote tooling proliferation.

Re: Thoughts on the Python packaging ecosystem

#14

Post by [deleted] » Sat, Jan 21, 2023, 10:08 PM UTC

chaxor wrote: post #5

Packaging and Nvidia are pretty much the main reasons python has degraded in status for the past 7 years. It started with the added and unnecessary confusion from conda (and silly things like pythonxy), but has really escalated through extra stupidities like poetry. Much of the features that may be enticing for these extra packages should have pushed the developers to improve the standard tools in python (pip). The o…

[deleted]

Re: Thoughts on the Python packaging ecosystem

#15

Post by ram_rar » Sat, Jan 21, 2023, 10:23 PM UTC

I get a lot of flak for saying this, but I do hope python committers and psf members take a hard look at perl and its demise. Perl had a very similar problem of having N equivalent choices for doing the same thing and python for better or worse is heading in the same direction. In the last few yrs of my professional career many projects I worked, were on are migrating python -> golang cuz frankly many people were fed up of python2->3 migration. Not that golang doesnt have it fair share of issues, but the the packaging and deploying aspects of a single binary makes CI/CD and operations a breeze. I hope python makes this is first class citizen in its ecosystem.

Re: Thoughts on the Python packaging ecosystem

#16

Post by philsnow » Sat, Jan 21, 2023, 10:23 PM UTC

> A class of users expect a packaging tool that provides a cohesive experience (like npm (NodeJS), [...], etc) – a single tool that provides a build system, dependency manager, publishing, running project-specific tasks/scripts, etc. I’ve referred to this as “workflow tool” in this post.

I don't know enough about node, but aren't there at least two or three package managers (npm, yarn, maybe pnpm)? Then there are half a dozen different things for the "build" / transpile / compile stage of frontend work, and

> Pick from N ~equivalent choices is a really bad user experience

this has caused me to bounce off of getting into frontend work several times. It's so aesthetically displeasing that my brain doesn't want to learn it.

Re: Thoughts on the Python packaging ecosystem

#17

Post by jessaustin » Sat, Jan 21, 2023, 10:35 PM UTC

philsnow wrote: post #16

> A class of users expect a packaging tool that provides a cohesive experience (like npm (NodeJS), [...], etc) – a single tool that provides a build system, dependency manager, publishing, running project-specific tasks/scripts, etc. I’ve referred to this as “workflow tool” in this post. I don't know enough about node, but aren't there at least two or three package managers (npm, yarn, maybe pnpm)? Then there are hal…

...aren't there at least two or three package managers (npm, yarn, maybe pnpm)?

I'm not sure about yarn, but pnpm has exactly the same API as npm has. It simply has a different disk organization and caching strategy. If npm maintainers so chose, pnpm's behavior would be an option within npm. Since they haven't chosen that, it seems completely reasonable to "compete" in the way that pnpm does.

Re: Thoughts on the Python packaging ecosystem

#18

Post by wiredfool » Sat, Jan 21, 2023, 10:43 PM UTC

Kwpolska wrote: post #13

atomicnumber3 wrote: post #10

Why does Ruby not have similar problems? (Or if it does, why does nobody seem to care?) Like why does Ruby have gem and bundler, each doing one thing, whereas python has fifty bazillion tools that all do nearly the same thing if you squint but all have their own weird problems? I've personally just ended up using poetry and that has mostly stopped me from having to care overmuch about the tooling. I feel like I'm spo…

Python packaging has too many people inventing their own tools and their own little fiefdoms instead of picking one good tool and pushing it forward. There’s also the PyPA, which is an organization (or a group of semi-related volunteers, depending which interpretation suits them best) that maintains many of the tools (although none of the most feature-complete, namely poetry and PDM) and that produces standards that…

I’ve just looked through the docs for poetry and I don’t see anything there about compiled extensions.

Re: Thoughts on the Python packaging ecosystem

#19

Post by totalhack » Sat, Jan 21, 2023, 10:46 PM UTC

kelsolaar wrote: post #7

chaxor wrote: post #5

Packaging and Nvidia are pretty much the main reasons python has degraded in status for the past 7 years. It started with the added and unnecessary confusion from conda (and silly things like pythonxy), but has really escalated through extra stupidities like poetry. Much of the features that may be enticing for these extra packages should have pushed the developers to improve the standard tools in python (pip). The o…

Poetry certainly made our life much easier and was a productivity improvement. It the the tool that brought us back from Conda because it was able to solve environments than nothing in Pip land was able to do.

Poetry seems pretty slow due to some of the extra dependency logic it builds on top of pip. Whether this trade off is worth it probably depends on the scope of the project. For most of my projects I stick with pip and almost never run into issues.

Re: Thoughts on the Python packaging ecosystem

#20

Post by JeremyBanks » Sat, Jan 21, 2023, 10:48 PM UTC

philsnow wrote: post #16

> A class of users expect a packaging tool that provides a cohesive experience (like npm (NodeJS), [...], etc) – a single tool that provides a build system, dependency manager, publishing, running project-specific tasks/scripts, etc. I’ve referred to this as “workflow tool” in this post. I don't know enough about node, but aren't there at least two or three package managers (npm, yarn, maybe pnpm)? Then there are hal…

[dead]