Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Thoughts on the Python packaging ecosystem

pradyunsg.me

Post reply on HN
Print view View original

41–50 of 121 posts

Re: Thoughts on the Python packaging ecosystem

#41

Post by solarkraft »

jessaustin wrote: post #17
philsnow wrote: post #16

> A class of users expect a packaging tool that provides a cohesive experience (like npm (NodeJS), [...], etc) – a single tool that provides a build system, dependency manager, publishing, running project-specific tasks/scripts, etc. I’ve referred to this as “workflow tool” in this post. I don't know enough about node, but aren't there at least two or three package managers (npm, yarn, maybe pnpm)? Then there are hal…

...aren't there at least two or three package managers (npm, yarn, maybe pnpm)? I'm not sure about yarn, but pnpm has exactly the same API as npm has. It simply has a different disk organization and caching strategy. If npm maintainers so chose, pnpm's behavior would be an option within npm. Since they haven't chosen that, it seems completely reasonable to "compete" in the way that pnpm does.

Yarn used to be compatible with npm package files too. But now that it's moving away from it I probably won't use it anymore; the advantages it brings are not worth the incompatibility.

Re: Thoughts on the Python packaging ecosystem

#42

Post by solarkraft »

KyeRussell wrote: post #27
jessaustin wrote: post #17

Earlier quoted context omitted.

...aren't there at least two or three package managers (npm, yarn, maybe pnpm)? I'm not sure about yarn, but pnpm has exactly the same API as npm has. It simply has a different disk organization and caching strategy. If npm maintainers so chose, pnpm's behavior would be an option within npm. Since they haven't chosen that, it seems completely reasonable to "compete" in the way that pnpm does.

Yes, but one has to come to that conclusion oneself, which increases friction. I realise you’re talking about pnpm, and not yarn, but when I think of issues like this, yarn is what comes to mind. A bunch of projects prefer yarn over npm, or at the very least place them side by side, in the project documentation. As someone reasonably familiar with JavaScript, but that only works with it sporadically, I always find my…

Yarn has a sightly nicer CLI than npm and (I think) innovated a bit more quickly. pnpm mitigates the node_modules problem while otherwise just being npm.

You can usually substitute the package manager in the docs with your favorite, but in doubt just use the one they use.

Re: Thoughts on the Python packaging ecosystem

#43

Post by cozzyd »

wheelerof4te wrote: post #21

There really needs to be one tool that combines installing, building, testing and running Python projects. This tool should be officially endorsed by core developers of Python and gradually added to the standard distribution model. The good starting point for this would be Pip. It is de facto standard tool for installing packages. It can be extended to do more. Also, the standard should favor pure-Python packages, in…

Python is barely newer than C++ and I predict it will die long before C and C++...

Re: Thoughts on the Python packaging ecosystem

#44

Post by LtWorf »

lmm wrote: post #32
divbzero wrote: post #26

Earlier quoted context omitted.

To me the natural choice would be pip which, as OP points out, has the advantage of shipping with Python by default. We could remedy the shortcomings of pip by incorporating the most successful features from the N alternatives that exist today. The npm / yarn fork that formed in 2016 is the closest analogy I can think of. My impression is that npm improved on its biggest weaknesses ( e.g. a lock file) and has remaine…

Shipping with python is a disadvantage; any real workflow tool needs to be able to manage multiple python installations, and if you try to use a part of the python installation to do that you've got a circular dependency problem. Plus the whole "where modules go to die" thing.

If you have multiple versions of python installed you can just do

python3.9 -m pip

python3.10 -m pip

The standard library is the reason why python is popular to begin with.

Re: Thoughts on the Python packaging ecosystem

#45

Post by aragilar »

woodruffw wrote: post #11
atomicnumber3 wrote: post #10

Why does Ruby not have similar problems? (Or if it does, why does nobody seem to care?) Like why does Ruby have gem and bundler, each doing one thing, whereas python has fifty bazillion tools that all do nearly the same thing if you squint but all have their own weird problems? I've personally just ended up using poetry and that has mostly stopped me from having to care overmuch about the tooling. I feel like I'm spo…

In some ways, it's actually pretty weird that Ruby hasn't had similar problems (I say wearing both my Ruby fanboy and Python packaging contributor hats). In others, it's less weird: Ruby's upswing happened with Rails and a handful of other "killer" frameworks, which helped to solidify developer workflows (and expectations) around packaging. Python, by contrast, has had multiple generations of "killer" usecases, each…

I think the main reason Ruby has avoided this (like most other newer languages) is the lack of a numeric/scientific ecosystem, which is where the real mess begins. Science code never dies (because the rewrite almost never implements everything the predecessor had, so now you need both the original tool and its "replacement"), so I have tools built from Fortran, C, C++, Tcl, Java, Python and now Rust to combine.

Re: Thoughts on the Python packaging ecosystem

#47

Post by westurner »

woodruffw wrote: post #6

This is a great writeup by a central figure in Python packaging, and gets to the core of one of Python packaging's biggest strengths (and weaknesses): the PyPA is primarily an "open tent," with mostly independent (but somewhat standards-driven) development within it. Pradyun's point about unnecessary competition rings especially true to me, and points to (IMO) a hard reality about where the ecosystem needs to go: at…

The primary (probably build and) packaging system for a software application should probably support the maximum level of metadata sufficient for downstream repackaging tools.

Metadata for the ultimate software package should probably include a sufficient number of attributes in its declarative manifest:

Package namespace and name,

Per-file paths and Checksums, and at least one cryptographic signature from the original publisher. Whether the server has signed what was uploaded is irrelevant if it and the files within don't match a publisher signature at upload time?

And then there's the permissions metadata, the ACLs and context labels to support any or all of: SELinux, AppArmor, Flatpak, OpenSnitch, etc.. Neither Python packages nor conda packages nor RPM support specifying permissions and capabilities necessary for operation of downstream packages.

You can change the resolver, but the package metadata would need to include sufficient data elements for Python packaging to be the ideal uni-language package manager imho

Re: Thoughts on the Python packaging ecosystem

#48

Post by quietbritishjim »

philsnow wrote: post #16

> A class of users expect a packaging tool that provides a cohesive experience (like npm (NodeJS), [...], etc) – a single tool that provides a build system, dependency manager, publishing, running project-specific tasks/scripts, etc. I’ve referred to this as “workflow tool” in this post. I don't know enough about node, but aren't there at least two or three package managers (npm, yarn, maybe pnpm)? Then there are hal…

I'm not an expert at JS, but can't you use npm to do multiple things if you choose? I.e. build, install, create a place to install some package. In Python, traditionally, this would need separate tools (setuptools, pip, virtualenv). That's what the author meant – not that there are no competitors.

Re: Thoughts on the Python packaging ecosystem

#49

Post by pronlover723 »

For me, before I even get to packaging, I hit the install/environment issue with python. Python, by default, wants to be installed at a system level and wants libraries/packages to be at a system level.

That shit has to stop. The default needs to be project local installs. Node might have issues but one thing they got right is defaulting to project local installs vs python where you need various incantations to get out of the default "globals" and you need other incantations to switch projects.

Re: Thoughts on the Python packaging ecosystem

#50

Post by ivoflipse »

pronlover723 wrote: post #49

For me, before I even get to packaging, I hit the install/environment issue with python. Python, by default, wants to be installed at a system level and wants libraries/packages to be at a system level. That shit has to stop. The default needs to be project local installs. Node might have issues but one thing they got right is defaulting to project local installs vs python where you need various incantations to get o…

Lucky for you, there's a new PEP proposing just that: https://discuss.python.org/t/pep-704-require-virtual-environ... Written by the author of this blog post no less!

However, you'll find that as with all packaging discussions there are people opposing it, because their workflow doesn't match yours and they don't want to change how they work. We, as a community, need a way to resolve such stalemates or I fear we won't make much headway

Post reply on HN