Live data from Hacker News

Thoughts on the Python packaging ecosystem

pradyunsg.me

61–70 of 121 posts

Re: Thoughts on the Python packaging ecosystem

#61

Asking as a complete newcomer to Python programming... Why doesn't somebody just fix pip? Or is pipenv just understood to be the fix?

There’s nothing really wrong with pip.

It doesn’t have a lot of features, but its position isn’t one to have those features.

It’s simple ina good way. I don’t need to futz with a bunch of stuff I don’t need to get it going

Re: Thoughts on the Python packaging ecosystem

#62

I don't know, I'm pretty happy with Pipenv. It's the standard among the people I talk to and very good at being "pip, but better". I've tried poetry but don't really find it good enough to justify being so different, maybe unless you already have a pyproject.toml anyway. There's a lot of talk about Python packaging but as a semi-casual user I've actually had very few problems with it.

Pipenv left a bitter taste in many mouths, but honestly for the last two years it's been fine.

What I want is for the Pipfile to get merged into the pyproject.toml at this point, so that we can speed up the move to pyproject.toml for everything.

Re: Thoughts on the Python packaging ecosystem

#63

This is a great writeup by a central figure in Python packaging, and gets to the core of one of Python packaging's biggest strengths (and weaknesses): the PyPA is primarily an "open tent," with mostly independent (but somewhat standards-driven) development within it. Pradyun's point about unnecessary competition rings especially true to me, and points to (IMO) a hard reality about where the ecosystem needs to go: at…

To me the natural choice would be pip which, as OP points out, has the advantage of shipping with Python by default. We could remedy the shortcomings of pip by incorporating the most successful features from the N alternatives that exist today. The npm / yarn fork that formed in 2016 is the closest analogy I can think of. My impression is that npm improved on its biggest weaknesses ( e.g. a lock file) and has remaine…

Even if pip was a little better I still wouldn’t use it, because Poetry solves the problem so well you’d need to pull it out of my cold hands. It solves not only the install package problem, but locking, venvs, project structure and packaging in a well integrated solution with a fantastic UX.

If poetry (or similar) were renamed to Pip and it were to be included in the stdlib id obviously switch.

It doesn’t solve the single static binary problem or docs problems, but it’s much much closer.

Re: Thoughts on the Python packaging ecosystem

#64
post #32

Earlier quoted context omitted.

Shipping with python is a disadvantage; any real workflow tool needs to be able to manage multiple python installations, and if you try to use a part of the python installation to do that you've got a circular dependency problem. Plus the whole "where modules go to die" thing.

Does anything speak against flipping the relationship? i.e. using pip (or something else) as the standard way to install Python.

pip is a python program.

Re: Thoughts on the Python packaging ecosystem

#66
post #32

Earlier quoted context omitted.

Shipping with python is a disadvantage; any real workflow tool needs to be able to manage multiple python installations, and if you try to use a part of the python installation to do that you've got a circular dependency problem. Plus the whole "where modules go to die" thing.

Does anything speak against flipping the relationship? i.e. using pip (or something else) as the standard way to install Python.

You might want to look at micromamba as a separate tool that can install python itself, libraries, etc. with zero dependency on a python being installed. It's a very fast tool written in C++ so it's easy to install and use almost anywhere. It uses the conda ecosystem of packages.

Re: Thoughts on the Python packaging ecosystem

#67

For me, before I even get to packaging, I hit the install/environment issue with python. Python, by default, wants to be installed at a system level and wants libraries/packages to be at a system level. That shit has to stop. The default needs to be project local installs. Node might have issues but one thing they got right is defaulting to project local installs vs python where you need various incantations to get o…

The Debian dist-packages/site-packages mechanism alleviates a lot of the problems with dependency hell and clobbering from packages needing to be installed. It's a shame it hasn't been embraced by mainline Python. A case of perfect being the enemy of good. Instead we get the chaos of pyproject as the next great thing.

Another lucky winner: https://peps.python.org/pep-0668/

This PEP is meant to ensure you don't clobber your Linux distro's base environment and break key OS applications.

Also, this problem doesn't exist on e.g. Windows, where there is no OS installed python to clobber. Other folks have taught themselves to always use virtual environments for this very reason and therefore don't share your problem.

Hence, there are tutorials out there that don't talk about your problem and tools exist where the default behavior might be dangerous on your Linux distro.

Re: Thoughts on the Python packaging ecosystem

#68
post #5

Packaging and Nvidia are pretty much the main reasons python has degraded in status for the past 7 years. It started with the added and unnecessary confusion from conda (and silly things like pythonxy), but has really escalated through extra stupidities like poetry. Much of the features that may be enticing for these extra packages should have pushed the developers to improve the standard tools in python (pip). The o…

Poetry certainly made our life much easier and was a productivity improvement. It the the tool that brought us back from Conda because it was able to solve environments than nothing in Pip land was able to do.

It's unlikely Python would still be a popular language without Conda. As the article says, many Python users are not traditional software engineers - they need a tool that just works and Conda is the closest thing in the ecosystem.

Used to spend much time to optimize Gentoo Linux but eventually switched to Ubuntu to be productive. I feel Conda is similarly useful, especially if not considering Docker i.e. local dev, and am disappointed hackers seem to be dismissing it.

Re: Thoughts on the Python packaging ecosystem

#69

As i see, most of issues is related to native extensions ? So it belongs to another issue instead of just packaging tool ?

Here's an entire website dedicated to these challenges: https://pypackaging-native.github.io/

Written by one of the maintainers of scipy

Re: Thoughts on the Python packaging ecosystem

#70

There really needs to be one tool that combines installing, building, testing and running Python projects. This tool should be officially endorsed by core developers of Python and gradually added to the standard distribution model. The good starting point for this would be Pip. It is de facto standard tool for installing packages. It can be extended to do more. Also, the standard should favor pure-Python packages, in…

> Also, the standard should favor pure-Python packages, in order to untangle Python from it's legacy as a glue language for modules written in C(++).

Why? Except for the C(++) part, which was never particularly accurate in the first place except that that’s the most popular set of lower-level languages to start with, what's thr problem with Python being a glue language?

> Otherwise, no progress will be made at the language level

Clearly false, as progress continues to be made at the language level.

> Python will die out once C becomes legacy language, replaced by safer systems programming languages.

Why? Python works as glue for Rust as well as anything else. If anything, the biggee threat to Python as a glue language is more dev-friendly system languages, not safer ones, but even there I don't see how packaging deemphasizing support for non-Python modules does anything but accelerate Python problems.

> We need to learn from Java's and Javascript's ecosystems. Those languages are now used mostly in their pure form.

They always were, though; both were designed for use cases without any reliable lower-level besides their own VM, and became popular in that environment. That's not what Python’s ecosystem grew on, and arguably what Java and JS teach here is lean into what made you a success.

Post reply on HN