Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Python: Please stop screwing over Linux distros

drewdevault.com

Post reply on HN
Print view View original

61–70 of 384 posts

Re: Python: Please stop screwing over Linux distros

#61

Post by WhyNotHugo »

nerdponx wrote: post #34
fer wrote: post #28

This is my decision flow and I rarely have an issue: Are you the end user of the Python code? Yes -> Is available in your distro? Yes->Use package manager No->Use pip install in user mode No -> Create virtualenv with the Python version you want (including pypy!) and do your pip thing there Some extreme use cases may benefit from anaconda, but personally I've never needed to use it. My only pain point is dealing with…

> Create virtualenv with the Python version you want (including pypy!) and do your pip thing there You might benefit from using Pipx in this case: https://pypa.github.io/pipx/ Pipx is good for the case of "I want to run a standalone Python application that is available through Pip, but not my system's package repo." This is a more common case than you might think. It's a sensible alternative to `pip install --user`,…

Yup, this kinda works.

It doesn't address the greater issue tho: that it's getting harder and harder for distributions to package things right, and provider packages for their users (evidenced by the fact that you need a second package manager just for python stuff).

Re: Python: Please stop screwing over Linux distros

#62

Post by komuher »

microtonal wrote: post #48
mangecoeur wrote: post #9

Everyone complained, no one addressed the fact that for years the whole of python packaging was handled by like 2.5 people. But as a rant this, like most of the ‘but just fix it’ rants, fails to acknowledge the hugely diverging needs of different users. I could not live without conda, since it’s the only sane way to get a working recent geospatial stack. Others need to run embedded environments, or portable ones, som…

And yet, other languages ship package managers [1] that are widely loved within their ecosystems and cover everything from microcontrollers to server applications. I think what it makes it more difficult in the case of Python is that it has decades of legacy to deal with. No consistent semantic versioning, packages that expect that they can modify their package path in-place (this is a nightmare for immutable systems…

Kinda diffrent Rust its like 3 years old when it start getting popular. Python is 13+ years old when it starts getting very popular. You cannot compare legacy to modern solution -.-

Re: Python: Please stop screwing over Linux distros

#63

Post by __float »

microtonal wrote: post #48
mangecoeur wrote: post #9

Everyone complained, no one addressed the fact that for years the whole of python packaging was handled by like 2.5 people. But as a rant this, like most of the ‘but just fix it’ rants, fails to acknowledge the hugely diverging needs of different users. I could not live without conda, since it’s the only sane way to get a working recent geospatial stack. Others need to run embedded environments, or portable ones, som…

And yet, other languages ship package managers [1] that are widely loved within their ecosystems and cover everything from microcontrollers to server applications. I think what it makes it more difficult in the case of Python is that it has decades of legacy to deal with. No consistent semantic versioning, packages that expect that they can modify their package path in-place (this is a nightmare for immutable systems…

Is it fair to compare an interpreted language and its package manager to Rust and Cargo? Python packages ship their source (in most cases), depend on a locally installed interpreter (with semantics possibly changing by version).

Yes, Python packages often make poor assumptions about what setup.py can do (i.e., _anything_), and so you end up choosing between "tested, supported by the author, and old" or "untested, unsupported, but up to date".

Re: Python: Please stop screwing over Linux distros

#64

Post by pimterry »

1337shadow wrote: post #23

> pin their dependencies to 10 versions and 6 vulnerabilities ago That is the real problem IMHO. Most python users like to pin dependencies so that "their program don't break", and that's also the reason why so much effort is put in what they call "correct" dependency resolution resulting in the creation of new python package managers that all do "more correct" dependency resolution and make programs "break less" and…

> I always avoid pinning dependencies

This is an very quick route to a maintenance nightmare imo.

If you have totally unpinned dependencies, and you come back to a project after a year untouched, or 5 years, and it no longer works - which dependency update broke it?

I don't agree that using an outdated package is necessarily a problem at all. Some versions are done! You don't need the latest version of every possible package. You don't necessarily need to update _ever_ (which is why this differs from CI). These updates are often entirely unnecessary churn.

There absolutely are vulnerabilities in some old versions, and those updates are necessary (but tooling & notifications to easily handle this have dramatically improved in recent years, especially on GitHub). There will also be vulnerabilities in new packages though, which may be unknown, and will often not exist in older much simpler versions.

Using a well-tested version of a dependency that does exactly what you need is not less secure than chasing the latest version at all times without a specific reason.

I've found manually updating packages on the rare occasions where relevant vulnerabilities arise, and using existing working versions without changes the rest of the time has been perfectly effective over many years now, and avoid the shifting sands of external dependencies wherever possible means that a project that worked 5 years ago still works _exactly_ the same today.

I rather see more software go in this direction, valuing reproducibility & known correctness (i.e. with isolated pinned dependencies, in some form) over 'always be latest' dependency updates and the complex & hard to reproduce bugs that those shifting dependency interactions can create.

Re: Python: Please stop screwing over Linux distros

#65

Post by turminal »

microtonal wrote: post #48
mangecoeur wrote: post #9

Everyone complained, no one addressed the fact that for years the whole of python packaging was handled by like 2.5 people. But as a rant this, like most of the ‘but just fix it’ rants, fails to acknowledge the hugely diverging needs of different users. I could not live without conda, since it’s the only sane way to get a working recent geospatial stack. Others need to run embedded environments, or portable ones, som…

And yet, other languages ship package managers [1] that are widely loved within their ecosystems and cover everything from microcontrollers to server applications. I think what it makes it more difficult in the case of Python is that it has decades of legacy to deal with. No consistent semantic versioning, packages that expect that they can modify their package path in-place (this is a nightmare for immutable systems…

Cargo is a good example of a survivorship bias of sorts. It's so deeply integrated into rust that if you can't stomach it you just leave the ecosystem and everyone that remains likes it ;)

Re: Python: Please stop screwing over Linux distros

#66

Post by smallerfish »

Yes, python packaging is a mess.

And agreed, there are two separate use cases: development and using the software.

But, are distros creating too much work for themselves by trying to package every itty-bitty python library (and for that matter, every npm library)? Are distros doing anything more than scanning CVE databases with the library versions, or are they _actually_ auditing the versions they choose? (Not that there's much choice, since python also has a shitty story when it comes to backwards compatibility; if you're going with 3.10, there's possibly only one version of a given library that will work.)

Java has a commonly used "fat jar" approach which rolls up all dependencies into a single file. It's excellent. In the python world, this doesn't exist, because virtualenvs aren't portable. If that can be fixed (perhaps a specific section in requirements.txt that captures anything that needs to compile C for the platform) then a distributable virtualenv would become possible. Distros would then scan the application for vulnerabilities (via requirement.txt's manifest), build the distributable-virtualenv, and ship _that_. Python library maintainers don't have to do anything different (except, of course, use the standard way to declare dependencies).

Re: Python: Please stop screwing over Linux distros

#67

Post by globular-toast »

I think there is an increasing trend for developers to get confused between their tools and their output.

These are your tools:

    * pip
    * pip-tools
    * pipenv
    * poetry
    * virtualenv
    * venv
    * setuptools
    * pyenv
    * requirements.txt
    * etc.
These are your outputs:

    * wheel (binary),
    * tarball (source).
The tools in use are completely irrelevant and only add noise to the discussions around packaging. So what actually is the problem here? Are projects delivering broken outputs (ie. bad packages)? Are they not delivering outputs at all? Those are real problems but pointing at the number of tools that exist is not helping.

Re: Python: Please stop screwing over Linux distros

#68

Post by rbanffy »

nerdponx wrote: post #37
formerly_proven wrote: post #20

Earlier quoted context omitted.

Your last sentence doesn't parse for me at all. How exactly do you get a pip installed on Python 3.6 to modify anything about another install of Python, let alone a different version?

If your distro dumps all the installed files into `/usr/lib/python3` without distinguishing among 3.8, 3.9, etc.

> If your distro dumps all the installed files into `/usr/lib/python3`

Oh dear... What kind of monster does this?

Re: Python: Please stop screwing over Linux distros

#69

Post by 1337shadow »

cdent wrote: post #46

I'm always confused by these sorts of posts because they happen often so there is clearly a problem but for some reason I've never had much of an issue. I've been using and developing with Python for about 15 years. In that time I've worked on Python projects large (OpenStack) and small (gabbi) and taken over maintenance for some old standbys (wsgi-intercept, paste to name two). Dealt with the 2->3 transition. Releas…

OpenStack uses pbr which they created and which is pretty cool. I use setupmeta for the same purpose but could have used pbr.

https://github.com/openstack/pbr https://github.com/codrsquad/setupmeta

Re: Python: Please stop screwing over Linux distros

#70

Post by rbanffy »

salawat wrote: post #58
formerly_proven wrote: post #20

Earlier quoted context omitted.

Your last sentence doesn't parse for me at all. How exactly do you get a pip installed on Python 3.6 to modify anything about another install of Python, let alone a different version?

Apt install python-pip3 You'll get pip3 from Python 3.6. apt install python3.8 You'll get system-level python3. Use pip3 in 3.6 to install the latest pip3 that requires Python 3.8, and drops that package in the dist-packages folder, which is shared between system Python runtime versions. Behold as further attempts to do something sane with APT blow up, because now you have 3.8 packages on 3.6's runtime path. I've los…

"Thou shalt not mess with the system Python"
Post reply on HN