Distros, please stop screwing over Python packaging. It is incredible that Debian/Ubuntu pick Python apart and put modules into different packages. They even create a fake venv command that tells you to please install python-venv. What they should just do is offer a bunch of packages like python3.7, python3.8 that install the official python package wholesale into /usr/python or someplace and then symlink one of them…
Python: Please stop screwing over Linux distros
101–110 of 384 posts
Re: Python: Please stop screwing over Linux distros
#102> pin their dependencies to 10 versions and 6 vulnerabilities ago That is the real problem IMHO. Most python users like to pin dependencies so that "their program don't break", and that's also the reason why so much effort is put in what they call "correct" dependency resolution resulting in the creation of new python package managers that all do "more correct" dependency resolution and make programs "break less" and…
> I always avoid pinning dependencies This is an very quick route to a maintenance nightmare imo. If you have totally unpinned dependencies, and you come back to a project after a year untouched, or 5 years, and it no longer works - which dependency update broke it? I don't agree that using an outdated package is necessarily a problem at all. Some versions are done! You don't need the latest version of every possible…
In this case, it doesn't matter to me which dependency update broke it, what matters to me is to have the tests passing again with all dependencies.
> I don't agree that using an outdated package is necessarily a problem at all. Some versions are done!
If a version is done then why is there a new release? Using old versions is tech debt that will one day blow up and cost much more to correct than if it had been corrected over the time, not to mention the security risk.
> You don't necessarily need to update _ever_
Another dependency might decide to use that dependency in a newer version, in which case aren't we all better off using the latest versions of everything? The cost is some effort, the benefit is more features, security, performance, less bugs, basically a better program.
> There will also be vulnerabilities in new packages though, which may be unknown, and will often not exist in older much simpler versions.
Then why upgrade at all when you have a dependency with a security issue? After all, in your upgrade you might be adding even more unknown security issues that might be even more dangerous.
> Using a well-tested version of a dependency that does exactly what you need is not less secure than chasing the latest version at all times without a specific reason.
If I can test well that a newer version of a dependency works for me, why not upgrade it? There might be performance, security or other bugfixes, and I'm allowing other maintainers of other dependencies to also use that newer version.
> I rather see more software go in this direction, valuing reproducibility & known correctness (i.e. with isolated pinned dependencies, in some form) over 'always be latest' dependency updates and the complex & hard to reproduce bugs that those shifting dependency interactions can create.
Ok, but then again, some version might fix a security bug that has not been backported to your old version, especially if it's 5 years old.
Basically you're advocating against continuous integration (I'm talking about the "practice", not talking about "the tool that runs automated tests that people call CI").
Re: Python: Please stop screwing over Linux distros
#103Yes, python packaging is a mess. And agreed, there are two separate use cases: development and using the software. But, are distros creating too much work for themselves by trying to package every itty-bitty python library (and for that matter, every npm library)? Are distros doing anything more than scanning CVE databases with the library versions, or are they _actually_ auditing the versions they choose? (Not that…
Re: Python: Please stop screwing over Linux distros
#104I am not one to install python packages using my distro's package manager, but I totally agree with the sentiment that we need a more standard build/dependency management system in python. I like poetry, and I think most people are heading that way, but it doesn't seem to play super nice with pyenv (which is a critical tool) a lot of the time, and I think that a first party endorsement of the "one true build system"…
I landed with poetry as well. The issue with python dependency managment in my book is, that it is incredibly (and needlessly) hard to learn how to do it properly. It took me years to figure out how things should work and there are still issues now and then that cost me a few hours to figure out and restore things. Meanwhile when I use Rust all these things are taken care of in cargo. It is part of the language. Ther…
Re: Python: Please stop screwing over Linux distros
#105Python libraries shipped by distributions are so old that this mechanism is mostly useless for python development. This also applies to many others programming languages which have their own packaging systems. While python packaging is indeed messy the needs of traditional linux distributions are by far the least important. Python packaging needs to better serve the needs of python developers, not those of sysadmins.…
Language specific package managers are the antithesis of package management. There is no "management" in "install multiple versions of stuff in this 10 levels deep dependency tree". Upstreams can pick their dependencies but at the same time cannot control what packages depend on them and make it into your app and neither can you. That's a job for your distro.
Cadences:
Distribution - e.g. Ubuntu 21.10, 22.04; RHEL 8.4, 8.5, ..., 9
Library - e.g. simplejson 3.17.4, 3.17.5, ..., 3.18.0
Language - e.g. Python 3.8.0, 3.9.0, 3.10.0 (This makes Python particularly annoying because for a C project it shouldn't matter if you used gcc 8 or gcc 9 when producing binaries, but with Python it very much matters which version of Python you run with)
Application - e.g. Gimp 2.8.22, ..., 2.9.0
Roles:
As a sysadmin you want to use the distro package manager to install tools for administering your system so that you can connect to wifi, monitor resources, etc.
As an application developer you want to talk to the language specific package manager so that you can use the latest versions of libraries so that bugs are fixed, and you're not shackled to people using RHEL 6 or 7.
As a Debian/Fedora packager you want to talk to the distro because that is required if you want to submit an application upstream to support users who want everything in the distro package manager.
As an application packager you may want to talk to the distro because of the above, but you could also target flatpak (or snap) so that you can use all the latest libraries without worrying about packaging for slow moving distributions.
As an end user you want to use the distro package manager because that's your embedded mental model and workflow. But you should definitely consider using flatpak (or snap if that floats your boat) so that you can use the release stream wherein the libraries are unpinned from the underlying distribution's cadence.
As an end user you should never need to deal with the language specific package manager to install applications or libraries. If you need to install using the language package manager (npm, gems, rocks, cargo, pip) then the application has not really been 'distributed' or 'packaged' imo. If you're doing this, you're off-piste trying out new stuff.
There are more details and a great conversation that can flow from this, but language specific package managers are not the antithesis of package management and supplement it; just not for the 'end user' role. Well, yes for the 'end user' role, just not directly.
Re: Python: Please stop screwing over Linux distros
#106I am not one to install python packages using my distro's package manager, but I totally agree with the sentiment that we need a more standard build/dependency management system in python. I like poetry, and I think most people are heading that way, but it doesn't seem to play super nice with pyenv (which is a critical tool) a lot of the time, and I think that a first party endorsement of the "one true build system"…
I’m not a Python dev. I do need to occasionally run things written in Python. I made the mistake of trying to get pip + Conda + pyenv (or whatever) to install a fairly simple tool. I have no idea how the dev got their setup working, but it was totally and utterly unreproducable, even after they sat down on my computer for several hours. In that amount of time, I could have probably rewritten it in PHP (that’s actuall…
Re: Python: Please stop screwing over Linux distros
#107This is my decision flow and I rarely have an issue: Are you the end user of the Python code? Yes -> Is available in your distro? Yes->Use package manager No->Use pip install in user mode No -> Create virtualenv with the Python version you want (including pypy!) and do your pip thing there Some extreme use cases may benefit from anaconda, but personally I've never needed to use it. My only pain point is dealing with…
> Use pip install in user mode This is a great recipe for disaster. Whatever you install in user mode will shadow anything installed system-wide, so when you try to run some system-wide project, it may now fail. I'm also not a fan of how it drops scripts into `./.local/bin`, since that's where I keep my own script, and is version controlled. The installation will also be frozen and never get updated -- unless you rem…
I used to just use pip to install to the system. Months/years later I would try to untangle the mess of packages I was just playing with, what the OS wanted/needed, I got those conflicting dependencies you mention, etc. I usually ended up reinstalling the OS. At the time I may not have been as knowledgeable about where the OS package manager keeps packages vs pip--but the whole thing wasn't very user-friendly either.
For years I've been installing into user knowing I can just blow it away. I've dabbled with virtualenv, but it's such a pain to set up and activate. If I have a few projects with similar libraries it's more of a pain to set them all up and switch around. If I end up using a script for something important, I just spend the extra time at that point to "package" it.
Re: Python: Please stop screwing over Linux distros
#108Python libraries shipped by distributions are so old that this mechanism is mostly useless for python development. This also applies to many others programming languages which have their own packaging systems. While python packaging is indeed messy the needs of traditional linux distributions are by far the least important. Python packaging needs to better serve the needs of python developers, not those of sysadmins.…
Perhaps the distributions would be more inclined to include up-to-date version if the standard in the python community was not to break everything all the time. There are distributions that keep up-to-date, though, e.g., archlinux. Serving the needs of python developers vs. sysadmins is a false dichotomy. Python developers develop on a system that they need to admin. One great thing about linux is that everything on…
Tell me how I'm supposed to define my lib python deps ?
How am I suppose to package the lib ? Distribute it ? Deal with os versions differences? Allow isolation for my user projects ?
Re: Python: Please stop screwing over Linux distros
#109Earlier quoted context omitted.
Kinda diffrent Rust its like 3 years old when it start getting popular. Python is 13+ years old when it starts getting very popular. You cannot compare legacy to modern solution -.-
Maven was launched 9 years after Java got popular. Everybody was using Ant, everybody decided Ant sucked and moved to the new solution. While Gradle is the new kid on the block, it keeps the infrastructure of dependency management and is thus compatible with Maven to that extent. It can be done.
Re: Python: Please stop screwing over Linux distros
#110Yes, python packaging is a mess. And agreed, there are two separate use cases: development and using the software. But, are distros creating too much work for themselves by trying to package every itty-bitty python library (and for that matter, every npm library)? Are distros doing anything more than scanning CVE databases with the library versions, or are they _actually_ auditing the versions they choose? (Not that…
Nuitka can also apparently compile an entire dependency tree into one binary.