Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Python: Please stop screwing over Linux distros

drewdevault.com

Post reply on HN
Print view View original

171–180 of 384 posts

Re: Python: Please stop screwing over Linux distros

#171

Post by rcarmo »

I have a lot of (possibly controversial) issues about this post, for various reasons:

- Debian/Ubuntu Python packaging is actually pretty comprehensive, and Ubuntu LTS ships with reasonably updated (although admittedly not latest revision) third-party packages that are usable out of the box, and that I could just list in Ansible to have usable environments spun up.

- Python packaging is hardly a mess. I have been using pip for ages without any issues other than forcing wheel downloads for unusual distros (like Alpine, where musl makes it chancy to use some low-level stuff). But if you're on a mainstream distro with modern pip, wheels just work.

- If you're not on Linux (or not on the mainstream), pyenv also just works. I have been using it across several years of macOS releases without any significant issues other than knowing to pass it the required build flags to build out the Cocoa bindings here and then (which is easy to do with the brew pyenv).

And, finally, I'm constantly shocked at the number of people who just don't get virtualenvs, or who don't know how to switch python interpreters by using environment variables.

I've never looked back, and they were instrumental in bridging the gap between 2.7 and 3.5 while I converted some code across (I never really found that the switch to Python 3 was as dramatic as many people made it, perhaps because the code I handled worked after a single pass with 2to3 and minor tweaks).

Re: Python: Please stop screwing over Linux distros

#172

Post by Joker_vD »

1337shadow wrote: post #140
__float wrote: post #132

Earlier quoted context omitted.

Docker is a very heavy-handed approach to mitigate more fundamental issues with Python's package management, don't you think?

I'd say it's a heavy-handed approach to mitigate more fundamental issues with how python packages are maintained, if everybody wants to pin different versions then we're going to have to install different versions of everything which is what npm does and I consider that heavier. Again, it's all a question of point of view, what we see as a package manager problem and causes us to keep reinventing packages managers, m…

I think it's a fundamental problem with managing dependencies. On one hand, any given application usually knows what version of dependencies it actually supports so it makes sense for the application to simply bundle those in: it most extreme cases it's statical linking/binary embedding, or (usually) putting the dependencies in subdirectories of the application's directory ― in cases where the application has a "directory where it lives in" instead of it being thinly spread all over the system (e.g. over /bin, /etc, /usr/bin/, /usr/lib/, etc.).

On the other hand, the users/sysadmins sometimes want to force the application to use different version of dependency, so the application may provide for that by somehow referencing the dependency from the ambient environment: usually it's done by either looking for a dependency in a well-known/hard-coded path, or getting that path from a well-known/hard-coded env var, or from a config file (which also you have to get from somewhere) or from some other injection/locator mechanism, thousands of those.

And all this stuff is bloody fractal: we have system-level packaging, then Python's own packaging on top of that, and then some particular Python application may decide to have its plugin-distribution system of sort (I've seen that), and that too goes on top of all of that, not to mention all the stuff happening in parallel (Qt sublibraries, GNOME modules, npm ecosystem)... well, you get the picture. It kinda reminds me of "Turing tarpit" and I doubt collapsing all this into one single layer of system-level packaging, and nothing on top, is really practical or even possible.

Re: Python: Please stop screwing over Linux distros

#173

Post by selfhoster11 »

pimterry wrote: post #95
captainmuon wrote: post #74

Distros, please stop screwing over Python packaging. It is incredible that Debian/Ubuntu pick Python apart and put modules into different packages. They even create a fake venv command that tells you to please install python-venv. What they should just do is offer a bunch of packages like python3.7, python3.8 that install the official python package wholesale into /usr/python or someplace and then symlink one of them…

> Get rid of the requirement that there is only one stable (minor) version of a package in the distribution at one time. I think this requirement made sense when disk space was scarce. I think this requirement makes sense if you trust that your distro is always better at choosing the 'best' version of a dependency that some software should use than the software author. Nowadays, I think neither is generally true. Dis…

> Disk is plentiful,

What makes you think so? SSDs aren't exactly stellar in the cost-per-TB department, as will be the case with each new higher-performance storage technology. Plenty of people cannot afford the prices of new Western tech either, what about them?

Re: Python: Please stop screwing over Linux distros

#175

Post by avidphantasm »

dgan wrote: post #44
elurg wrote: post #33

Python libraries shipped by distributions are so old that this mechanism is mostly useless for python development. This also applies to many others programming languages which have their own packaging systems. While python packaging is indeed messy the needs of traditional linux distributions are by far the least important. Python packaging needs to better serve the needs of python developers, not those of sysadmins.…

Exactly agreed on that. I don't understand what value distribution is providing by repackaging python libs, they re always way too old to be usable, and they re global while I work on many projects, with their own incompatible requirements. Maybe I am dumb, but I exclusively use virtualenv and pip..

I agree with this for pure Python libraries. However, as soon as you get into things that bind to C libraries (Numpy, GDAL, etc.) it quickly becomes much easier to use the package from your distro.

Re: Python: Please stop screwing over Linux distros

#176

Post by znpy »

toyg wrote: post #134
ajsnigrutin wrote: post #120

Earlier quoted context omitted.

> Perhaps the distributions would be more inclined to include up-to-date version if the standard in the python community was not to break everything all the time. Yep... Perl is a godsend... take a code from 20 years ago, run it on a modern system, and everything works. Python? Three different software versions need three different versions of the same library, and new library versions are not backwards compatible wi…

> Perl is a godsend... take a code from 20 years ago, run it on a modern system, and everything works. That's more or less like "take a VB6 binary from 20 years ago, run it on some modern Windows, and everything works" - that's just because the ecosystem is effectively dead, so supporting it on new releases just means carrying over some stuff that worked 20 years ago.

What's the sin in supporting stuff that works?

Re: Python: Please stop screwing over Linux distros

#177

Post by popcube »

toyg wrote: post #134
ajsnigrutin wrote: post #120

Earlier quoted context omitted.

> Perhaps the distributions would be more inclined to include up-to-date version if the standard in the python community was not to break everything all the time. Yep... Perl is a godsend... take a code from 20 years ago, run it on a modern system, and everything works. Python? Three different software versions need three different versions of the same library, and new library versions are not backwards compatible wi…

> Perl is a godsend... take a code from 20 years ago, run it on a modern system, and everything works. That's more or less like "take a VB6 binary from 20 years ago, run it on some modern Windows, and everything works" - that's just because the ecosystem is effectively dead, so supporting it on new releases just means carrying over some stuff that worked 20 years ago.

they still publish new main version of perl every year, and still support code from perl4.

Re: Python: Please stop screwing over Linux distros

#178

Post by agumonkey »

cjfd wrote: post #97
elurg wrote: post #33

Python libraries shipped by distributions are so old that this mechanism is mostly useless for python development. This also applies to many others programming languages which have their own packaging systems. While python packaging is indeed messy the needs of traditional linux distributions are by far the least important. Python packaging needs to better serve the needs of python developers, not those of sysadmins.…

Perhaps the distributions would be more inclined to include up-to-date version if the standard in the python community was not to break everything all the time. There are distributions that keep up-to-date, though, e.g., archlinux. Serving the needs of python developers vs. sysadmins is a false dichotomy. Python developers develop on a system that they need to admin. One great thing about linux is that everything on…

Instead of assigning blame we should interpret the tension as a sign of an important need having no good solution. It's quite challenging to overlay packaging/versionning systems .. but theoretically it's an interesting and general question. Maybe people should model and solve the whole issue (if there's some solution at all; which I hope)

Re: Python: Please stop screwing over Linux distros

#179

Post by omnicognate »

So here's a suggestion to shoot down.

The biggest insight in software dependency management is that applications and libraries are different. Both have dependencies but they sit at different places in their dependency graphs.

A library can be used together with other libraries and so cannot pin its own dependency versions (if all libraries did so there would conflicts everywhere) but instead can only specify constraints on its dependency versions (eg >=1.0.0).

An application sits at the front of the dependency graph. Nothing depends on it. It can therefore lord it over its dependencies, pinning everything in the graph to specific versions. This only works, though, if it doesn't have to share an environment with other applications and unrelated libraries.

Systems like poetry (akin to npm or cargo) allow a "lock file" to be generated with pinned versions for all dependencies, satisfying all the version constraints. Applications must commit this to revision control, libraries can if they want (should IMO). This is great as it allows CI and other devs to use consistent versions.

The missing piece is that for an application, the lock file should also be used for deployment of the app. If you can ensure that an application is installed in an isolated environment with the dependency versions from the lock file (i.e. that the app was tested against) then a lot of the pain disappears.

So, the suggestion:

* Add support to standard python distribution (wheels, pypi, etc) for application packages to specify (in addition to ordinary version constraints) a pinned set of "preferred" dependency versions.

* Have tools like poetry/pipenv set these to the lock file versions.

* Allow a notion of "application packages", which are required to have this information in.

This should work very nicely with tools like pipx that deal specifically with python applications. The relevance to linux distros is that linux distros also should only be packaging applications (and their dependencies). Developers using libraries should be managing them with tools like poetry/pipenv, not the system package manager.

If the system package manager could install python applications in their own isolated environments along with their pinned dependency versions, most of the pain goes away for distro maintainers. If an application isn't working with the dependency versions it has specifically asked for, it's a clearcut problem with the application as published, and needs to be fixed upstream.

I realise this would be a significant change for package managers, but I think the same model makes sense for other languages with similar tooling and at least some of the work should only need to be done once.

Go on then, tell me why I'm wrong :-D

Post reply on HN