Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Python: Please stop screwing over Linux distros

drewdevault.com

Post reply on HN
Print view View original

121–130 of 384 posts

Re: Python: Please stop screwing over Linux distros

#121

Post by Joker_vD »

tuldia wrote: post #18

Am I the only one that is not having issues with python and distributions in general? I get all my dependencies from Debian and they all work, when I need something that is not yet packaged, I use pip. What are people doing to get all this issues? I don't understand...

Once upon a time, I did "apt upgrade python-pip3" or something like this (was it just "python-pip"? Or maybe it was "apt upgrade"? It was a couple years ago). Anyway, what I do remember is that it quite literally killed apt: invoking it with any command would lead to a dump of a stack trace with ImportError coming from pip. Apparently, apt uses system-wide pip internally so if you touch it, everything breaks? Don't know, don't much care: since it was just a VM so I simply rolled to the previous snapshot and forgot about the details.

Edit: Ah, apparently the steps to reproduce are: do "apt install python-pip3"; do "apt install python3.8"; when pip3 complains that it's outdated, update it with the command it itself suggests.

Re: Python: Please stop screwing over Linux distros

#122

Post by toyg »

The flexibility the author decries is one of the strengths of the ecosystem. Oh, things don't work for datascientists on Windows? Here, use conda. Bunch of different webapps on same server? Use venv. And so on and so forth, every niche will use what works best for them.

Re: Python: Please stop screwing over Linux distros

#123

Post by smallerfish »

RegW wrote: post #109
smallerfish wrote: post #72

Earlier quoted context omitted.

Maven was launched 9 years after Java got popular. Everybody was using Ant, everybody decided Ant sucked and moved to the new solution. While Gradle is the new kid on the block, it keeps the infrastructure of dependency management and is thus compatible with Maven to that extent. It can be done.

Ant didn't do package management.

No, but ivy was commonly used. That said, the bigger point is that the standard way to do things can be changed. For some reason, despite (or because of?) PEP, the python community seems unable to coalesce around standard ways of packaging.

Re: Python: Please stop screwing over Linux distros

#124

Post by __float »

wildbook wrote: post #89
__float wrote: post #63

Earlier quoted context omitted.

Is it fair to compare an interpreted language and its package manager to Rust and Cargo? Python packages ship their source (in most cases), depend on a locally installed interpreter (with semantics possibly changing by version). Yes, Python packages often make poor assumptions about what setup.py can do (i.e., _anything_), and so you end up choosing between "tested, supported by the author, and old" or "untested, uns…

Rust almost does the exact same thing, so I'd say it's fair. Dependencies (crates) are grabbed in source format and compiled locally as part of the build process, and installing Rust programs through Cargo also compiles them (and their dependencies) locally. Some crates have the same issue where build scripts rely on outside tooling being installed, but it's definitely not common to (unless you're relying on compilin…

I think there needs to be a distinction between fetching crates as source at _build time_ and what happens with Python. Python's "build time" will still require the source to be present on the target machine it's deployed to -- unfortunately these tools are complex because "packaging" is only half of the issue, it's also the distribution and _deployment_ which makes things messy.

Consider that, even if you want to use packages only from your application's virtualenv, the default (footgun warning!) is that Python will still use the "system" packages -- this means you may have installed Ansible or some other tool that relies on Python and many packages from the distro package manager. But your app could pick up one of those dependencies! At best, this will work fine. But in the worse cases, perhaps it subtly behaves differently or simply does not work at all.

My understanding is that Rust will, by default, statically link all of these dependencies. This, in Python, would be like a "pex" or "par" (or one of the many other options :^)), which does make the distribution aspect much simpler. (At the cost of build-time complexity, slowdowns, and occasional incompatibility.)

Re: Python: Please stop screwing over Linux distros

#125

Post by ajsnigrutin »

komuher wrote: post #62
microtonal wrote: post #48

Earlier quoted context omitted.

And yet, other languages ship package managers [1] that are widely loved within their ecosystems and cover everything from microcontrollers to server applications. I think what it makes it more difficult in the case of Python is that it has decades of legacy to deal with. No consistent semantic versioning, packages that expect that they can modify their package path in-place (this is a nightmare for immutable systems…

Kinda diffrent Rust its like 3 years old when it start getting popular. Python is 13+ years old when it starts getting very popular. You cannot compare legacy to modern solution -.-

Perl is 33 years old, and perl 5.x is 27 years old.

I can literally take a 20+yo book and all the examples still work. CPAN still works. I literally have 20+yo scripts still copied from server to server, from laptop to laptop, without any changes.

Re: Python: Please stop screwing over Linux distros

#126

Post by 1337shadow »

gjulianm wrote: post #115
1337shadow wrote: post #51

Earlier quoted context omitted.

Interesting problem but I think it's more a problem in virtualenv, which don't use global site package by default, I'm surprised that it uses user site packages by default. However, installing every dependency in the same environment is also what distro package managers do, I don't see that as a recipe for disaster, it all depends how well the said packages are maintained in which case just upgrading them should fix…

> However, installing every dependency in the same environment is also what distro package managers do, I don't see that as a recipe for disaster, Distro packages are done in a way where it's hard to get conflicts. I haven't found an instance where python3-X and python3-Y can't be installed because one requires python3-Z v1.2 and the other python3-Z v2.1. With Python packages, that happens way more often. Also, distr…

> I haven't found an instance where python3-X and python3-Y can't be installed because one requires python3-Z v1.2 and the other python3-Z v2.1. With Python packages, that happens way more often.

Exactly, because the packages were created/updated at a point in time were they were compatible: a distro ships python 3.x with packages of python modules that are compatible with python 3.x.

Now if you as a user want to use a python package that works with python 3.y, you either have to wait, or install python 3.y, or use a container of python 3.y.

Again, if all python package were up to date as in "working together at this point in time" then `pip upgrade` would work.

> Also, distro package managers keep track of everything installed, while python ones (at least pip I'm 100% sure) don't.

Are you sure you checked in the `*.dist-info` directories? There should be one per package, containing: a METADATA file, with all dependency versions, a RECORD file, with every installed file and their hash, and much more! see for yourself ;)

> You could install a package which upgrades another one, and that breaks another that you had installed before and pip won't say a word.

Let's agree to disagree right there:

- in your point of view: this is a problem in pip

- in my point of view: this is a maintenance problem in the python packages themselves, caused by the general practice of pinning

Re: Python: Please stop screwing over Linux distros

#127

Post by toyg »

nerdponx wrote: post #92
captainmuon wrote: post #74

Distros, please stop screwing over Python packaging. It is incredible that Debian/Ubuntu pick Python apart and put modules into different packages. They even create a fake venv command that tells you to please install python-venv. What they should just do is offer a bunch of packages like python3.7, python3.8 that install the official python package wholesale into /usr/python or someplace and then symlink one of them…

Homebrew does this very well with its "cellar" system. Every version of every package gets installed to its own root tree, eg `/usr/local/Cellar/python/3.9.7/`. The currently-active version is then symlinked into `/usr/local/opt/python` and from there into `/usr/local`. I believe Nix and Guix also work this way.

I'll remember that "Homebrew does this very well" the next time I have to fix a bunch of shit because it has updated the currently-active version, or removed this or that bugfix release, as part of a general upgrade. After the third time this happened, I started using pyenv - which is another mountain of brokenness, I grant you, but at least I have some degree of control on what happens when.

Re: Python: Please stop screwing over Linux distros

#128

Post by bregma »

fnord123 wrote: post #105
turminal wrote: post #49

Earlier quoted context omitted.

Language specific package managers are the antithesis of package management. There is no "management" in "install multiple versions of stuff in this 10 levels deep dependency tree". Upstreams can pick their dependencies but at the same time cannot control what packages depend on them and make it into your app and neither can you. That's a job for your distro.

There are different cadences and roles in play here. Cadences: Distribution - e.g. Ubuntu 21.10, 22.04; RHEL 8.4, 8.5, ..., 9 Library - e.g. simplejson 3.17.4, 3.17.5, ..., 3.18.0 Language - e.g. Python 3.8.0, 3.9.0, 3.10.0 (This makes Python particularly annoying because for a C project it shouldn't matter if you used gcc 8 or gcc 9 when producing binaries, but with Python it very much matters which version of Pytho…

This is the most insightful comment.

All the other observations on this topic (and I've read very many over the years) are akin to the six blind men and the elephant.

Re: Python: Please stop screwing over Linux distros

#129

Post by simonh »

cjfd wrote: post #97
elurg wrote: post #33

Python libraries shipped by distributions are so old that this mechanism is mostly useless for python development. This also applies to many others programming languages which have their own packaging systems. While python packaging is indeed messy the needs of traditional linux distributions are by far the least important. Python packaging needs to better serve the needs of python developers, not those of sysadmins.…

Perhaps the distributions would be more inclined to include up-to-date version if the standard in the python community was not to break everything all the time. There are distributions that keep up-to-date, though, e.g., archlinux. Serving the needs of python developers vs. sysadmins is a false dichotomy. Python developers develop on a system that they need to admin. One great thing about linux is that everything on…

>Serving the needs of python developers vs. sysadmins is a false dichotomy. Python developers develop on a system that they need to admin.

I couldn't disagree more. Even if the same person is admining a system they develop on, they almost certainly aren't going to admin the systems their users deploy on. The admin role and developer role should be completely separate with different goals and requirements.

The system Python should only be used for system scripts. The end. Nowadays ideally it shouldn't even be visible to non-admin user accounts, and users certainly shouldn't be installing packages into the python being used by system scripts, not even using virtual envs.

The python your system uses and how it's configured should be decided by the distro. If they want to break it up into weird packages and funky paths, whatever. That's their problem because they and maybe (_maybe_) system admins should be the only people using it.

As a developer you should be making your own decision about which version of Python you are using, what modules get installed into it, what venvs you have and how you manage them. This should all be considered with one eye firmly fixed on the target deployment environment and how your Python will be configured there. The right answer, of course, being that you should be packaging the required python along with the application.

Re: Python: Please stop screwing over Linux distros

#130

Post by toyg »

simonh wrote: post #114
BiteCode_dev wrote: post #101

Earlier quoted context omitted.

Yes, explaining to a beginer how to bootstrap python is hell because windows, mac and linux keeps making sure nothing work the same.

Actually, explaining how to get Python working on Windows is far, far easier than on either linux (modulo various distros) or the Mac. That's because there is one obvious distribution of Python to use, the official one, and new versions of it are always consistent and play well together. Yes you can use Anaconda if you want, and people who do that are probably data scientists or something and know what they want to d…

> That's because there is one obvious distribution of Python to use

Lol, no. There is one in the Windows appstore too, which has issues with privileges. And I'm almost sure now MS ships it in some other tool too.

Post reply on HN