Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Python: Please stop screwing over Linux distros

drewdevault.com

Post reply on HN
Print view View original

81–90 of 384 posts

Re: Python: Please stop screwing over Linux distros

#81

Post by uranusjr »

microtonal wrote: post #48
mangecoeur wrote: post #9

Everyone complained, no one addressed the fact that for years the whole of python packaging was handled by like 2.5 people. But as a rant this, like most of the ‘but just fix it’ rants, fails to acknowledge the hugely diverging needs of different users. I could not live without conda, since it’s the only sane way to get a working recent geospatial stack. Others need to run embedded environments, or portable ones, som…

And yet, other languages ship package managers [1] that are widely loved within their ecosystems and cover everything from microcontrollers to server applications. I think what it makes it more difficult in the case of Python is that it has decades of legacy to deal with. No consistent semantic versioning, packages that expect that they can modify their package path in-place (this is a nightmare for immutable systems…

Rust has and “advantage” here that it’s not generally shipped with your distro’s package manager. I think my biggest problem with this article is that the distros put Python there in the first place, and all of them apply patches to make Python work how they want it. And when it doesn’t work, it’s Python’s fault…? I mean yeah Python can definitely do something to make distributing Python easier, but it can only do so much without distros’ direct involvement.

I’ll add that most of Linux distro packaging contributors are generally very nice people, understand the problem at hand, and are very open to collaboration. But sometimes you see this kind of “it’s all your fault” complaints and it’s doing exactly the opposite of helping the cause.

Re: Python: Please stop screwing over Linux distros

#82

Post by m4rtink »

jimnotgym wrote: post #30

It seems a bit weird to criticise lots of people for trying to solve the dependency problem. I know lots of people hold up npm as the standard. Npm came out around 10 years ago, Node around 12. Both of which had the benefit of hindsight at how the problem had developed for other languages. Python came out 30 years ago and pip came out 20 years later. Of course there is a lot of stuff left behind. What do people sugge…

I'm not sure NPM is a good example of anything, with its insane dependency chains and the barrage of security issues.

Re: Python: Please stop screwing over Linux distros

#83

Post by samwillis »

elurg wrote: post #33

Python libraries shipped by distributions are so old that this mechanism is mostly useless for python development. This also applies to many others programming languages which have their own packaging systems. While python packaging is indeed messy the needs of traditional linux distributions are by far the least important. Python packaging needs to better serve the needs of python developers, not those of sysadmins.…

The fundamental problem is the difference between Curated and Uncurated packaging systems:

- Linux dictros have curated packaging, often outdated but stable.

- PyPi, NPM, and others are Uncurated, anyone can publish, could be unstable, could be insecure.

It's down to the developer to decide which route they want to take, and at the moment most want to move quickly with the latest tools. To do that you have to go the uncurated route. It only becomes an issue for a developer if their software is published by a Linux package management system, but 99.9% of developers will never have that.

Re: Python: Please stop screwing over Linux distros

#84

Post by Denvercoder9 »

elurg wrote: post #33

Python libraries shipped by distributions are so old that this mechanism is mostly useless for python development. This also applies to many others programming languages which have their own packaging systems. While python packaging is indeed messy the needs of traditional linux distributions are by far the least important. Python packaging needs to better serve the needs of python developers, not those of sysadmins.…

> All decisions related to dependency choices fundamentally belongs with upstream.

No. As a user I want dependency management (and all of software distribution, to be honest) to be handled by the party that's best able to keep things working while at the same time keeping them secure. Linux distributions have a much, much better track record at that than most upstreams.

Re: Python: Please stop screwing over Linux distros

#85

Post by robertlagrant »

1337shadow wrote: post #8
rbanffy wrote: post #4

Earlier quoted context omitted.

Virtual environments are a good way to have different packages for different applications running on different versions of Python. My usual rule is you don't mess with the system-provided Python environments for specific applications you are working with. I would even suggest dropping support for many packages at the distro level unless they are required by other non-python packages (the same way Django and Twisted a…

> don't mess with the system-provided Python environments for specific applications you are working with Right, but then just use pip install --user instead of a virtualenv. Actually --user is the default now when running pip install from non-root user, so, just pip install as a user will work and not mess with the system packages, just don't do sudo pip install.

> Right, but then just use pip install --user instead of a virtualenv

This seems almost as bad as system python. I suppose it's fine if you only work on one thing, but as soon as you don't, your dev environment will become chaotic and lots of confusing WFM will happen. E.g. this is why npm has a separate node_modules folder for each project.

Re: Python: Please stop screwing over Linux distros

#87

Post by pfranz »

turminal wrote: post #49
elurg wrote: post #33

Python libraries shipped by distributions are so old that this mechanism is mostly useless for python development. This also applies to many others programming languages which have their own packaging systems. While python packaging is indeed messy the needs of traditional linux distributions are by far the least important. Python packaging needs to better serve the needs of python developers, not those of sysadmins.…

Language specific package managers are the antithesis of package management. There is no "management" in "install multiple versions of stuff in this 10 levels deep dependency tree". Upstreams can pick their dependencies but at the same time cannot control what packages depend on them and make it into your app and neither can you. That's a job for your distro.

What happens if your project needs an update, but it will hose the OS? Do you expect someone working in Python in Windows who wants to distribute a datetime package should package for RPM and APT?

I used to think OS package managers should be the end-all be-all, but the use cases for OS package managers are very different from language runtimes. While different Linux distros were fighting between themselves, they completely ignored use cases for projects and language runtimes. Sadly, the result is a mess for everyone.

I think people need to figure out what approach works best for them. I'm of the opinion that any core technology to your business needs to be decoupled from the OS. It makes OS updates too messy and you're tethered to whatever the OS supports. My field uses a lot of Python and every company figured out quickly they need to run their own binaries in addition to packages.

At one company, they packaged up custom RPMs. It wouldn't be a problem to package up and distribute Python libraries. Others had their own package system (no OS or runtime fit their needs). It seems like most people use something like virtualenv.

Regrettably, this means there's no easy answer for people new to Python and the right choice will probably change as you grow. But I really think the answer is Python should come up with something that works for Python and let OSes do their thing.

Re: Python: Please stop screwing over Linux distros

#88

Post by rizkeyz »

captainmuon wrote: post #74

Distros, please stop screwing over Python packaging. It is incredible that Debian/Ubuntu pick Python apart and put modules into different packages. They even create a fake venv command that tells you to please install python-venv. What they should just do is offer a bunch of packages like python3.7, python3.8 that install the official python package wholesale into /usr/python or someplace and then symlink one of them…

This is the real legacy problem: Python comes from a world, where only one version of one packaged seemed the right way to do.

I do not have a good idea, but other ecosystems evolved much more sane in the realm of packaging. While not ideal, Go has done a fairly good job - and the "module" operations are instant - which they should be.

Re: Python: Please stop screwing over Linux distros

#89

Post by wildbook »

__float wrote: post #63
microtonal wrote: post #48

Earlier quoted context omitted.

And yet, other languages ship package managers [1] that are widely loved within their ecosystems and cover everything from microcontrollers to server applications. I think what it makes it more difficult in the case of Python is that it has decades of legacy to deal with. No consistent semantic versioning, packages that expect that they can modify their package path in-place (this is a nightmare for immutable systems…

Is it fair to compare an interpreted language and its package manager to Rust and Cargo? Python packages ship their source (in most cases), depend on a locally installed interpreter (with semantics possibly changing by version). Yes, Python packages often make poor assumptions about what setup.py can do (i.e., _anything_), and so you end up choosing between "tested, supported by the author, and old" or "untested, uns…

Rust almost does the exact same thing, so I'd say it's fair. Dependencies (crates) are grabbed in source format and compiled locally as part of the build process, and installing Rust programs through Cargo also compiles them (and their dependencies) locally.

Some crates have the same issue where build scripts rely on outside tooling being installed, but it's definitely not common to (unless you're relying on compiling C/C++ code for FFI for example, in which case it's somewhat frequent).

Re: Python: Please stop screwing over Linux distros

#90

Post by smallerfish »

keymone wrote: post #80
smallerfish wrote: post #66

Yes, python packaging is a mess. And agreed, there are two separate use cases: development and using the software. But, are distros creating too much work for themselves by trying to package every itty-bitty python library (and for that matter, every npm library)? Are distros doing anything more than scanning CVE databases with the library versions, or are they _actually_ auditing the versions they choose? (Not that…

there's https://github.com/spotify/dh-virtualenv

That's cool. Anybody using this? How robust is it?
Post reply on HN