Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Python: Please stop screwing over Linux distros

drewdevault.com

Post reply on HN
Print view View original

141–150 of 384 posts

Re: Python: Please stop screwing over Linux distros

#141

Post by Denvercoder9 »

Gigachad wrote: post #119
Denvercoder9 wrote: post #84

Earlier quoted context omitted.

> All decisions related to dependency choices fundamentally belongs with upstream. No. As a user I want dependency management (and all of software distribution, to be honest) to be handled by the party that's best able to keep things working while at the same time keeping them secure. Linux distributions have a much, much better track record at that than most upstreams.

I really doubt that the python libraries packaged by Debian are any more secure or stable than the latest release of those libraries. At best they just limit to breaking updates to once every few years when they update them. It’s essentially like version locking packages except some random Debian maintainer decides when it’s time to update.

> I really doubt that the python libraries packaged by Debian are any more secure or stable than the latest release of those libraries.

They are more stable because I can keep using the same version for two years, and I'm not being pushed to the latest version that has (intentional or unintentional) breaking changes every two months. Yes, there might be a bug or two in there that have since been fixed, but I very much prefer the failure I know over unexpected failures.

They are secure because Debian (and distros like it) backport security fixes to their packages. You can argue about whether they do a good enough job keeping up with vulnerabilities, but at least I know that once I install the update from Debian, my machine is secure, and I don't have to wait for the upstream authors of all software on my machine to release updates that upgrade their dependency.

> It’s essentially like version locking packages except some random Debian maintainer decides when it’s time to update.

Yes, but version locking isn't my problem. The crucial difference is that distros pick a version and support those for years, while upstreams usually force you to use the latest version all the time to get security support. With distros _I_ get to decide when I upgrade, and the reduced frequency is a nice bonus. Having a single entity for all software on the system is also valuable, as there's just one tool to learn and one place to check for updates.

Re: Python: Please stop screwing over Linux distros

#142

Post by jacquesm »

toyg wrote: post #134
ajsnigrutin wrote: post #120

Earlier quoted context omitted.

> Perhaps the distributions would be more inclined to include up-to-date version if the standard in the python community was not to break everything all the time. Yep... Perl is a godsend... take a code from 20 years ago, run it on a modern system, and everything works. Python? Three different software versions need three different versions of the same library, and new library versions are not backwards compatible wi…

> Perl is a godsend... take a code from 20 years ago, run it on a modern system, and everything works. That's more or less like "take a VB6 binary from 20 years ago, run it on some modern Windows, and everything works" - that's just because the ecosystem is effectively dead, so supporting it on new releases just means carrying over some stuff that worked 20 years ago.

So, Perl is the new Latin?

Re: Python: Please stop screwing over Linux distros

#143

Post by rvdginste »

samwillis wrote: post #83
elurg wrote: post #33

Python libraries shipped by distributions are so old that this mechanism is mostly useless for python development. This also applies to many others programming languages which have their own packaging systems. While python packaging is indeed messy the needs of traditional linux distributions are by far the least important. Python packaging needs to better serve the needs of python developers, not those of sysadmins.…

The fundamental problem is the difference between Curated and Uncurated packaging systems: - Linux dictros have curated packaging, often outdated but stable. - PyPi, NPM, and others are Uncurated, anyone can publish, could be unstable, could be insecure. It's down to the developer to decide which route they want to take, and at the moment most want to move quickly with the latest tools. To do that you have to go the…

> The fundamental problem is the difference between Curated and Uncurated packaging systems

Exactly this! It is expected from linux distros that they have curated packaging. I think that is good and I really expect it to stay that way.

Whether you want curated or uncurated packages depends on the use case.

As the user of a program, I definitely want curated packages.

As the developer of a program, I want to specify myself which version of a dependency I want to develop against and I don't want to be hindered by the linux distro in doing that. I do think that developers in that context are not always supported that well in linux distros. And on the other hand, I do think that tools supported by the programming language can assist in that scenario. (for example installing multiple versions of the compiler and runtime in the users home directory and being able to easily switch between those versions on a project basis...)

Re: Python: Please stop screwing over Linux distros

#144

Post by 1337shadow »

BiteCode_dev wrote: post #133
1337shadow wrote: post #116

Earlier quoted context omitted.

I'm sorry, doesn't pip already works there? Otherwise, there's pyinstaller which is great but it requires an entrypoint so it won't do with a pure lib that doesn't expose any script at all.

Yes that's the point. We won't use distro packages because only regular python packages allow this. Edit : arf, sorry, I answered the wrong comment, I actually agree with my parent.

All right, sorry I didn't get it (I'm non native)

Re: Python: Please stop screwing over Linux distros

#145

Post by 1337shadow »

rizkeyz wrote: post #88
captainmuon wrote: post #74

Distros, please stop screwing over Python packaging. It is incredible that Debian/Ubuntu pick Python apart and put modules into different packages. They even create a fake venv command that tells you to please install python-venv. What they should just do is offer a bunch of packages like python3.7, python3.8 that install the official python package wholesale into /usr/python or someplace and then symlink one of them…

This is the real legacy problem: Python comes from a world, where only one version of one packaged seemed the right way to do. I do not have a good idea, but other ecosystems evolved much more sane in the realm of packaging. While not ideal, Go has done a fairly good job - and the "module" operations are instant - which they should be.

Ok but Go compiles statically, while you can do the same with pyinstaller, I don't think that's really comparable as we're talking about deployment right there.

Re: Python: Please stop screwing over Linux distros

#147

Post by pfranz »

turminal wrote: post #100
pfranz wrote: post #87

Earlier quoted context omitted.

What happens if your project needs an update, but it will hose the OS? Do you expect someone working in Python in Windows who wants to distribute a datetime package should package for RPM and APT? I used to think OS package managers should be the end-all be-all, but the use cases for OS package managers are very different from language runtimes. While different Linux distros were fighting between themselves, they com…

> What happens if your project needs an update, but it will hose the OS? Whay do you mean by "needs an update"? > Do you expect someone working in Python in Windows who wants to distribute a datetime package should package for RPM and APT? Absolutely not. I expect a serious user of that library on an apt based system to package it and submit the package to their distro.

This was 10+ years ago, but I remember something like the installed Python package had a caching bug that was affecting production. The update wasn't compatible with some OS scripts so the machine would no longer boot using the updated package. Problems like that came up fairly often, but I remember that was the most egregious.

Another scenario is that the OS shipped with Python 2.5 (supported until May 2011), but we had third-party tools that required Python 2.7 (shipped July 2010). Switching OSes (where things like monitor or hardware drivers weren't yet supported) was a ridiculous pain to test and certify. Decoupling OS and Python+package versioning was a huge relief for everyone, but won't make sense for everyone.

> Absolutely not. I expect a serious user of that library on an apt based system to package it and submit the package to their distro.

I try my best to personally do this and push for a work culture that does this, but even if this was done I can't fathom waiting on an OS update for existing code to percolate down. The risk tolerance, scope of concern, and agility between an OS and whatever project pays the bills are very different.

Re: Python: Please stop screwing over Linux distros

#148

Post by gjulianm »

1337shadow wrote: post #126
gjulianm wrote: post #115

Earlier quoted context omitted.

> However, installing every dependency in the same environment is also what distro package managers do, I don't see that as a recipe for disaster, Distro packages are done in a way where it's hard to get conflicts. I haven't found an instance where python3-X and python3-Y can't be installed because one requires python3-Z v1.2 and the other python3-Z v2.1. With Python packages, that happens way more often. Also, distr…

> I haven't found an instance where python3-X and python3-Y can't be installed because one requires python3-Z v1.2 and the other python3-Z v2.1. With Python packages, that happens way more often. Exactly, because the packages were created/updated at a point in time were they were compatible: a distro ships python 3.x with packages of python modules that are compatible with python 3.x. Now if you as a user want to use…

> Again, if all python package were up to date as in "working together at this point in time" then `pip upgrade` would work.

That is impossible, it will never happen. Packages will have different maintenance cycles, some will get deprecated, others abandoned... You can't base your upgrade policy on an impossible situation.

> Are you sure you checked in the `*.dist-info` directories? There should be one per package, containing: a METADATA file, with all dependency versions, a RECORD file, with every installed file and their hash, and much more! see for yourself ;)

I know, but pip doesn't track all of those all the time. An example that has happened to me multiple times: install package X that depends on Y = 2.0. Pip will install Y >= 2.0 and won't care that package X is now broken.

> - in my point of view: this is a maintenance problem in the python packages themselves, caused by the general practice of pinning

Regardless on your views on pinning, it's a problem in pip. A version conflict should be reported as an installation failure, not allow you to continue.

And again, version restrictions will always be there. The "live at master" philosophy only works for small groups of similar output capacity. It won't work for an ecosystem as wide as Python. Even without version pinning, you'll still have packages breaking because another one was updated. Sometimes it will be necessary, such as for example a package dropping support for a feature that another one needs.

Re: Python: Please stop screwing over Linux distros

#150

Post by rollcat »

elurg wrote: post #33

Python libraries shipped by distributions are so old that this mechanism is mostly useless for python development. This also applies to many others programming languages which have their own packaging systems. While python packaging is indeed messy the needs of traditional linux distributions are by far the least important. Python packaging needs to better serve the needs of python developers, not those of sysadmins.…

> Python packaging needs to better serve the needs of python developers, not those of sysadmins.

Please substitute s/sysadmins/users/, and realise your developers are users too.

I've been doing Python for almost 15 years; and I'm getting really fed up with some things. Packaging is a mess. Distribution is a mess (for servers/IoT - Docker saves the day; for desktop - I feel like giving up). Managing the installations is a mess; upgrades can be impossible - I'm hard-stuck on Python3.6 on one project!

I find myself rewriting many smaller tools in Go or Rust, just because I can upgrade the toolchain at any time, and/or ship a static binary. But Rust has a very high barrier to entry, and Go tends to be simplistic.

I'd fully jump ship today, but Python has just too much momentum behind it.

Post reply on HN