Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Python: Please stop screwing over Linux distros

drewdevault.com

Post reply on HN
Print view View original

311–320 of 384 posts

Re: Python: Please stop screwing over Linux distros

#311

Post by kijin »

thereddaikon wrote: post #289
kijin wrote: post #200

Earlier quoted context omitted.

So, patch two dozen copies of slightly different versions of the same library included in all those programs, instead of patching just one?

Why not? Its cheap resource wise whereas dependency hell is potentially debilitating. For some reason many proponents of the package management status quo are blind to this. Having multiple versions of a dependency is only bad in so far as its "messy". It isn't objectively bad. But having a system that breaks applications because two or more cant agree on a package version is objectively bad. Its arguing aesthetics v…

Disk is cheap. RAM is cheap. Man-hours are not. Distros are maintained by people, who are often volunteers. You are asking them to do extra work (i.e. porting the same patch to multiple versions of the same library) so that someone else can have it easy. But why should they? Why not the other way around?

It's not just aesthetics. If a new vulnerability is found in, say, libjpeg, then every Windows app that uses libjpeg needs to be updated separately. Tough luck if your favorite image manipulation tool takes a while to catch up. On the Linux side of the fence, the distro patches libjpeg and every app is automatically fixed. This is a huge win in terms of protecting the user. Why should we give up that benefit just because some developer wants to use his own special snowflake version of a library?

Re: Python: Please stop screwing over Linux distros

#312

Post by maple3142 »

goodpoint wrote: post #228
Gigachad wrote: post #119

Earlier quoted context omitted.

I really doubt that the python libraries packaged by Debian are any more secure or stable than the latest release of those libraries. At best they just limit to breaking updates to once every few years when they update them. It’s essentially like version locking packages except some random Debian maintainer decides when it’s time to update.

The python libraries packaged by Debian are provably more secure or stable: they receive security backports while also not receiving feature updates. The combination of this 2 aspects is what provides better stability and better security. > It’s essentially like version locking packages except some random Debian maintainer decides when it’s time to update. Not at all.

I am not sure if Debian can really keep give packages security updates without feature updates. For example, Debian packaged Chromium seems really outdated and having many unpatched CVEs: https://security-tracker.debian.org/tracker/source-package/c...

Re: Python: Please stop screwing over Linux distros

#313

Post by hatsuseno »

BiteCode_dev wrote: post #261
nerdponx wrote: post #12

> Every one of these package managers is designed for a reckless world in which programmers chuck packages wholesale into ~/.pip, set up virtualenvs and pin their dependencies to 10 versions and 6 vulnerabilities ago, and ship their computers directly into production in Docker containers which aim to do the minimum amount necessary to make their user’s private data as insecure as possible. Is this any different from…

There are some differences indeed: - python is more active than most alternatives, you have new packages created every day. - python is massively used outside of the web, unlike JS, ruby or PHP, that are 99% web. You get Python in SIG softwares, data analytics, automation, pen testing, sysadmin, biology, etc. It's a huge graph. - python is used by the distro themselves to code features of the OS. E.G: you remove Pyth…

Aside from the variety of languages in the extension ecosystem, how does this explain Perl doesn't generally share these dependency issues?

Re: Python: Please stop screwing over Linux distros

#314

Post by robertlagrant »

1337shadow wrote: post #112
robertlagrant wrote: post #85

Earlier quoted context omitted.

> Right, but then just use pip install --user instead of a virtualenv This seems almost as bad as system python. I suppose it's fine if you only work on one thing, but as soon as you don't, your dev environment will become chaotic and lots of confusing WFM will happen. E.g. this is why npm has a separate node_modules folder for each project.

> why npm has a separate node_modules folder for each project This sacrifies time and disk space and hides tech debt under the carpet, I'd rather have a solution like `pip upgrade` that would upgrade all packages and fix the environment, like `pacman -Syu`, but people would have to stop pinning versions and actually maintain their codebases and the dependencies they uses.

> but people would have to stop pinning versions and actually maintain their codebases and the dependencies they uses

Virtual environments is the mechanism by which you do that in a non-silly way. How do you think people cope with a dependency that has a bug introduced in the most recent version?

Re: Python: Please stop screwing over Linux distros

#315

Post by 1337shadow »

gjulianm wrote: post #293
1337shadow wrote: post #169

Earlier quoted context omitted.

> That is impossible, it will never happen. Packages will have different maintenance cycles, some will get deprecated, others abandoned... You can't base your upgrade policy on an impossible situation. You realize that if that was "impossible" and "never happening", then absolutely no Python environment would be working ever? > A version conflict should be reported as an installation failure, not allow you to continu…

> You realize that if that was "impossible" and "never happening", then absolutely no Python environment would be working ever? No, it means that you can't be fully sure that upgrading everything to the latest version is not going to break anything. And right now, you can't. You can only do that in controlled environments (say, a distro's official package repositories), not with Python packages. > Please don't make t…

> you can't be fully sure that upgrading everything to the latest version is not going to break anything.

Well, it's still what you do every time you create a new project.

> It is mandatory in quite a lot of package managers.

Ok but many packages won't be installable at all anymore, because people pin different versions instead of maintaining their packages by continuously integrating upstream releases.

> You can't live at latest release with software coming from wildly different developers, with wildly different policies on compatibility, versioning, breaking changes, bugs...

Well I didn't know can't so I did.

> I can't just take the latest numpy and patch it for Python 3.6.

Instead, you should be upgrading your code to support the newer numpy version. Seriously, try it out, you'll be spending less effort at the end of the year.

> You also spend time doing maintenance and debugging on new installations to debug and fix compatibility issues, when those upgrades might not bring any value to the product.

I spend less time and more spread over the year, bringing value to the whole ecosystem of packages which are bringing value to my product, and bringing value to myself as I develop new products with the dependencies I like and support as such.

> In my case, I do that debugging and fixing in a controlled environment

I wait for CI or users to report that a new release isn't compatible so I can fix it as soon as possible, in which case we temporarily pin versions, instead of piling up tech debt until it's so huge the whole project becomes trash.

That's what I call taking responsibility with the dependencies that you include in your product.

Re: Python: Please stop screwing over Linux distros

#316

Post by zamadatix »

josefx wrote: post #283
thereddaikon wrote: post #267

Earlier quoted context omitted.

Fortunately there are a thousand gigabytes in a terabyte and nobody is using a system with single or double digit gigs of storage anymore. Unless you are trying to save a buck it seems 1TB is the standard today. My primary desktop has 4.

> Unless you are trying to save a buck it seems 1TB is the standard today. Buying a 1TB external SSD would more than double the cost of a raspberry pi 4 that and my ancient beagle board does fine running from 32 GB. > My primary desktop has 4. Those are rookie numbers for a primary system. Of course my Office system next to it is a lot lower specked with the test system next to it even lower.

Embedded systems really shouldn't be brought into play here but even then a 256 GB uSD card for the Pi is $25 dollars and itself far overkill. My entire primary desktop OS, firmware, DEs, and very extensive package set fits in 15 GB. Multiplying my primary system by 10 and sticking it on a Pi taking $25 of storage with plenty to spare is still not an argument against binary sizes, especially since there are niche distro spins used for that niche space anyways.

Re: Python: Please stop screwing over Linux distros

#317

Post by benibela »

toyg wrote: post #134
ajsnigrutin wrote: post #120

Earlier quoted context omitted.

> Perhaps the distributions would be more inclined to include up-to-date version if the standard in the python community was not to break everything all the time. Yep... Perl is a godsend... take a code from 20 years ago, run it on a modern system, and everything works. Python? Three different software versions need three different versions of the same library, and new library versions are not backwards compatible wi…

> Perl is a godsend... take a code from 20 years ago, run it on a modern system, and everything works. That's more or less like "take a VB6 binary from 20 years ago, run it on some modern Windows, and everything works" - that's just because the ecosystem is effectively dead, so supporting it on new releases just means carrying over some stuff that worked 20 years ago.

It is because Windows has a stable API and still has all the old functions

Re: Python: Please stop screwing over Linux distros

#318

Post by atoav »

jhrmnn wrote: post #104
atoav wrote: post #17

Earlier quoted context omitted.

I landed with poetry as well. The issue with python dependency managment in my book is, that it is incredibly (and needlessly) hard to learn how to do it properly. It took me years to figure out how things should work and there are still issues now and then that cost me a few hours to figure out and restore things. Meanwhile when I use Rust all these things are taken care of in cargo. It is part of the language. Ther…

To run code developed with Poetry, you shouldn’t even know it’s developed with Poetry. It should be released as a source distribution or a wheel on PyPI or as a conda package if it contains nontrivial binary extensions. These distributions should be then either packaged by the OS or they should be installed with all their dependencies into a separate virtual environment in /opt.

Sure, if I release the code I tend to do just that. However a lot of the code I have been writing was to be used on servers where installing poetry was not an option. That means there was no straightforward, well documented or otherwise easy way to copy the project onto the server an "just have it work". And this was not due to me not taking care of my dev environment.

Now I know how to do this, so this is not a problem. My complaint was mostly, that this was a waste of time.

Re: Python: Please stop screwing over Linux distros

#319

Post by TingPing »

captainmuon wrote: post #74

Distros, please stop screwing over Python packaging. It is incredible that Debian/Ubuntu pick Python apart and put modules into different packages. They even create a fake venv command that tells you to please install python-venv. What they should just do is offer a bunch of packages like python3.7, python3.8 that install the official python package wholesale into /usr/python or someplace and then symlink one of them…

Fedora works similar to this. They just package them all and you use which ever one for your private needs.

Re: Python: Please stop screwing over Linux distros

#320

Post by cozzyd »

silvestrov wrote: post #195
selfhoster11 wrote: post #173

Earlier quoted context omitted.

> Disk is plentiful, What makes you think so? SSDs aren't exactly stellar in the cost-per-TB department, as will be the case with each new higher-performance storage technology. Plenty of people cannot afford the prices of new Western tech either, what about them?

> cost-per-TB There you have it: You measure in TB, not gigabytes, not megabytes. Python packages are megabytes.

A typical virtualenv for a project I work on is > 500 MB. That adds up.
Post reply on HN