Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Python: Please stop screwing over Linux distros

drewdevault.com

Post reply on HN
Print view View original

191–200 of 384 posts

Re: Python: Please stop screwing over Linux distros

#191

Post by hocuspocus »

thinker5555 wrote: post #165

As a "self taught and still learning developer-lite", I love the python language, but the ecosystem drives me nuts. I feel a lot of the pain expressed in the article, and it pretty much speaks to my current conclusion of "I'm trying to do things the 'right way' but there doesn't seem to be a 'right way'". I've seen a few comments here about how Nix/NixOS fixes the whole python binary/library mess, but I'm having trou…

Scala has its own issues, but it's worth a try.

It's easy to learn coming from Python, even more so if you can start with Scala 3 right away. You cannot really get more high-level than that. There are good libraries to work with databases, Quill comes to mind if you want something user-friendly. And you can always fall back to the many Java libraries in the big data ecosystem (even though you should probably avoid it if you can).

Re: Python: Please stop screwing over Linux distros

#192

Post by 3np »

captainmuon wrote: post #74

Distros, please stop screwing over Python packaging. It is incredible that Debian/Ubuntu pick Python apart and put modules into different packages. They even create a fake venv command that tells you to please install python-venv. What they should just do is offer a bunch of packages like python3.7, python3.8 that install the official python package wholesale into /usr/python or someplace and then symlink one of them…

The problem arises when non-Python packages depend on Python modules.

> If I would get to redesign package management (both for Linux distros and for languages), I would have one package manager that installs everything into a global package cache, and then pick the correct version of libraries at run time (for Python: at import time). Get rid of the requirement that there is only one stable (minor) version of a package in the distribution at one time. This has become unworkable. Instead, make it easy to get bleeding edge versions into the repositories. They can be installed side by side and only picked up by the things that actually use them.

You may want to check out Guix and Nix - their approach is pretty close to what you're describing.

A common solution to this is if you still want to run traditional distros is to just run "bare infra" (whatever that means) on the host OS and everything else in containers or Nix.

Re: Python: Please stop screwing over Linux distros

#193

Post by drunkpotato »

toyg wrote: post #127
nerdponx wrote: post #92

Earlier quoted context omitted.

Homebrew does this very well with its "cellar" system. Every version of every package gets installed to its own root tree, eg `/usr/local/Cellar/python/3.9.7/`. The currently-active version is then symlinked into `/usr/local/opt/python` and from there into `/usr/local`. I believe Nix and Guix also work this way.

I'll remember that "Homebrew does this very well" the next time I have to fix a bunch of shit because it has updated the currently-active version, or removed this or that bugfix release, as part of a general upgrade. After the third time this happened, I started using pyenv - which is another mountain of brokenness, I grant you, but at least I have some degree of control on what happens when.

pyenv is pretty good for working around this problem. I've recently switched to asdf-vm which I like even more, since it handles versions for multiple languages and tools.

Re: Python: Please stop screwing over Linux distros

#194

Post by hnarn »

captainmuon wrote: post #74

Distros, please stop screwing over Python packaging. It is incredible that Debian/Ubuntu pick Python apart and put modules into different packages. They even create a fake venv command that tells you to please install python-venv. What they should just do is offer a bunch of packages like python3.7, python3.8 that install the official python package wholesale into /usr/python or someplace and then symlink one of them…

> Get rid of the requirement that there is only one stable (minor) version of a package in the distribution at one time.

I’m not exactly sure how it works but I think I’ve heard that newer releases of Enterprise Linux (EL8+) support multiple channels of the same package or something similar.

Re: Python: Please stop screwing over Linux distros

#195

Post by silvestrov »

selfhoster11 wrote: post #173
pimterry wrote: post #95

Earlier quoted context omitted.

> Get rid of the requirement that there is only one stable (minor) version of a package in the distribution at one time. I think this requirement made sense when disk space was scarce. I think this requirement makes sense if you trust that your distro is always better at choosing the 'best' version of a dependency that some software should use than the software author. Nowadays, I think neither is generally true. Dis…

> Disk is plentiful, What makes you think so? SSDs aren't exactly stellar in the cost-per-TB department, as will be the case with each new higher-performance storage technology. Plenty of people cannot afford the prices of new Western tech either, what about them?

> cost-per-TB

There you have it: You measure in TB, not gigabytes, not megabytes.

Python packages are megabytes.

Re: Python: Please stop screwing over Linux distros

#196

Post by amalcon »

selfhoster11 wrote: post #173
pimterry wrote: post #95

Earlier quoted context omitted.

> Get rid of the requirement that there is only one stable (minor) version of a package in the distribution at one time. I think this requirement made sense when disk space was scarce. I think this requirement makes sense if you trust that your distro is always better at choosing the 'best' version of a dependency that some software should use than the software author. Nowadays, I think neither is generally true. Dis…

> Disk is plentiful, What makes you think so? SSDs aren't exactly stellar in the cost-per-TB department, as will be the case with each new higher-performance storage technology. Plenty of people cannot afford the prices of new Western tech either, what about them?

It's plentiful relative to the size of compiled or source code. E.g the biggest .so file on this system right now is a <150MB libxul.so. That's only used by one piece of software anyway, and the drop-off is pretty steep after that. A 64GB drive (tiny these days) can fit more than four hundred of that unusually large file.

Re: Python: Please stop screwing over Linux distros

#197

Post by AnIdiotOnTheNet »

selfhoster11 wrote: post #173
pimterry wrote: post #95

Earlier quoted context omitted.

> Get rid of the requirement that there is only one stable (minor) version of a package in the distribution at one time. I think this requirement made sense when disk space was scarce. I think this requirement makes sense if you trust that your distro is always better at choosing the 'best' version of a dependency that some software should use than the software author. Nowadays, I think neither is generally true. Dis…

> Disk is plentiful, What makes you think so? SSDs aren't exactly stellar in the cost-per-TB department, as will be the case with each new higher-performance storage technology. Plenty of people cannot afford the prices of new Western tech either, what about them?

> SSDs aren't exactly stellar in the cost-per-TB department

First of all, 1TB for binaries and libraries may as well be infinite. Secondly, you can get a 1TB SSD for under $100, which is pretty damned inexpensive when you consider it took until 2009 to get HDDs that affordable.

Re: Python: Please stop screwing over Linux distros

#198

Post by pwang »

dagw wrote: post #170
jacquesm wrote: post #138

Earlier quoted context omitted.

I'm using Anaconda, it's so far hands down the best way to manage python and its packages. It just works.

hands down the best way to manage python and its packages. Agreed, especially on Windows. It just works. This is pushing it. It's not hard to break conda or put yourself in situation where the updater/dependency checker gets stuck and doesn't know what to do, especially once you start adding conda-forge packages. But it does do a better job than anything else I've tried (although poetry + pyenv on Linux is getting mu…

Thanks!

FWIW, we are soon going to be releasing a much faster depedency resolver. We are also thinking hard about how best to address the "growing ecosystem" problem, in a future-proof way.

Re: Python: Please stop screwing over Linux distros

#199

Post by AnIdiotOnTheNet »

nerdponx wrote: post #92
captainmuon wrote: post #74

Distros, please stop screwing over Python packaging. It is incredible that Debian/Ubuntu pick Python apart and put modules into different packages. They even create a fake venv command that tells you to please install python-venv. What they should just do is offer a bunch of packages like python3.7, python3.8 that install the official python package wholesale into /usr/python or someplace and then symlink one of them…

Homebrew does this very well with its "cellar" system. Every version of every package gets installed to its own root tree, eg `/usr/local/Cellar/python/3.9.7/`. The currently-active version is then symlinked into `/usr/local/opt/python` and from there into `/usr/local`. I believe Nix and Guix also work this way.

IIRC GoboLinux was the first distro to do things this way. Sadly, it didn't catch on and the Linux world doubled down on labor intensive volunteer package maintenance.

Re: Python: Please stop screwing over Linux distros

#200

Post by kijin »

scotty79 wrote: post #168
kijin wrote: post #139

Earlier quoted context omitted.

It's not just about disk space, though. Distros assume responsibility for fixing major bugs and security vulnerabilities in the packages they ship. Old versions often contain bugs and vulnerabilities that new versions don't. Distros have two choices here: either ship the new version and remove the old version, or backport the fix to the old version. Continuing to ship the old version without the fix is not an option…

> Distros assume responsibility for fixing major bugs and security vulnerabilities in the packages they ship. I think they should just ship Python programs, not libraries. They could check what libraries given Python program uses are safe in the version that it uses them. And just don't care if each of Python programs has a separate copy of the libraries or if particular version of particular library is shared betwee…

So, patch two dozen copies of slightly different versions of the same library included in all those programs, instead of patching just one?
Post reply on HN