Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Python: Please stop screwing over Linux distros

drewdevault.com

Post reply on HN
Print view View original

251–260 of 384 posts

Re: Python: Please stop screwing over Linux distros

#251

Post by pxc »

omnicognate wrote: post #179

So here's a suggestion to shoot down. The biggest insight in software dependency management is that applications and libraries are different. Both have dependencies but they sit at different places in their dependency graphs. A library can be used together with other libraries and so cannot pin its own dependency versions (if all libraries did so there would conflicts everywhere) but instead can only specify constrai…

Some Nix tools do this for Python, giving you separate tools for building libraries and applications. Overall, you're describing how Nix/Nixpkgs works today.

> A library can be used together with other libraries and so cannot pin its own dependency versions (if all libraries did so there would conflicts everywhere) but instead can only specify constraints on its dependency versions (eg >=1.0.0).

This is a Python defect. IIRC, for example, Node libraries can and do pin their dependencies without conflict, because Node has no problem using multiple versions of the same library in a process.

Python can't do this. It could (there have been proofs of concept that make this work), but it doesn't.

Re: Python: Please stop screwing over Linux distros

#252

Post by BiteCode_dev »

debiandev wrote: post #236
smallerfish wrote: post #66

Yes, python packaging is a mess. And agreed, there are two separate use cases: development and using the software. But, are distros creating too much work for themselves by trying to package every itty-bitty python library (and for that matter, every npm library)? Are distros doing anything more than scanning CVE databases with the library versions, or are they _actually_ auditing the versions they choose? (Not that…

> Are distros doing anything more than scanning CVE databases with the library versions, or are they _actually_ auditing the versions they choose? Debian Developer here. Part of packaging work, for Python libraries or anything else, is to verify the reliability of the upstream developers, audit the code, set hardening flags, add sandboxing and so on. I spotted and reported vulnerability myself and it's not uncommon.

Please don't take it the wrong way, I have a lot of respect for distros packagers and maintainers. I donate to debian, and I report bugs. I think you are basically heroes of the FOSS world, because without your invisible (and frankly thankless) work, mine wouldn't exist.

But come on, there are 300k entries on pypi, 200k more for perl and 160k for ruby. I'm not even counting the whooping 1.3M on npm because I assume this is considered taboo at this point.

You cannot package 0.0001% of that, not to mentions updates.

And unless distros make it as easy to package and distribute deb/rpm/etc than it is to use the native package manager, this means distro packaging will never be attractive to most users because:

- they don't have access to most packages

- the provided packages are obsolete

- packages authors have no way to easily provide an update to the users

- it's very hard to isolate projects with various versions of the same libs, or incompatible libs

And that's not even mentioning that:

- package authors may not have the right to use apt/dnf on their machine.

- libs may be just a bunch of files in a git repo, which pip/gem/npm support installing from

- this is not compatible with anaconda, which has a huge corporate presence

- this is not compatible with heroku/databrics/pythonanywhere, etc

- this is hard to make it work with jupyter kernels

Now let's say all those issues are solved. A miracle happens. Sorry, 47 miracles happen.

That would force the users to create a special case for each distros, then for mac, then for windows. I have a limited amount of time and energy, I'm not going to learn and code for 10 different packaging systems.

It's not that we want to screw over linux distros. It's that it's not practical, nor economically viable to use the provided infra. The world is not suddenly going to slow down, vulnerabilities will not stop creeping up, managers will not stop to ask you to use $last_weird_things. This ship has sailed. We won't stay stuff only published 5 years ago with delays of months for every updates.

Re: Python: Please stop screwing over Linux distros

#253

Post by tyingq »

toyg wrote: post #134
ajsnigrutin wrote: post #120

Earlier quoted context omitted.

> Perhaps the distributions would be more inclined to include up-to-date version if the standard in the python community was not to break everything all the time. Yep... Perl is a godsend... take a code from 20 years ago, run it on a modern system, and everything works. Python? Three different software versions need three different versions of the same library, and new library versions are not backwards compatible wi…

> Perl is a godsend... take a code from 20 years ago, run it on a modern system, and everything works. That's more or less like "take a VB6 binary from 20 years ago, run it on some modern Windows, and everything works" - that's just because the ecosystem is effectively dead, so supporting it on new releases just means carrying over some stuff that worked 20 years ago.

>that's just because the ecosystem is effectively dead

Perl may be "dead", but that's not why it's so backwards compatible.

Re: Python: Please stop screwing over Linux distros

#254

Post by omnicognate »

thesuperbigfrog wrote: post #242
omnicognate wrote: post #179

So here's a suggestion to shoot down. The biggest insight in software dependency management is that applications and libraries are different. Both have dependencies but they sit at different places in their dependency graphs. A library can be used together with other libraries and so cannot pin its own dependency versions (if all libraries did so there would conflicts everywhere) but instead can only specify constrai…

>> An application sits at the front of the dependency graph. Nothing depends on it. In many cases, applications depend on other applications. This can easily been seen in most Linux package management systems and also for many applications that act as more user-friendly front-ends or automation for command-line utilities. Examples: * Debugger front-ends * Package management GUIs * Multi-media processing pipelines * M…

If they're depending on them as processes to launch then that doesn't mean they can't be installed separately/isolated with their own dependencies.

If they're depending on them as python modules to import, they're libraries not applications and should be installed as such.

Re: Python: Please stop screwing over Linux distros

#255

Post by simonh »

rolandog wrote: post #201
simonh wrote: post #129

Earlier quoted context omitted.

>Serving the needs of python developers vs. sysadmins is a false dichotomy. Python developers develop on a system that they need to admin. I couldn't disagree more. Even if the same person is admining a system they develop on, they almost certainly aren't going to admin the systems their users deploy on. The admin role and developer role should be completely separate with different goals and requirements. The system…

Interesting; how would you isolate the system python from the user in Debian/Ubuntu. Do you have a link where I could learn more about this?

You may not be able to completely, but basically if users can write python scripts and run them using system python that's a side effect. It's purpose is to run system scripts, and any other use must not interfere or compete with that in any way.

To be hones that wasn't always the case. In the early days Python was provided on these systems for users just as much as administrators and in the early days there weren't any system script as part of the OS that used it. However now there are, and the packaging and configuration of the system python has in some cases even been mangled somewhat to suit the needs of the distro. It's time to make a clean split between system python and user/developer python.

Re: Python: Please stop screwing over Linux distros

#257

Post by ShamelessC »

rollcat wrote: post #150
elurg wrote: post #33

Python libraries shipped by distributions are so old that this mechanism is mostly useless for python development. This also applies to many others programming languages which have their own packaging systems. While python packaging is indeed messy the needs of traditional linux distributions are by far the least important. Python packaging needs to better serve the needs of python developers, not those of sysadmins.…

> Python packaging needs to better serve the needs of python developers, not those of sysadmins. Please substitute s/sysadmins/users/, and realise your developers are users too. I've been doing Python for almost 15 years; and I'm getting really fed up with some things. Packaging is a mess. Distribution is a mess (for servers/IoT - Docker saves the day; for desktop - I feel like giving up). Managing the installations…

I agree to some extent, but I believe they were mostly trying to convey that much of the work is done by volunteers who are largely only motivated/have the resources to test a small subset of potential deployment scenarios. Those contributions are still valuable and contributors themselves don't owe you anything. The best way to fix this issue would be to get involved (or switch languages as you suggested, that's fine too).

Re: Python: Please stop screwing over Linux distros

#258

Post by nerdponx »

pwang wrote: post #209
nerdponx wrote: post #27

Earlier quoted context omitted.

This is especially frustrating to read because one of the main selling points of Conda is reproducibility. In data science teams, I've found it indispensable for making sure people can all run each others' project code. So for anyone reading this in the future: don't try to use Pyenv to install Conda. Pyenv tries to set up shims for every binary in the Conda env, which will likely break your PATH. Pyenv supports inst…

Thanks for the feedback... I had no idea that pyenv was hosing conda envs. Do you have more details on this brokenness?

My understanding of the problem is that Pyenv attempts to detect the contents of "/bin" relative to the top level of every Python installation that it manages.

It does this so that it can set up its shim to handle any executable that gets installed in any Pyenv-managed environment.

This is how Pyenv creates the "foobar is not available in your current environment, but is available in x.y.z" message. It's also a much more reliable solution than trying to explicitly whitelist every possible script that might get installed.

The problem is that this was only designed to work for Python executables and scripts installed by Pip. Conda environments can contain a lot more than that; it's not hard to end up with an entire C compiler toolchain in there (possibly even both GCC and LLVM) or even Coreutils.

If Pyenv detects `bin/gcc` in a Conda env, it will set up a system-wide shim for GCC, which no longer passes the `gcc` command along to the OS, but intercepts it, only to inform you that no such command exists in the current env!

So it's not that Pyenv hoses Conda envs. It's that Pyenv can hose PATH if you have it manage a Conda installation, and if that Conda installation ends up with non-Python stuff in `/bin`.

Obviously I don't know what exactly was broken when you tried to set up that application. But this particular adverse interaction bit me at work a few years ago, and ever since then I have insisted that Pyenv should never manage a Conda installation.

I think that's a reasonable policy anyway, in light of the facts that:

1) Conda isn't really a "Python distribution" anymore.

2) The Pyenv installer just runs the opaque Conda installer script and there's basically no way to control the version that gets installed.

3) They are different tools that serve different purposes and it doesn't make sense to have one manage the other anyway.

4) You probably shouldn't use the Python that's installed in the base Conda environment anyway. You need that to run Conda itself, and you want to keep the list of requirements small to make sure that updates can progress cleanly. It's basically the same as any Linux package manager like APT. Except of course, those tools don't generally support "environments" other than chroot.

Re: Python: Please stop screwing over Linux distros

#259

Post by bmj »

toyg wrote: post #202
ajsnigrutin wrote: post #183

Earlier quoted context omitted.

But it works... it's universally supported on pretty much every os, even by default on most *nix based ones, and new versions and new CPAN modules are still written. In 10 years, with python 4, or even maybe 5, everything will still be broken, and you'll still be claiming perl is dead, and i'll still be using the same stuff I use now, that worked years ago, works now, and will work then.

And people will still be shipping code faster and more efficiently than they would do in Perl. Because in the end, the advantages of using Python, in terms of readability and productivity, easily offset a bit of packaging pain - whereas the disadvantages of using Perl don't offset whatever marginal gain you get by using old infrastructure.

And people will still be shipping code faster and more efficiently than they would do in Perl.

I feel that quite a bit of software written these days is designed to have a short shelf life. I wonder how many of the "Show HN" posts will work, or even be useful, in five years? How much will be maintainable?

I've been thinking a lot about Matthew Crawford's writings on mechanical things[0] and how they might apply to my own craft as a software developer. The work, as described generally on HN, is still all about moving fast and breaking things. It doesn't matter if the Python code I crank out today works in five years (or is maintainable in five years) -- I just need to get my product to market. Some of us work in other domains where sustainability and repair-ability are important: if my software is going to be in the field, in users' hands, for ten years, I need to consider how reasonable it might be fix bugs. If I have to fight just to get the software to run, I've already lost that battle. In these cases, the dependability of something like Perl is really great. The complaint against Python here isn't about the language itself (I think many people would agree it is quite nice to use), but rather the larger ecosystem, which makes it very hard to maintain software over the long haul.

[0] See Shop Class as Soulcraft, The World Beyond Your Head, and Why We Drive.

Re: Python: Please stop screwing over Linux distros

#260

Post by skykooler »

AnIdiotOnTheNet wrote: post #197
selfhoster11 wrote: post #173

Earlier quoted context omitted.

> Disk is plentiful, What makes you think so? SSDs aren't exactly stellar in the cost-per-TB department, as will be the case with each new higher-performance storage technology. Plenty of people cannot afford the prices of new Western tech either, what about them?

> SSDs aren't exactly stellar in the cost-per-TB department First of all, 1TB for binaries and libraries may as well be infinite. Secondly, you can get a 1TB SSD for under $100, which is pretty damned inexpensive when you consider it took until 2009 to get HDDs that affordable.

On a desktop, sure. Now that more and more laptops are starting to have soldered-down storage, this argument falls apart.
Post reply on HN