Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Python: Please stop screwing over Linux distros

drewdevault.com

Post reply on HN
Print view View original

51–60 of 384 posts

Re: Python: Please stop screwing over Linux distros

#51

Post by 1337shadow »

gjulianm wrote: post #25
1337shadow wrote: post #8

Earlier quoted context omitted.

> don't mess with the system-provided Python environments for specific applications you are working with Right, but then just use pip install --user instead of a virtualenv. Actually --user is the default now when running pip install from non-root user, so, just pip install as a user will work and not mess with the system packages, just don't do sudo pip install.

I've run into problems when running pip install --user. Someone installs a package in the user environment, and that works. But later you'll install a package in a shared virtualenv (because a Python program needs to run as a service or by another users or whatever) and pip doesn't install it because it's already on the user environment. However, other users don't have access to that environment and they will have an…

Interesting problem but I think it's more a problem in virtualenv, which don't use global site package by default, I'm surprised that it uses user site packages by default.

However, installing every dependency in the same environment is also what distro package managers do, I don't see that as a recipe for disaster, it all depends how well the said packages are maintained in which case just upgrading them should fix whatever problem.

Re: Python: Please stop screwing over Linux distros

#52

Post by sufehmi »

cdent wrote: post #46

I'm always confused by these sorts of posts because they happen often so there is clearly a problem but for some reason I've never had much of an issue. I've been using and developing with Python for about 15 years. In that time I've worked on Python projects large (OpenStack) and small (gabbi) and taken over maintenance for some old standbys (wsgi-intercept, paste to name two). Dealt with the 2->3 transition. Releas…

Just scroll down - loads of horror stories shared by others

Re: Python: Please stop screwing over Linux distros

#53

Post by rbanffy »

ivirshup wrote: post #42
Sesse__ wrote: post #36

Earlier quoted context omitted.

Users generally don't want Python packages; they want software. They don't care what it's written in, and shouldn't have to install it differently depending on what you chose. “Just learn $language_package_manager_of_the_day, and hope it doesn't break anything when your system changes” is the equivalent of the 90s “./configure ; make ; make install”, and we should have moved past that for end-users by now. Most users…

But our package is for scientific data analysis. It’s a python library. If you’re distributing an application, shouldn’t you just ship an environment with our package bundled?

> shouldn’t you just ship an environment with our package bundled?

That's how I would prefer it, but, if the intended form of distribution is a distro package, then it should pin itself to the versions the distro provides and avoid (or vendor in) packages that aren't available.

It is possible to just place everything in the app directory and distribute it this way, but it's kind of ugly (and doesn't pick up security updates from the distro)

Re: Python: Please stop screwing over Linux distros

#54

Post by BerislavLopac »

noodlesUK wrote: post #2

I am not one to install python packages using my distro's package manager, but I totally agree with the sentiment that we need a more standard build/dependency management system in python. I like poetry, and I think most people are heading that way, but it doesn't seem to play super nice with pyenv (which is a critical tool) a lot of the time, and I think that a first party endorsement of the "one true build system"…

> I like poetry, and I think most people are heading that way, but it doesn't seem to play super nice with pyenv

Do you have an example of a situation when the two tools had issues? I've been using both for years, and don't remember having any problems.

Re: Python: Please stop screwing over Linux distros

#55

Post by dolmen »

tuldia wrote: post #18

Am I the only one that is not having issues with python and distributions in general? I get all my dependencies from Debian and they all work, when I need something that is not yet packaged, I use pip. What are people doing to get all this issues? I don't understand...

Looks like the "Works on my machine" syndrome.

What about working on a project with a team?

What about deploying the code on another machine?

Re: Python: Please stop screwing over Linux distros

#56

Post by roenxi »

mangecoeur wrote: post #9

Everyone complained, no one addressed the fact that for years the whole of python packaging was handled by like 2.5 people. But as a rant this, like most of the ‘but just fix it’ rants, fails to acknowledge the hugely diverging needs of different users. I could not live without conda, since it’s the only sane way to get a working recent geospatial stack. Others need to run embedded environments, or portable ones, som…

> Everyone complained, no one addressed the fact that for years the whole of python packaging was handled by like 2.5 people.

If I can make an observation - it has nothing to do with the number of maintainers. The problem is deep, cultural and occupies a difficult space where it might be a bug or a feature.

The root cause here is that the Python project, and surrounding community, have little real respect for backwards compatibility. The complexity of Python setups is driven by the need to run multiple - potentially even mutually incompatible in the case of 2.x v. 3.x - versions of Python.

All languages have packaging problems, but Python is unique in my experience in the sheer number of Python installs that I need to manage simultaneously. I still have C code that works from around the time that I learned C. I'd need another Python environment installed to say the same thing about Python.

Re: Python: Please stop screwing over Linux distros

#57

Post by 1337shadow »

CornCobs wrote: post #35

I once attempted to install a program that was distributed via pypi (pip was the only way I could install software on that server). The shebang in the script file was #!/bin/python. Thats python2 on most Linux distros and so it couldn't run

That's weird, the console_scripts hook for python packages generates a script with the same python in the shebang that pip is executed with to install the said package.

Re: Python: Please stop screwing over Linux distros

#58

Post by salawat »

formerly_proven wrote: post #20
salawat wrote: post #10

Earlier quoted context omitted.

Apt and desktop environments need to break their Python dependencies then, or stop doing daft things like colliding different 3.x run levels on the same runtime path. Go ahead and install pip3 from Python 3.6. use that to install pip3 for 3.8, and try using apt. Backup first.

Your last sentence doesn't parse for me at all. How exactly do you get a pip installed on Python 3.6 to modify anything about another install of Python, let alone a different version?

Apt install python-pip3

You'll get pip3 from Python 3.6.

apt install python3.8

You'll get system-level python3.

Use pip3 in 3.6 to install the latest pip3 that requires Python 3.8, and drops that package in the dist-packages folder, which is shared between system Python runtime versions.

Behold as further attempts to do something sane with APT blow up, because now you have 3.8 packages on 3.6's runtime path.

I've lost weeks to this particular brand of distro daftness. I thought that surely no one would do that .. Alas...

Re: Python: Please stop screwing over Linux distros

#59

Post by gjvc »

pipeline_peak wrote: post #3

And stop putting everything on the planet into a god damn virtual environment.

In an enterprise world, you can't rely on dependencies being present on the target machine, so install a .venv/ directory in your application distribution containing all the libraries (populated by pip) and a bin/venv-python wrapper script to set PYTHONPATH correctly and to call the .venv/bin/python. Then create a bin/run-app script to call bin/venv-python, with the location of your entry point module from src/python/

In summary: bin/run-app -> bin/venv-python -> .venv/bin/python -- I'm not sure how to make it simpler than this. The cost is perhaps in disk space, but I don't care about that.

Or don't do the above and deploy via docker and use pip to install to the system python in the image.

Re: Python: Please stop screwing over Linux distros

#60

Post by geofft »

turminal wrote: post #40
geofft wrote: post #32

Earlier quoted context omitted.

Virtualenvs don't require using unfiltered/unsupervised/untrusted sources. They're a place to install things into, not a place to get things from. The specific model that distros could adopt, if we went this route, is that each Python package builds into a .whl, they're build-dependencies of applications, and applications install .whls into a virtualenv at build time. You'd still restrict packages to come from the di…

The problem here is that distros only supply one version of the package which means you gain nothing by having that same version installed in multiple virtualenvs. So I guess the problem actually lies in the python library ecosystem that's becoming a npm like dependency hell. The relevant question here is probably whether the library ecosystem is like that because the packaging tooling sucks or is it the other way ar…

Well, that's not true:

- Almost all distros don't just supply one version of a package. They try to avoid it, but I can't recall one with a hard rule against it. For instance, Debian packages multiple versions of autoconf https://packages.debian.org/search?keywords=autoconf2 , the Linux kernel, etc.

- One reason that distros try not to install multiple versions of a package is that it's hard to specify which one you want. If you have, say, requests 1.0 and 2.0 installed, which one does "import requests" get you? The virtualenv approach, where "import requests" simply does not work in an un-virtualenv'd Python, avoids this problem entirely. Each application can independently build-depend on python-requests-1 or python-requests-2, and each user can create their own virtualenv and pip install /usr/share/wheels/requests-1.whl or requests-2.whl as they prefer.

- Even if you do not have different versions of the same library, it may well be the case that you have two different libraries with the same importable name. As a great example, see CJ Wright's talk from PackagingCon last week "Will the Real Slugify Please Stand Up" https://pretalx.com/packagingcon-2021/talk/P3983F/ (I expect they'll post videos online soon). tl;dr there are three packages you can get via "import slugify", and they expose different APIs.

- The possibility you haven't accounted for is "The library ecosystem is like that because things are fast-moving, because people have actual problems they want to solve, and upgrading dependencies and sorting out conflicts is work." It would be great for that work to be done, but we go back to the problem I mentioned at the top - limited volunteer time. In the absence of time to engineer things perfectly, your options are to ship something that's engineered imperfectly or decide not to ship it. We went through the dark ages of "We'll ship the next Debian release when all the bugs are solved" over a decade ago, and it turns out that this doesn't actually help users in any way.

Post reply on HN