Python: Please stop screwing over Linux distros • Hacker News Board

Re: Python: Please stop screwing over Linux distros

#211

Post by rolandog » Tue, Nov 16, 2021, 1:13 PM UTC

withinboredom wrote: post #13

noodlesUK wrote: post #2

I am not one to install python packages using my distro's package manager, but I totally agree with the sentiment that we need a more standard build/dependency management system in python. I like poetry, and I think most people are heading that way, but it doesn't seem to play super nice with pyenv (which is a critical tool) a lot of the time, and I think that a first party endorsement of the "one true build system"…

I’m not a Python dev. I do need to occasionally run things written in Python. I made the mistake of trying to get pip + Conda + pyenv (or whatever) to install a fairly simple tool. I have no idea how the dev got their setup working, but it was totally and utterly unreproducable, even after they sat down on my computer for several hours. In that amount of time, I could have probably rewritten it in PHP (that’s actuall…

I had to learn this by FUBAR'ing my system a long time ago, but my setup process for working with a python package from PyPI (i.e. not installable by or updated enough in OS's package manager) nowadays is:

- sudo apt install pip3

- pip3 install --user --upgrade pipenv

(In workdir):

- pipenv install --three package

- pipenv run package --option

Works like a charm and doesn't mess with my system.

Re: Python: Please stop screwing over Linux distros

#212

Post by the__alchemist » Tue, Nov 16, 2021, 1:14 PM UTC

Blaming Python vice Linux distros isn't important. The implicit agreement that a system Python will be set up in a certain way, with a certain version etc is fragile. Programs and processes included with a Linux distro expecting a certain Python set up is a problem - it's making assumptions about system state. Python doesn't include tools for managing versions, dependencies, and standalone programs. Linux distros use programs depend on it despite this.

At the core of this is implicit cooperation between system programs, third party programs, and users. This concept underlies a common frustration with Linux: It works reliably out-of-the-box, but customization and installing programs introduces problems.

I wrote a Python dependency and version / installation Manager in Rust to help deal with this sort of thing, as well as related issues like dep conflicts between Python projects.

Re: Python: Please stop screwing over Linux distros

#213

Post by mangecoeur » Tue, Nov 16, 2021, 1:18 PM UTC

microtonal wrote: post #48

mangecoeur wrote: post #9

Everyone complained, no one addressed the fact that for years the whole of python packaging was handled by like 2.5 people. But as a rant this, like most of the ‘but just fix it’ rants, fails to acknowledge the hugely diverging needs of different users. I could not live without conda, since it’s the only sane way to get a working recent geospatial stack. Others need to run embedded environments, or portable ones, som…

And yet, other languages ship package managers [1] that are widely loved within their ecosystems and cover everything from microcontrollers to server applications. I think what it makes it more difficult in the case of Python is that it has decades of legacy to deal with. No consistent semantic versioning, packages that expect that they can modify their package path in-place (this is a nightmare for immutable systems…

Worth noting that the PSF has no authority to tell people how to do their packaging.

Rust's Cargo is 20 years newer than Python and benefits from those decades of experience. It's a very different proposition to start a new system from fresh than to try to migrate a huge and diverse community towards it.

Re: Python: Please stop screwing over Linux distros

#214

Post by yxhuvud » Tue, Nov 16, 2021, 1:19 PM UTC

hkt wrote: post #7

noodlesUK wrote: post #2

I am not one to install python packages using my distro's package manager, but I totally agree with the sentiment that we need a more standard build/dependency management system in python. I like poetry, and I think most people are heading that way, but it doesn't seem to play super nice with pyenv (which is a critical tool) a lot of the time, and I think that a first party endorsement of the "one true build system"…

Golang is a great model to follow. I think the fundamental thing that python (and ruby, when I still used it) has taught me is to run a mile if a language is without a robust dependency management system. The pain is just not worth it, even for a nice language with an otherwise robust ecosystem.

Huh? The package dependency system in Ruby is a breeze, and has been a solved problem since 2007 or whenever Bundler was made.

Re: Python: Please stop screwing over Linux distros

#215

Post by RegW » Tue, Nov 16, 2021, 1:20 PM UTC

smallerfish wrote: post #123

RegW wrote: post #109

Earlier quoted context omitted.

Ant didn't do package management.

No, but ivy was commonly used. That said, the bigger point is that the standard way to do things can be changed. For some reason, despite (or because of?) PEP, the python community seems unable to coalesce around standard ways of packaging.

Perhaps I have lived too long out in the provences, but Maven was my first experience of dependency management in Java. After Ant it was a no-brainer because Ant didn't do dependencies. I didn't come across references to Ivy for some years and personally I have never seen it used in the wild.

That said, I didn't actually like early Maven that much, it was pretty inflexible and often required the AntRun plugin to do something novel. However, it is still top dog everywhere I work and very much the standard way to do things.

Re: Python: Please stop screwing over Linux distros

#216

Post by katbyte » Tue, Nov 16, 2021, 1:20 PM UTC

toyg wrote: post #202

ajsnigrutin wrote: post #183

Earlier quoted context omitted.

But it works... it's universally supported on pretty much every os, even by default on most *nix based ones, and new versions and new CPAN modules are still written. In 10 years, with python 4, or even maybe 5, everything will still be broken, and you'll still be claiming perl is dead, and i'll still be using the same stuff I use now, that worked years ago, works now, and will work then.

And people will still be shipping code faster and more efficiently than they would do in Perl. Because in the end, the advantages of using Python, in terms of readability and productivity, easily offset a bit of packaging pain - whereas the disadvantages of using Perl don't offset whatever marginal gain you get by using old infrastructure.

Or instead use another modern language? I avoid any python that I can’t apt-get/brew and haven’t considered writing anything with it for a very long time. There is always go/ruby/c# which seem to generally avoid the problems discussed here

Re: Python: Please stop screwing over Linux distros

#217

Post by dgan » Tue, Nov 16, 2021, 1:21 PM UTC

rlpb wrote: post #77

dgan wrote: post #44

Earlier quoted context omitted.

Exactly agreed on that. I don't understand what value distribution is providing by repackaging python libs, they re always way too old to be usable, and they re global while I work on many projects, with their own incompatible requirements. Maybe I am dumb, but I exclusively use virtualenv and pip..

> I don't understand what value distribution is providing by repackaging python libs... I want to easily and safely use some app my distribution ships. I want to receive security updates automatically for all such apps. I don't care what language it's written in or what its dependencies are. These app packages provided by the distribution have dependencies that are also packaged by the distribution so that dependency…

Valid remark. I had "development side" point of view, not "usage"

Re: Python: Please stop screwing over Linux distros

#218

Post by goodpoint » Tue, Nov 16, 2021, 1:22 PM UTC

pimterry wrote: post #95

captainmuon wrote: post #74

Distros, please stop screwing over Python packaging. It is incredible that Debian/Ubuntu pick Python apart and put modules into different packages. They even create a fake venv command that tells you to please install python-venv. What they should just do is offer a bunch of packages like python3.7, python3.8 that install the official python package wholesale into /usr/python or someplace and then symlink one of them…

> Get rid of the requirement that there is only one stable (minor) version of a package in the distribution at one time. I think this requirement made sense when disk space was scarce. I think this requirement makes sense if you trust that your distro is always better at choosing the 'best' version of a dependency that some software should use than the software author. Nowadays, I think neither is generally true. Dis…

> I think this requirement made sense when disk space was scarce.

No, the main reason is security. I need a distro to guarantee me that the library that I use are going to stay the same for the next 3 to 5 years, while also receiving small targeted patches.

> I think this requirement makes sense if you trust that your distro is always better at choosing the 'best' version of a dependency that some software should use than the software author.

No, it just has to be better at choosing version than picking them randomly using pip.

Furthermore, when thousands of developers use the same combination of libraries from a distro the stack receives a ton of testing.

The same cannot be said when each developer pick a random set of versions.

Re: Python: Please stop screwing over Linux distros

#219

Post by katbyte » Tue, Nov 16, 2021, 1:24 PM UTC

1337shadow wrote: post #116

BiteCode_dev wrote: post #108

Earlier quoted context omitted.

Sure, so my lib works on mac, windows, centos and debian. Tell me how I'm supposed to define my lib python deps ? How am I suppose to package the lib ? Distribute it ? Deal with os versions differences? Allow isolation for my user projects ?

I'm sorry, doesn't pip already works there? Otherwise, there's pyinstaller which is great but it requires an entrypoint so it won't do with a pure lib that doesn't expose any script at all.

I have apt/brew - I don’t want to install yet another package manager. Anytime I see just use pip I just walk away and find something else.

Re: Python: Please stop screwing over Linux distros

#220

Post by cturner » Tue, Nov 16, 2021, 1:25 PM UTC

pfranz wrote: post #107

WhyNotHugo wrote: post #47

Earlier quoted context omitted.

> Use pip install in user mode This is a great recipe for disaster. Whatever you install in user mode will shadow anything installed system-wide, so when you try to run some system-wide project, it may now fail. I'm also not a fan of how it drops scripts into `./.local/bin`, since that's where I keep my own script, and is version controlled. The installation will also be frozen and never get updated -- unless you rem…

So you're suggesting always using virtualenv? I used to just use pip to install to the system. Months/years later I would try to untangle the mess of packages I was just playing with, what the OS wanted/needed, I got those conflicting dependencies you mention, etc. I usually ended up reinstalling the OS. At the time I may not have been as knowledgeable about where the OS package manager keeps packages vs pip--but the…

I was slow to get to grips with venv. It sounds like you are on the same path. This note tries to be constructive advice -

* Some distro software uses python. Let the package manager take care of dependencies for that.

* For everything else, use an dedicated virtualenv for each codebase you are working with.

   > I used to just use pip to install to the system

Never do this, for the reasons you cite.

   > I've dabbled with virtualenv, but it's such a pain to set up and activate

Setup for virtualenv: "python3 -B -m venv venv". Have a shell alias 'alias v=". venv/bin/activate"' that allows you to activate it if you need to install libraries or access a shell. "pip install blah" for library install. That should be all you need.

   > If I have a few projects with similar libraries
   > it's more of a pain to set them all up and switch around

Have a think about why you feel this way, and whether you could mitigate the problems.

Here is what I do. Once my libraries are installed for the current project, I rarely activate venv in the current shell. Rather, for each python project, I have a bash script "app" in the root of the project, and a dedicated "venv" directory.

The app script does the following: (1) sources the local venv; (2) does pip freeze > requirements.txt to capture any dependency changes; (3) launches the project. Often I will have multiple launchers in that script, with all of them commented out except for the active one. Be in a habit of always launching from that app script.

To reiterate the approach above, whenever you sit down to write some python code, ensure that you have a dedicated venv for it, and that you are only ever launching code from that local venv.

I have spoken to developers who get upset at the extra hard disk overhead. You don't need to optimise for hard disk usage. Hard disk space is almost free.

I don't bother creating setup.py files, except for the odd occasion that I want to publish code to pip. Good luck.