I must admit I just gave up on all these tools. Instead I just do pip install -r requirements.txt -t .lib and I have import sys import os sys.path.append(f"{os.getcwd()}/.lib") in the top of my script. Some will tell me this is silly, but it just works. Rememer to at .lib to your .gitignore. Else you'll have lot of fun.
As the person who implemented the -t flag and knows the horrendously sharp edge cases that it fails on, all I can say is bless your heart.
How to improve Python packaging
161–170 of 204 posts
Re: How to improve Python packaging
#162Python packaging is a solved problem: https://python-poetry.org/
Poetry is one of the better options, but its nonstandard pyproject.toml is not ideal. PDM is basically Poetry, with the metadata standard in place, and with support for __pypackages__.
Re: How to improve Python packaging
#163Re: How to improve Python packaging
#164Re: How to improve Python packaging
#165Earlier quoted context omitted.
Yup. I like to make it so that each executable has a symlink to a shell script. The shell script checks if there’s a virtual environment set up with the packages in the requirements.txt installed (it takes a snapshot of the file because virtualenv doesn’t have a DB to query cheaply). Once the environment is set up, it dispatches to running from the virtualenv. That way when you update a requirements.in file it recomp…
You’ve described a worse, homegrown version of poetry.
Re: How to improve Python packaging
#166Re: How to improve Python packaging
#167Earlier quoted context omitted.
No way. Docker is not designed for security.
Well, it's probably more secure than doing `pip install` and letting every dependency run whatever to install itself on your host...
Additionally, even if you only rely on a container - still no: other container engines have a lower attack surface.
Re: How to improve Python packaging
#168I have a single, simple script (not a package!) that has dependencies. Actually, I have a few of these, just sitting in /usr/local/bin so I can execute them whenever. How should I be managing environments for these scripts? Do I install dependencies in shared system python? Should I create a shared venv? Where should I store it? Any tools out there that make this decision for you and manage it? Just the fact that hom…
---
Disclaimer before I continue with my own suggestions: I have made small code contributions to pip and pip-tools, and maintain a Zsh frontend to pip-tools+venv, called zpy.
---
Suggestions:
- each folder of code maps to its own default venv, and may map to more venvs for different Python runtimes
- each folder of code has a requirements.in file with top-level dependencies, and a requirements.txt as a lock file
- you can either add a shebang line for your script which explicitly invokes the venv's Python and link that file into ~/.local/bin/, or instead create an external launcher script in ~/.local/bin/.
---
Here's an example of how that might be done using zpy functions:
$ mkdir simple-scripts
$ cd simple-scripts
$ envin # or in subcommand form: zpy envin
==> creating -> ~/.local/share/venvs/280…/venv :: ~/Code/simple-scripts
$ pipacs httpx
==> appending -> requirements.in :: ~/Code/simple-scripts
httpx
==> compiling requirements.in -> requirements.txt :: ~/Code/simple-scripts
anyio==3.6.2 # via httpcore
certifi==2022.12.7 # via httpcore, httpx
h11==0.14.0 # via httpcore
httpcore==0.16.3 # via httpx
httpx==0.23.3 # via -r requirements.in
idna==3.4 # via anyio, rfc3986
rfc3986==1.5.0 # via httpx
sniffio==1.3.0 # via anyio, httpcore, httpx
==> syncing requirements.txt -> env :: ~/Code/simple-scripts
$ print -rl -- 'from httpx import get' 'print(get("https://ifconfig.co/json").json())' >do_it.py
$ vpyshebang do_it.py # or: zpy vpyshebang do_it.py
$ ln -s $PWD/do_it.py ~/.local/bin/do_it
If a Python runtime update breaks your venvs, you can probably fix things up using zpy's pipup function.Re: How to improve Python packaging
#169Note this post is 8700 words long and it doesn't really delve into how bad it is to build packages on Python; mostly it's about how bad it is to use them. It's an excellent writeup and the length is necessary. My guess is a post about creating packages would need to be 2x as long for the same level of detail.
Even this book doesn't cover all options in each area, and it skips almost wholly over conda because I have no personal experience using it. conda and the work in the scientific community adds complexity both to the creation and the consumption side of packaging, and that's one area I'm not sure this post covers all the nuance of when considering how a "one size fits all" solution might work in practice.
Re: How to improve Python packaging
#170Earlier quoted context omitted.
And one of the reasons I moved away from it! My blog post on the subject was posted here and it was very unpopular despite having similar points and also making comparisons with Node. There's nothing special about Python that makes it worth enduring this pain. We have other languages with a better developer UX.
I'd be keen to read your blog post if you have a link?
Bear in mind this was not written for an audience, let alone a HN audience.