Live data from Hacker News

How to improve Python packaging

chriswarrick.com

161–170 of 204 posts

Re: How to improve Python packaging

#161

I must admit I just gave up on all these tools. Instead I just do pip install -r requirements.txt -t .lib and I have import sys import os sys.path.append(f"{os.getcwd()}/.lib") in the top of my script. Some will tell me this is silly, but it just works. Rememer to at .lib to your .gitignore. Else you'll have lot of fun.

As the person who implemented the -t flag and knows the horrendously sharp edge cases that it fails on, all I can say is bless your heart.

I'd be curious to know what they are, as I'm interested in giving the -t flag a try myself.

Re: How to improve Python packaging

#162

Python packaging is a solved problem: https://python-poetry.org/

Poetry is one of the better options, but its nonstandard pyproject.toml is not ideal. PDM is basically Poetry, with the metadata standard in place, and with support for __pypackages__.

I would like to add that PDM also supports having multiple dependencies groups, and the developer is superfast at fixing issues.

Re: How to improve Python packaging

#164
Note this post is 8700 words long and it doesn't really delve into how bad it is to build packages on Python; mostly it's about how bad it is to use them. It's an excellent writeup and the length is necessary. My guess is a post about creating packages would need to be 2x as long for the same level of detail.

Re: How to improve Python packaging

#165
post #94

Earlier quoted context omitted.

Yup. I like to make it so that each executable has a symlink to a shell script. The shell script checks if there’s a virtual environment set up with the packages in the requirements.txt installed (it takes a snapshot of the file because virtualenv doesn’t have a DB to query cheaply). Once the environment is set up, it dispatches to running from the virtualenv. That way when you update a requirements.in file it recomp…

You’ve described a worse, homegrown version of poetry.

That's pretty rude and ungenerous considering that I did this in 2015 before poetry even existed. Also, I could be wrong, but briefly taking a look, it seems like it still has the problem that it doesn't automatically update your virtualenv when the dependencies for an executable have changed (i.e. someone merges in code that uses a new dependency, you still have to know to run some commands to update your own local virtualenv).

Re: How to improve Python packaging

#167

Earlier quoted context omitted.

No way. Docker is not designed for security.

Well, it's probably more secure than doing `pip install` and letting every dependency run whatever to install itself on your host...

Actually not at all assuming you are using a VM for a production workload.

Additionally, even if you only rely on a container - still no: other container engines have a lower attack surface.

Re: How to improve Python packaging

#168

I have a single, simple script (not a package!) that has dependencies. Actually, I have a few of these, just sitting in /usr/local/bin so I can execute them whenever. How should I be managing environments for these scripts? Do I install dependencies in shared system python? Should I create a shared venv? Where should I store it? Any tools out there that make this decision for you and manage it? Just the fact that hom…

If you don't want to rely on the system or brew-installed Pythons for all your scripts, you might want to manage multiple Python installations with asdf-vm.

---

Disclaimer before I continue with my own suggestions: I have made small code contributions to pip and pip-tools, and maintain a Zsh frontend to pip-tools+venv, called zpy.

---

Suggestions:

- each folder of code maps to its own default venv, and may map to more venvs for different Python runtimes

- each folder of code has a requirements.in file with top-level dependencies, and a requirements.txt as a lock file

- you can either add a shebang line for your script which explicitly invokes the venv's Python and link that file into ~/.local/bin/, or instead create an external launcher script in ~/.local/bin/.

---

Here's an example of how that might be done using zpy functions:

  $ mkdir simple-scripts
  $ cd simple-scripts
  $ envin  # or in subcommand form: zpy envin
  ==> creating -> ~/.local/share/venvs/280…/venv :: ~/Code/simple-scripts

  $ pipacs httpx
  ==> appending -> requirements.in :: ~/Code/simple-scripts
  httpx
  ==> compiling requirements.in -> requirements.txt :: ~/Code/simple-scripts
  anyio==3.6.2              # via httpcore
  certifi==2022.12.7        # via httpcore, httpx
  h11==0.14.0               # via httpcore
  httpcore==0.16.3          # via httpx
  httpx==0.23.3             # via -r requirements.in
  idna==3.4                 # via anyio, rfc3986
  rfc3986==1.5.0            # via httpx
  sniffio==1.3.0            # via anyio, httpcore, httpx
  ==> syncing requirements.txt -> env :: ~/Code/simple-scripts

  $ print -rl -- 'from httpx import get' 'print(get("https://ifconfig.co/json").json())' >do_it.py

  $ vpyshebang do_it.py  # or: zpy vpyshebang do_it.py
  $ ln -s $PWD/do_it.py ~/.local/bin/do_it
If a Python runtime update breaks your venvs, you can probably fix things up using zpy's pipup function.

Re: How to improve Python packaging

#169

Note this post is 8700 words long and it doesn't really delve into how bad it is to build packages on Python; mostly it's about how bad it is to use them. It's an excellent writeup and the length is necessary. My guess is a post about creating packages would need to be 2x as long for the same level of detail.

I don't agree with all the points from the article, but I do agree there is a depth of learning to be had about creating packages (and doing it repeatably/scalably). I wrote a book about creating Python packages that just came out: https://pypackages.com

Even this book doesn't cover all options in each area, and it skips almost wholly over conda because I have no personal experience using it. conda and the work in the scientific community adds complexity both to the creation and the consumption side of packaging, and that's one area I'm not sure this post covers all the nuance of when considering how a "one size fits all" solution might work in practice.

Re: How to improve Python packaging

#170

Earlier quoted context omitted.

And one of the reasons I moved away from it! My blog post on the subject was posted here and it was very unpopular despite having similar points and also making comparisons with Node. There's nothing special about Python that makes it worth enduring this pain. We have other languages with a better developer UX.

I'd be keen to read your blog post if you have a link?

https://cedwards.xyz/breaking-up-with-python/

Bear in mind this was not written for an audience, let alone a HN audience.

Post reply on HN