Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How to improve Python packaging

chriswarrick.com

Post reply on HN
Print view View original

61–70 of 204 posts

Re: How to improve Python packaging

#61

Post by wpietri »

DangitBobby wrote: post #31

I have a single, simple script (not a package!) that has dependencies. Actually, I have a few of these, just sitting in /usr/local/bin so I can execute them whenever. How should I be managing environments for these scripts? Do I install dependencies in shared system python? Should I create a shared venv? Where should I store it? Any tools out there that make this decision for you and manage it? Just the fact that hom…

I don't understand how there still isn't a good answer for this. It seems like such an obvious need, and for years I've heard go being promoted for solving this problem. I get why Python didn't start out with an answer for this, but in an era of 20 TB hard drives, I'm more than willing to burn space for ease of install.

Re: How to improve Python packaging

#62

Post by npage97 »

tempest_ wrote: post #12

I've pretty much settled on poetry for most things at this point. It still has a ton of rough edges like error messages still closer to a stack trace than something actionable and it's slow as hell resolving but it does the job 90 percent of the time without issue. Also running everything in docker now anyway which side steps some other problems.

For anyone curious, the reasoning for the slowness is briefly described here: https://python-poetry.org/docs/faq/#why-is-the-dependency-re...

Re: How to improve Python packaging

#63

Post by odiroot »

tempest_ wrote: post #12

I've pretty much settled on poetry for most things at this point. It still has a ton of rough edges like error messages still closer to a stack trace than something actionable and it's slow as hell resolving but it does the job 90 percent of the time without issue. Also running everything in docker now anyway which side steps some other problems.

Poetry still doesn't seem to support PEP621, instead requiring its custom, vendor-specific shape of pyproject.toml.

Re: How to improve Python packaging

#65

Post by goodpoint »

matsemann wrote: post #7

Because of the insanity of python I run everything in Docker (through compose). No more issues with it not working on some dev's computer because of a missing wheel, or that they need to have X and Y c++ toolchain packages installed locally etc. No more trying to fix a broken setup after upgrading python or poetry versions, just "docker compose build" and you're up and running. No spending days getting a freshly clon…

No way. Docker is not designed for security.

Re: How to improve Python packaging

#66

Post by fleekonpoint »

DangitBobby wrote: post #31

I have a single, simple script (not a package!) that has dependencies. Actually, I have a few of these, just sitting in /usr/local/bin so I can execute them whenever. How should I be managing environments for these scripts? Do I install dependencies in shared system python? Should I create a shared venv? Where should I store it? Any tools out there that make this decision for you and manage it? Just the fact that hom…

I use a separate directory and venv for each script. To execute the script, I use a shell script to call the venv's python interpreter. This is also how I use python scripts with cron/systemd.

    #!/bin/bash
    # myscript.sh
    venv/bin/python3 myscript.py
You could also skip the shell script and use aliases in your .bashrc.

Re: How to improve Python packaging

#67

Post by vlovich123 »

jasonhansel wrote: post #14

Over time, I've grown to appreciate "pip-tools." Since it's a dead-simple extension of pip, I wish it could be upstreamed into pip itself; that seems like the most straightforward way of fixing a number of Python's packaging issues.

Yup. I like to make it so that each executable has a symlink to a shell script.

The shell script checks if there’s a virtual environment set up with the packages in the requirements.txt installed (it takes a snapshot of the file because virtualenv doesn’t have a DB to query cheaply). Once the environment is set up, it dispatches to running from the virtualenv.

That way when you update a requirements.in file it recompiles it (if the txt is out of date) and installs any new packages, removes packages that shouldn’t be there anymore and updates ones whose version changed (if there’s any changes found). It also lets you trivially run tools with disparate requirements.in without conflicts because each is siloed behind its own virtualenv.

This makes it a trivial experience to use these tools in a shared repo because there’s no worrying about packages / needing to remember to run some command before the right environment is set up. You just modify your code and run it like a regular command-line tool and packages automatically get deployed. It’s also amenable to offline snapshotting / distribution. In fact, I used this to distribute support tooling for factory lines of the Pixel Buds and it worked extremely well.

Re: How to improve Python packaging

#69

Post by DangitBobby »

fleekonpoint wrote: post #66
DangitBobby wrote: post #31

I have a single, simple script (not a package!) that has dependencies. Actually, I have a few of these, just sitting in /usr/local/bin so I can execute them whenever. How should I be managing environments for these scripts? Do I install dependencies in shared system python? Should I create a shared venv? Where should I store it? Any tools out there that make this decision for you and manage it? Just the fact that hom…

I use a separate directory and venv for each script. To execute the script, I use a shell script to call the venv's python interpreter. This is also how I use python scripts with cron/systemd. #!/bin/bash # myscript.sh venv/bin/python3 myscript.py You could also skip the shell script and use aliases in your .bashrc.

I do something kind of like this, but all of my scripts break when the underlying env is suddenly broken when e.g. brew updates python without asking me and breaks all existing environments.

I'm sure I could come up with solutions that are very robust for my particular machine, but I would like something that allows me to share it as a gist and a teammate could just as easily use it, or I can use it myself on another machine without hassle. In other words, a solution contained within the script iteself and maybe one or two binaries outside that make it possible.

Re: How to improve Python packaging

#70

Post by wheelerof4te »

matsemann wrote: post #7

Because of the insanity of python I run everything in Docker (through compose). No more issues with it not working on some dev's computer because of a missing wheel, or that they need to have X and Y c++ toolchain packages installed locally etc. No more trying to fix a broken setup after upgrading python or poetry versions, just "docker compose build" and you're up and running. No spending days getting a freshly clon…

When the solution to basic problem like this is "use Docker", you realise how deeply flawed modern software development (in Python, at least) is.
Post reply on HN