Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How to improve Python packaging

chriswarrick.com

Post reply on HN
Print view View original

71–80 of 204 posts

Re: How to improve Python packaging

#71

Post by samwillis »

woodruffw wrote: post #11

It's not very well documented, but the PyPA tools do provide a unified experience when used correctly. Here's a PyPA project (FD: one I work on) that uses a single pyproject.toml to handle all aspects of packaging (and most non-packaging tool configuration, to boot)[1]. With a single file like that, the only thing you need to do to start a local development environment is: python -m venv env && . env/bin/activate pyt…

Ultimately it needs to be "Python.org" that endorses the tool, not PyPA, no one in the scheme of things know who PyPA are and if it's the "one true way".

If you go to Python.org and follow through to the beginners guide [0] this is what's suggested:

> There are several methods to install additional Python packages:

> Packages can be installed via the standard Python distutils mode (python setup.py install).

> Many packages can also be installed via the setuptools extension or pip wrapper, see https://pip.pypa.io/.

That is so out of date and fundamentally confuses people coming to Python for the first time. How is pip secondary, and no mention of venv?

The PyPA need to get buy in from Python Core to put one tool front and centre first. It needs to be like Rust with Cargo, literally the first thing you learn to use and core to all beginners guides.

That's not to diminish the work of PyPA, you are all amazing I just want your work to be more obvious!

0: https://docs.python.org/3/using/mac.html#installing-addition...

Re: How to improve Python packaging

#72

Post by agumonkey »

woodruffw wrote: post #11

It's not very well documented, but the PyPA tools do provide a unified experience when used correctly. Here's a PyPA project (FD: one I work on) that uses a single pyproject.toml to handle all aspects of packaging (and most non-packaging tool configuration, to boot)[1]. With a single file like that, the only thing you need to do to start a local development environment is: python -m venv env && . env/bin/activate pyt…

where is the

    install .[dev]
format/syntax defined ? I was trying to find what was possible and how to make sense of it in setup.cfg tools

I found one mention in the docs but no more.

Re: How to improve Python packaging

#73

Post by woodruffw »

agumonkey wrote: post #72
woodruffw wrote: post #11

It's not very well documented, but the PyPA tools do provide a unified experience when used correctly. Here's a PyPA project (FD: one I work on) that uses a single pyproject.toml to handle all aspects of packaging (and most non-packaging tool configuration, to boot)[1]. With a single file like that, the only thing you need to do to start a local development environment is: python -m venv env && . env/bin/activate pyt…

where is the install .[dev] format/syntax defined ? I was trying to find what was possible and how to make sense of it in setup.cfg tools I found one mention in the docs but no more.

It’s a pip-ism; as far as I know, it’s not defined in any PEP. It should be in their documentation, however.

Re: How to improve Python packaging

#75

Post by saila »

DangitBobby wrote: post #31

I have a single, simple script (not a package!) that has dependencies. Actually, I have a few of these, just sitting in /usr/local/bin so I can execute them whenever. How should I be managing environments for these scripts? Do I install dependencies in shared system python? Should I create a shared venv? Where should I store it? Any tools out there that make this decision for you and manage it? Just the fact that hom…

If you don't mind adding a pyproject.toml, you could use pipx[1] to install these scripts. The directory structure would look like this:

    /
        pyproject.toml
        .py
The pyproject.toml would look like this (using poetry, but you could use a different tool for this):

    [tool.poetry]
    # NOTE: Set  to your project name
    name = ""
    description = "My Script"
    version = "0.0.0"
    authors = ["Me "]

    [tool.poetry.dependencies]
    python = "^3.11"
    requests = "^2.28.2"

    [tool.poetry.scripts]
    # NOTE: Change  to match your package name in [tool.poetry]
     = ":main"

    [build-system]
    requires = ["poetry-core"]
    build-backend = "poetry.core.masonry.api"
Then you'd run `pipx install -e .` and the the executable script will be installed in ~/.local/bin.

[1] https://pypa.github.io/pipx/

Re: How to improve Python packaging

#77

Post by fleekonpoint »

DangitBobby wrote: post #69
fleekonpoint wrote: post #66

Earlier quoted context omitted.

I use a separate directory and venv for each script. To execute the script, I use a shell script to call the venv's python interpreter. This is also how I use python scripts with cron/systemd. #!/bin/bash # myscript.sh venv/bin/python3 myscript.py You could also skip the shell script and use aliases in your .bashrc.

I do something kind of like this, but all of my scripts break when the underlying env is suddenly broken when e.g. brew updates python without asking me and breaks all existing environments. I'm sure I could come up with solutions that are very robust for my particular machine, but I would like something that allows me to share it as a gist and a teammate could just as easily use it, or I can use it myself on another…

I see, it might be heavy handed but running them inside Docker containers might provide you with the isolation you're looking for. You could also build and share these images with your teammates.

I've actually started using a lot of different CLI tools with Docker, especially when the tool isn't available for my OS.

Re: How to improve Python packaging

#78

Post by DangitBobby »

saila wrote: post #75
DangitBobby wrote: post #31

I have a single, simple script (not a package!) that has dependencies. Actually, I have a few of these, just sitting in /usr/local/bin so I can execute them whenever. How should I be managing environments for these scripts? Do I install dependencies in shared system python? Should I create a shared venv? Where should I store it? Any tools out there that make this decision for you and manage it? Just the fact that hom…

If you don't mind adding a pyproject.toml, you could use pipx[1] to install these scripts. The directory structure would look like this: / pyproject.toml .py The pyproject.toml would look like this (using poetry, but you could use a different tool for this): [tool.poetry] # NOTE: Set to your project name name = " " description = "My Script" version = "0.0.0" authors = ["Me "] [tool.poetry.dependencies] python = "^3.1…

Thank you for trying to provide a workable solution, it's really not bad, but it has some downsides for me. pipx itself is installed inside a python environment, so when brew breaks my pythons, it breaks pipx as well. Anytime brew breaks my pythons, I would need to do the install step again for every script (or write a tool myself which does it). Not a total deal breaker, but not really much better than my current situation which pretty much just assumes any version of `requests` or `fire` is acceptable. Because python itself is constantly being updated to break the base python environment on my machine, a workable solution would need to include the fact that the base python environment might need to have things installed.

Re: How to improve Python packaging

#80

Post by Kwpolska »

oconnor663 wrote: post #34
nixpulvis wrote: post #23

Earlier quoted context omitted.

Ruby gems and Cargo are pretty awesome. Also, I find I like Arch linux's Pacman quite a bit, though that's a slightly different use case where versioning isn't resolved.

Cargo is a gold standard imho, but there are definitely some simplifying decisions it makes that might not fit Python: - Dependencies and executables can run arbitrary code during build. For example, Cargo knows almost nothing about how to build C code, and the common workflow is to pull in the popular `cc` library for this and call it in your build.rs. - There's mostly no such thing as "installing a library", and ea…

> - Dependencies and executables can run arbitrary code during build. For example, Cargo knows almost nothing about how to build C code, and the common workflow is to pull in the popular `cc` library for this and call it in your build.rs.

Python packaging involves arbitrary code execution, even today. While it’s less obvious than it was with `setup.py`, if you’re installing from source, the package can specify any build backend it wants, and that build backend can do anything. This could be mitigated by having an allowlist of build backends, but who would be responsible for vetting the tools, and how would it be guaranteed that an allowed tool is not taken over by someone hostile, and that an allowed tool does not have a system to run arbitrary package-provided code?

Post reply on HN