Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How to improve Python packaging

chriswarrick.com

Post reply on HN
Print view View original

91–100 of 204 posts

Re: How to improve Python packaging

#91

Post by woodruffw »

baq wrote: post #87
agumonkey wrote: post #72

Earlier quoted context omitted.

where is the install .[dev] format/syntax defined ? I was trying to find what was possible and how to make sense of it in setup.cfg tools I found one mention in the docs but no more.

you may have done pip install jupyter[notebook] this is the same thing, just for . (current directory) and dev variant.

Yep. The key difference is that the former is specified in PEP 508, while `.` and its variants are a convenience feature that pip (and maybe some other tools) provide.

Re: How to improve Python packaging

#93

Post by noisy_boy »

DangitBobby wrote: post #78
saila wrote: post #75

Earlier quoted context omitted.

If you don't mind adding a pyproject.toml, you could use pipx[1] to install these scripts. The directory structure would look like this: / pyproject.toml .py The pyproject.toml would look like this (using poetry, but you could use a different tool for this): [tool.poetry] # NOTE: Set to your project name name = " " description = "My Script" version = "0.0.0" authors = ["Me "] [tool.poetry.dependencies] python = "^3.1…

Thank you for trying to provide a workable solution, it's really not bad, but it has some downsides for me. pipx itself is installed inside a python environment, so when brew breaks my pythons, it breaks pipx as well. Anytime brew breaks my pythons, I would need to do the install step again for every script (or write a tool myself which does it). Not a total deal breaker, but not really much better than my current si…

One option could be to have Python installed in a separate location specifically for this purpose and to NOT include it in PATH. Then it is "out-of-sight" of brew and such packages and sort. You can even make the entire location read-only once you are done with installation of Python + pipx etc.

Re: How to improve Python packaging

#94

Post by orf »

vlovich123 wrote: post #67
jasonhansel wrote: post #14

Over time, I've grown to appreciate "pip-tools." Since it's a dead-simple extension of pip, I wish it could be upstreamed into pip itself; that seems like the most straightforward way of fixing a number of Python's packaging issues.

Yup. I like to make it so that each executable has a symlink to a shell script. The shell script checks if there’s a virtual environment set up with the packages in the requirements.txt installed (it takes a snapshot of the file because virtualenv doesn’t have a DB to query cheaply). Once the environment is set up, it dispatches to running from the virtualenv. That way when you update a requirements.in file it recomp…

You’ve described a worse, homegrown version of poetry.

Re: How to improve Python packaging

#95

Post by groestl »

matsemann wrote: post #7

Because of the insanity of python I run everything in Docker (through compose). No more issues with it not working on some dev's computer because of a missing wheel, or that they need to have X and Y c++ toolchain packages installed locally etc. No more trying to fix a broken setup after upgrading python or poetry versions, just "docker compose build" and you're up and running. No spending days getting a freshly clon…

I secretly believe Docker only exists because of how f'up Python's distribution story is.

Re: How to improve Python packaging

#96

Post by nprescott »

agumonkey wrote: post #72
woodruffw wrote: post #11

It's not very well documented, but the PyPA tools do provide a unified experience when used correctly. Here's a PyPA project (FD: one I work on) that uses a single pyproject.toml to handle all aspects of packaging (and most non-packaging tool configuration, to boot)[1]. With a single file like that, the only thing you need to do to start a local development environment is: python -m venv env && . env/bin/activate pyt…

where is the install .[dev] format/syntax defined ? I was trying to find what was possible and how to make sense of it in setup.cfg tools I found one mention in the docs but no more.

PEP-508[0] explains the grammar for "extras":

> Optional components of a distribution may be specified using the extras field:

  identifier_end = letterOrDigit | (('-' | '_' | '.' )* letterOrDigit)
  identifier    = letterOrDigit identifier_end*
  name          = identifier
  extras_list   = identifier (wsp* ',' wsp* identifier)*
  extras        = '[' wsp* extras_list? wsp* ']'
as well as explaining their behavior, albeit briefly:

> Extras union in the dependencies they define with the dependencies of the distribution they are attached to.

The resolution on . is explained by the pip documentation[1]:

> pip looks for packages in a number of places: on PyPI (if not disabled via --no-index), in the local filesystem, and in any additional repositories specified via --find-links or --index-url. There is no ordering in the locations that are searched. Rather they are all checked, and the “best” match for the requirements (in terms of version number - see PEP 440 for details) is selected.

[0]: https://peps.python.org/pep-0508/#grammar

[1]: https://pip.pypa.io/en/stable/cli/pip_install/#finding-packa...

Re: How to improve Python packaging

#97

Post by DangitBobby »

DangitBobby wrote: post #51
DangitBobby wrote: post #31

I have a single, simple script (not a package!) that has dependencies. Actually, I have a few of these, just sitting in /usr/local/bin so I can execute them whenever. How should I be managing environments for these scripts? Do I install dependencies in shared system python? Should I create a shared venv? Where should I store it? Any tools out there that make this decision for you and manage it? Just the fact that hom…

In an amazing coincidence, I just found another front-page post exploring this same topic! For Python, it seems to suggest using nix-shell. 1. https://dbohdan.com/scripts-with-dependencies

My comment with an example of a single file script which can specify its own dependencies:

https://news.ycombinator.com/item?id=34393630

Re: How to improve Python packaging

#98

Post by groestl »

LtWorf wrote: post #56
matsemann wrote: post #7

Because of the insanity of python I run everything in Docker (through compose). No more issues with it not working on some dev's computer because of a missing wheel, or that they need to have X and Y c++ toolchain packages installed locally etc. No more trying to fix a broken setup after upgrading python or poetry versions, just "docker compose build" and you're up and running. No spending days getting a freshly clon…

.deb packages…

Do you happen to have a link to a python application that ships it's own dependencies and has state-of-the art debian packaging?

Re: How to improve Python packaging

#99

Post by marsten »

samwillis wrote: post #13

This post is prompted by the survey of Python users, their feedback, and a current thread [0] on the Python forms discussing a way forward. I read the thread the other day, it's long, and there are a lot of opinions. Towards the end of this post there is an interesting observation: > Discourse, the platform that the discussion was held on, shows the number of times a link was clicked. Granted, this count might not be…

> there will likely be a higher than average representation from people who would sit in the 10%

And also an under-representation of "average users".

Python the language has always benefitted from its relative simplicity, which I attribute to people like GVR saying "no" to specialized features that accrue on languages like barnacles on a ship (looking at you C++).

With newer languages we see core design teams being much more opinionated about tooling for build and dependendency/packaging management. This is no doubt in response to the dumpster fires of C++ and Python, where "the ecosystem will sort itself out" clearly hasn't worked.

Re: How to improve Python packaging

#100

Post by egberts1 »

What is clearly still missing on python.org website is the obvious edit-code-run-debug step-by-step HOWTO for a typical tight but circular Python development iterations, complete with standardized packaging approach.
Post reply on HN