Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How to improve Python packaging

chriswarrick.com

Post reply on HN
Print view View original

31–40 of 204 posts

Re: How to improve Python packaging

#31

Post by DangitBobby »

I have a single, simple script (not a package!) that has dependencies. Actually, I have a few of these, just sitting in /usr/local/bin so I can execute them whenever.

How should I be managing environments for these scripts? Do I install dependencies in shared system python? Should I create a shared venv? Where should I store it? Any tools out there that make this decision for you and manage it?

Just the fact that homebrew occasionally updates my installed Pythons and breaks everything makes me reluctant to use Python as a scripting language at all. It's pretty reliable when I have a folder dedicated to a project (poetry is good for this) but I'm pretty fed up with how brittle it all is.

Re: How to improve Python packaging

#32

Post by Denvercoder9 »

dataflow wrote: post #4

Regarding system package upgrades breaking virtual environments: would hard links address this? If the system package manager removes a file, your venv still has a hard link to it so it doesn't see a difference.

It would also prevent the Python in your venv from receiving any updates when the system-wide Python is installed. At that point it's probably easier to just use your own Python installation, e.g. with pyenv.

Re: How to improve Python packaging

#33

Post by benreesman »

Trying to build Python under Nix (i.e. solid, not “close enough) is an education in how fucked up the Python packaging ecosystem is.

Which is a shame because Python is an amazing scripting language, the de facto numerical computing standard, and probably a better bash for most bash use cases.

Re: How to improve Python packaging

#34

Post by oconnor663 »

nixpulvis wrote: post #23
Lyngbakr wrote: post #21

It seems most people agree that Python's packaging isn't great, but, conversely, is there a language where most people agree that the approach to packaging is awesome? I mean, what's the gold standard to aspire to?

Ruby gems and Cargo are pretty awesome. Also, I find I like Arch linux's Pacman quite a bit, though that's a slightly different use case where versioning isn't resolved.

Cargo is a gold standard imho, but there are definitely some simplifying decisions it makes that might not fit Python:

- Dependencies and executables can run arbitrary code during build. For example, Cargo knows almost nothing about how to build C code, and the common workflow is to pull in the popular `cc` library for this and call it in your build.rs.

- There's mostly no such thing as "installing a library", and each project builds its dependencies from source. This is baked in at the Cargo level (the install command just doesn't work for library crates) and also at the language level (no stable ABI besides "extern C").

- Related to that, the final product of a Cargo workflow is a mostly-statically-linked binary, so there isn't really an equivalent to virtualenv.

Re: How to improve Python packaging

#35

Post by dataflow »

Denvercoder9 wrote: post #32
dataflow wrote: post #4

Regarding system package upgrades breaking virtual environments: would hard links address this? If the system package manager removes a file, your venv still has a hard link to it so it doesn't see a difference.

It would also prevent the Python in your venv from receiving any updates when the system-wide Python is installed. At that point it's probably easier to just use your own Python installation, e.g. with pyenv.

Sure but installing a new copy would defeat the space savings. The goal of this was to get the desired effect without wasting as much space. I wasn't trying to optimize purely for ease of use.

Put another way, my suggestion was to migrate from symbolic links to hard links.

Re: How to improve Python packaging

#36

Post by Kwpolska »

rektide wrote: post #6

Cant resist digging at Node.js even when writing up how infinitely better Node.js is at dealing with packages than python, haha: > Let’s try removing is-odd to demonstrate how badly designed this package is: You literally just deleted is-even 's dependency on is-odd then have the audacity to be shocked that it broke? There's a lot of hatred for the small package philosophy of node.js, but it's also a huge win, stands…

There’s a place for small packages, but is-even/is-odd is a bit too small to be a reasonable package. It is far easier to just write `x % 2 === 0` inline, which is an obvious idiom, instead of installing and importing a separate package for this. The use of is-odd by is-even can be confusing for users. For example, you may call isEven(0.5) and get the following error:

RangeError: is-odd expects an integer. at isOdd (/tmp/mynodeproject/node_modules/is-odd/index.js:17:11) at isEven (/tmp/mynodeproject/node_modules/is-even/index.js:13:11) at Object. (/tmp/mynodeproject/index.js:3:13)

(But the main point of the demonstration was to showcase dependency resolution and where it looks for packages.)

Re: How to improve Python packaging

#37

Post by woodruffw »

Denvercoder9 wrote: post #30
woodruffw wrote: post #11

It's not very well documented, but the PyPA tools do provide a unified experience when used correctly. Here's a PyPA project (FD: one I work on) that uses a single pyproject.toml to handle all aspects of packaging (and most non-packaging tool configuration, to boot)[1]. With a single file like that, the only thing you need to do to start a local development environment is: python -m venv env && . env/bin/activate pyt…

That repository doesn't seem to pin dependency versions. How do you integrate that in this workflow?

You could use `pip-compile` if you want full pinning. That's what we do on another project -- we use GitHub Actions with `pip-compile` to provide a fully frozen copy of the dependency tree for users who'd like that[1].

In the context of `pip-audit`, that makes a little less sense: most of our dependencies are semantically versioned, and we'd rather users receive patches and fixes to our subdependencies automatically, rather than having to wait for us to release a corresponding fix version. Similarly, we expect users to install `pip-audit` into pre-existing virtual environments, meaning that excessive pinning will produce overly conservative dependency conflict errors.

[1]: https://github.com/sigstore/sigstore-python/tree/main/instal...

Re: How to improve Python packaging

#38

Post by Kwpolska »

Lyngbakr wrote: post #21

It seems most people agree that Python's packaging isn't great, but, conversely, is there a language where most people agree that the approach to packaging is awesome? I mean, what's the gold standard to aspire to?

I don’t think there’s much complaining in the C#/.NET ecosystem. Sure, there are alternate tools (FAKE for F#), but I believe most people are happy enough with dotnet.exe/msbuild/nuget.

Re: How to improve Python packaging

#39

Post by mkoubaa »

samwillis wrote: post #13

This post is prompted by the survey of Python users, their feedback, and a current thread [0] on the Python forms discussing a way forward. I read the thread the other day, it's long, and there are a lot of opinions. Towards the end of this post there is an interesting observation: > Discourse, the platform that the discussion was held on, shows the number of times a link was clicked. Granted, this count might not be…

I always like to say that's barring divine inspiration, it's impossible to make the right choice. It's far more possible to make a choice and then work hard to make it right

Re: How to improve Python packaging

#40

Post by Izkata »

qbasic_forever wrote: post #15
woodruffw wrote: post #11

It's not very well documented, but the PyPA tools do provide a unified experience when used correctly. Here's a PyPA project (FD: one I work on) that uses a single pyproject.toml to handle all aspects of packaging (and most non-packaging tool configuration, to boot)[1]. With a single file like that, the only thing you need to do to start a local development environment is: python -m venv env && . env/bin/activate pyt…

Nowhere in pypa documentation is your simple workflow described or mentioned. Instead it's a jumble of links to a myriad of tools including hatchling, flit, pdm, etc. and basically just a shoulder shrug, 'I don't know, figure it all out yourself' message. This article makes a great point that the current pypa 'guidance' is too confusing and vague for actual end users (i.e. people that don't work directly on pypa or h…

AFAIK in the python 2 era it was the de-facto standard (just swap "python -m venv" with "virtualenv"). All these new tools have just made it more complicated and it's not clear to me what they gain, seems to me like they've simply succeeded in convincing people it's complicated by obscuring what's actually going on.
Post reply on HN