Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How to improve Python packaging

chriswarrick.com

Post reply on HN
Print view View original

171–180 of 204 posts

Re: How to improve Python packaging

#171

Post by lmm »

gen220 wrote: post #116
imglorp wrote: post #18

Earlier quoted context omitted.

That's where we are as well, but yes the problems you didn't sidestep with poetry+docker are still there: the pypi fecosystem out there does not stop quantum fluctuations just because you want it to. If you pin too much you're not getting security fixes and if you don't pin enough, you get bitten every time a tertiary dep changes that you never asked for. Oh yeah and there are now trojans upstream almost daily.

Does any FOSS package ecosystem have a good solution to this particular set of problems? I know Go has historically done something interesting with selecting the "minimum viable version number" for each dependency, but that's the only relevant idea that comes to mind. With PyPi, at least it's relatively trivial to self-host a pypi instance and self-manage dependency updates. In the Python ecosystem (vs e.g. Node), at…

In the JVM ecosystem you get the version your immediate dependency was built/released against, transitive dependencies are never silently upgraded (though if you want to upgrade then it's one command). IME that's a much better default; it does mean that when a security issue is found in a low-level library you get everyone doing a bunch of releases that just bump their dependencies, but the consistent behaviour is very much worth it.

Re: How to improve Python packaging

#172

Post by greazy »

whatever1 wrote: post #144

Please allow me to vent on python packaging. I was working to debug a very weird error that was happening in my code only when my app was raising a warning through my `warnings.py`. My code was importing numpy, which in turn was trying to import its own `warnings.py`, but because of the python environment precedence, it was loading my own `warnings.py` which unfortunately had the same name. How on earth was this some…

How does 'warnings.py' work with the rest of the code? Intrigued as I've put myself into a corner with the way warnings/errors are handled in my project.

By the way I think there is a convention to handle these issues but it appears numpy isn't using it. If I remember correctly, any module with __ infront will be called {package}.__{module}.

Re: How to improve Python packaging

#173

Post by EdwardDiego »

c7DJTLrn wrote: post #170
EdwardDiego wrote: post #124

Earlier quoted context omitted.

I'd be keen to read your blog post if you have a link?

https://cedwards.xyz/breaking-up-with-python/ Bear in mind this was not written for an audience, let alone a HN audience.

Thanks, appreciate the link, I'll be back to yell at you shortly :D

...and having read it, the only thing I'd yell is that "I AGREE WITH ALL YOUR PAIN POINTS". Especially around documentation and package management.

I do like type annotations for improving the reading experience of Python codebases, even if you're not using a type checker, but trying to get people to use them consistently for new code in an org that has a large pre-existing codebase involves either a lot of carrots, or some CI based sticks, and I don't quite have the organisational influence to just drop that on people... ...yet.

Re: How to improve Python packaging

#174

Post by actinium226 »

randomluck040 wrote: post #148
actinium226 wrote: post #146

Glad to see this getting discussion. I hate virtualenvs for several of the reasons mentioned in the article - extra state to keep track of, and potentially installing packages to the wrong location if I failed to keep track of it. I always just force my user packages to work. I usually only use numpy/scipy/matplotlib and occasionally a few others so its not that hard, but some sort of npm like experience would be wel…

As someone who has next to no experience with npm, which benefits of npm would you like to see in venv or package management in Python?

I think the article describes it pretty well, it actually highlighted a few features of npm I wasn't familiar with. But to speak to my own experience, it's convenient that npm installs all the packages you need into a project-local node_modules folder. This eliminates the need for venv.

Mostly I just want to see venv eliminated. I really don't like the workflow it requires.

Re: How to improve Python packaging

#175

Post by abraxas »

charliermarsh wrote: post #59

> There are two tools for dealing with packages in the Node world, namely npm and Yarn. Not to mention pnpm. Or Bun. Or even Deno. All of which deal with packages, and make different choices. This isn't to be a pedant, but I think it's a reasonable example of why it's less important that there's a single way to do a thing, or even a single tool vs. multiple alternatives; instead, what I care about is that I don't hav…

Well, no. The JS package ecosystem is a slightly lesser mess than the Python ecosystem but only slightly so. I think it's a bad example to base any redesign off of. At least the author brought up .NET which mostly has its shit together when it comes to builds and packaging.

Re: How to improve Python packaging

#176

Post by abraxas »

c7DJTLrn wrote: post #88
sam345 wrote: post #76

The main reason I have stayed away from Python. packaging is a mess!

And one of the reasons I moved away from it! My blog post on the subject was posted here and it was very unpopular despite having similar points and also making comparisons with Node. There's nothing special about Python that makes it worth enduring this pain. We have other languages with a better developer UX.

Well, not for ML we don't. Python killed it all off. If you plan to work on core algorithms or finetuning existing models there is nowhere to run.

Re: How to improve Python packaging

#177

Post by u801e »

LtWorf wrote: post #56
matsemann wrote: post #7

Because of the insanity of python I run everything in Docker (through compose). No more issues with it not working on some dev's computer because of a missing wheel, or that they need to have X and Y c++ toolchain packages installed locally etc. No more trying to fix a broken setup after upgrading python or poetry versions, just "docker compose build" and you're up and running. No spending days getting a freshly clon…

.deb packages…

On that note, why are we trying to use language specific packaging tools to build packages rather than just building OS specific packages where dependencies are handled by apt for deb packages or dnf for rpm packages?

These are language agnostic and can get the job done.

Re: How to improve Python packaging

#178

Post by uranusjr »

benreesman wrote: post #33

Trying to build Python under Nix (i.e. solid, not “close enough) is an education in how fucked up the Python packaging ecosystem is. Which is a shame because Python is an amazing scripting language, the de facto numerical computing standard, and probably a better bash for most bash use cases.

One of the problems with Python packaging is Python Core Devs view “Python packaging” and “packaging Python” as two different things, but most people don’t bother with (or can’t tell) the difference :)

Re: How to improve Python packaging

#179

Post by uranusjr »

nezirus wrote: post #92

Reading this is really depressing. I just want Cargo for Python. Poetry it is for now, but it has quirks, and it is dog slow...

A lot of people said they want Cargo for Python but immediately backed out when you require them to `cargo run` their program. They want `python myscript.py` to still magically work, but that’s exactly where a lot of the magic comes from. Running `python` directly is like manually invoking `rustc` (or maybe slightly more automated like a make script); it works, but is on an entirely different layer of abstraction and doesn’t fit well with the Cargo one.

I’m not trying to imply you’re one of those people, but this may provide some insight why “Cargo for Python” is not a more widely adopted workflow.

Re: How to improve Python packaging

#180

Post by LtWorf »

groestl wrote: post #98
LtWorf wrote: post #56

Earlier quoted context omitted.

.deb packages…

Do you happen to have a link to a python application that ships it's own dependencies and has state-of-the art debian packaging?

> that ships it's own dependencies

You mean a link to a malformed .deb packages?

It's not difficult to do but it's also the wrong way to do it.

Post reply on HN