Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How to improve Python packaging

chriswarrick.com

Post reply on HN
Print view View original

121–130 of 204 posts

Re: How to improve Python packaging

#121

Post by jessekv »

wheelerof4te wrote: post #70
matsemann wrote: post #7

Because of the insanity of python I run everything in Docker (through compose). No more issues with it not working on some dev's computer because of a missing wheel, or that they need to have X and Y c++ toolchain packages installed locally etc. No more trying to fix a broken setup after upgrading python or poetry versions, just "docker compose build" and you're up and running. No spending days getting a freshly clon…

When the solution to basic problem like this is "use Docker", you realise how deeply flawed modern software development (in Python, at least) is.

Not sure I agree with "deeply flawed" verdict.

The flaws of python's packaging system (as well as the GIL and other hangups) emerge from the tradeoffs of inventing a language optimised for reusing and connecting as many disparate system binaries as possible.

Its not surprising that such a server scripting language is tightly coupled with the server environment that it runs in. Docker is just one way to ensure a reproducible server environment across machines.

What do you think it could have done differently?

Re: How to improve Python packaging

#122

Post by lmm »

woodruffw wrote: post #11

It's not very well documented, but the PyPA tools do provide a unified experience when used correctly. Here's a PyPA project (FD: one I work on) that uses a single pyproject.toml to handle all aspects of packaging (and most non-packaging tool configuration, to boot)[1]. With a single file like that, the only thing you need to do to start a local development environment is: python -m venv env && . env/bin/activate pyt…

That's still about 4 times as much command as I'd consider normal/reasonable, compared to say "mvn install".

Re: How to improve Python packaging

#123

Post by lmm »

jessekv wrote: post #121
wheelerof4te wrote: post #70

Earlier quoted context omitted.

When the solution to basic problem like this is "use Docker", you realise how deeply flawed modern software development (in Python, at least) is.

Not sure I agree with "deeply flawed" verdict. The flaws of python's packaging system (as well as the GIL and other hangups) emerge from the tradeoffs of inventing a language optimised for reusing and connecting as many disparate system binaries as possible. Its not surprising that such a server scripting language is tightly coupled with the server environment that it runs in. Docker is just one way to ensure a repro…

What could have been done differently:

- Make dependency resolution deterministic by default. Unfortunately the whole ecosystem has to buy into it, but the benefits are huge.

- Stop building castles of sand by making more layers of tooling that have to run in Python (and will therefore run in some random poorly managed Python). The language install can't manage the build system install - if anything the build system install should manage the language install. Adding pip to the language distribution was such a backwards decision that it marks the point where I gave up on Python ever fixing their stuff.

- Ignore Linux system package managers (apt etc.), they have a fundamentally broken model and will infect your language ecosystem with that breakage if you try to cater to them.

Re: How to improve Python packaging

#124

Post by EdwardDiego »

c7DJTLrn wrote: post #88
sam345 wrote: post #76

The main reason I have stayed away from Python. packaging is a mess!

And one of the reasons I moved away from it! My blog post on the subject was posted here and it was very unpopular despite having similar points and also making comparisons with Node. There's nothing special about Python that makes it worth enduring this pain. We have other languages with a better developer UX.

I'd be keen to read your blog post if you have a link?

Re: How to improve Python packaging

#125

Post by EdwardDiego »

Lyngbakr wrote: post #21

It seems most people agree that Python's packaging isn't great, but, conversely, is there a language where most people agree that the approach to packaging is awesome? I mean, what's the gold standard to aspire to?

After joining a company with a Python codebase, goddamn I miss Java so damn much.

For a single example, all packages in repositories are namespaced, and namespace ownership is verified. [0]

So there's no chance of typosquatting an existing package, nor do you have to worry about someone jumping in and claiming all the "good" names.

[0]: https://central.sonatype.org/publish/#individual-projects-op...

Re: How to improve Python packaging

#126

Post by mixmastamyk »

DangitBobby wrote: post #31

I have a single, simple script (not a package!) that has dependencies. Actually, I have a few of these, just sitting in /usr/local/bin so I can execute them whenever. How should I be managing environments for these scripts? Do I install dependencies in shared system python? Should I create a shared venv? Where should I store it? Any tools out there that make this decision for you and manage it? Just the fact that hom…

Put them in ~/.local/bin, and that in your path. Then pip install reqs with —user.

If you want this automated use pipx, but it is overkill and loses simplicity.

The reason folks give dire warnings against simplicity is that sysad skills have plummeted in recent years. It’s easy to fix the rare conflict by putting the troublemaker in a venv. Outside of a large work project I never need to.

Re: How to improve Python packaging

#127

Post by KyeRussell »

nmstoker wrote: post #113
noisy_boy wrote: post #89

Earlier quoted context omitted.

> Ultimately they need one system that is front and centre, that is the "officially endorsed" tool, and very clear messaging on that. I think this is important; otherwise there is a risk behind switching to a new packaging tool and then it being subject to neglect/lack of resources - that makes people averse to switching to it. That is why virtualenv/pip is the lowest common denominator - everyone knows that worst ca…

Yes, there was some questionable comms around pipenv being "officially recommdnded" which upon closer inspection seemed to have come from the people who wrote it and not really been that official so far as I could see! That seems to have been walked back after a while but not before it gained traction and randoms bring up that it's official even now.

I really hate to be this “inside baseball” and negative about it all, but I think that the Kenneth Reitz factor is…unique (within the Python community), and uniquely dangerous in these situations. I am so deeply fed up with the Python packaging experience. At this point, both my eyes are set on the fastest way to some sort of standardisation and improvement. Its because of this that I want to clearly acknowledge that IMO over the last 5 years we were thrown off course in no small part by one person’s lust for personal notoriety when the community clearly needed to bring their heads together on a solution. Pipenv was touted as the be all end all way too quickly, as you said not necessarily by ‘official sources’, and those shaky shaky years left everyone a bit battered.

When I did my every-12-months checkin to see how Python dependency management was going, I sure enough saw that Pipenv is ‘under’ PyPA now. I then learned that a PyPA endorsement or association. doesn’t mean a HUGE amount for the reasons that you and others have already noted. Sure enough, PyPA’s own packaging documentation is hesitant to strongly prefer a front end tool let alone any other part of the stack.

A significantly improved Python dependency management story could very well include Pipenv on the front end for all I care. But not acknowledging the elephant in the room risks repeating past mistakes. Lest we end up with more of the celebrity developer culture we see in communities like JS, but without the seemingly limitless effort and resources.

None of this is to understate the technical challenges, especially taking into account pulling together the current fragmented ecosystem.

Re: How to improve Python packaging

#128

Post by KyeRussell »

Godel_unicode wrote: post #112
samwillis wrote: post #71

Earlier quoted context omitted.

Ultimately it needs to be "Python.org" that endorses the tool, not PyPA, no one in the scheme of things know who PyPA are and if it's the "one true way". If you go to Python.org and follow through to the beginners guide [0] this is what's suggested: > There are several methods to install additional Python packages: > Packages can be installed via the standard Python distutils mode (python setup.py install). > Many pa…

The future is here, it’s just inconsistently distributed. There’s also the problem of python.org not wanting to pick winners, i.e. not promoting activestate vs conda vs pypa. https://packaging.python.org/en/latest/tutorials/installing-... https://packaging.python.org/en/latest/tutorials/packaging-p...

Which is just, utterly utterly ridiculous. The Python packaging story will continue to be a mess right up until this particular inconsistency is resolved.

Re: How to improve Python packaging

#129

Post by KyeRussell »

qbasic_forever wrote: post #53
woodruffw wrote: post #22

Earlier quoted context omitted.

I agree that it's confusing. That being said, there is an official PyPA tutorial that goes through the exact steps need to produce the commands I suggested, so I don't think it's accurate to say that it's nowhere to be found[1]. Edit: Unfortunately it's easy to confuse the above tutorial with this one[2], which is specifically for setuptools. So I can appreciate end user confusion around the documentation, particular…

No I'm looking at link 1 and specifically have an issue with the huge table of options for how to setup your pyproject.toml--it has tabs for hatchling, flit, pdm, etc. but _zero_ description of why I would want to use one of those tools. It's just more confusing, like you're being thrown a bag of parts and no description of what to build or how to do it. To be honest that entire pypa doc should be like two paragraphs…

Yep. There being no documentation to really empower someone to work out what to use, to me very directly says “there’s some political spat going on that I’m being exposed to here”.

Re: How to improve Python packaging

#130

Post by kelnage »

So, I have recently returned to Python development after several years out. In my first project [1], I was building on an existing library that was already using Poetry - so obviously I went with that for my work. Although it was a bit of a learning curve, I quickly got accustomed to it, but still wondered why it had come about given my recollections of the other tools (virtualenv et al.) being “good enough”.

Then, more recently, I had to run a different project that lacked any documentation as to how I was to run it, had a setup.py file, a Pipfile, and more in it. In trying to get this to run, I managed to make a real pigs ear such that (no doubt thanks to my lack of experience with those tools) I eventually had to delete all my virtual environments, as none of them worked anymore…

So yes, I am 100% in the “one tool to rule them all” camp these days - and although PDM does look promising, right now it isn’t offering me anything above Poetry that I care that strongly about.

As Cato might say, PyPA delenda est.

1. http://github.com/grafana/pySigma-backend-loki

Post reply on HN