Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How to improve Python packaging

chriswarrick.com

Post reply on HN
Print view View original

111–120 of 204 posts

Re: How to improve Python packaging

#111

Post by gen220 »

phailhaus wrote: post #52
kdazzle wrote: post #19

Earlier quoted context omitted.

I’m biased, but I do like Python’s package management much better than Node. Even just regular old virtualenvs. Can’t tell you how many times deleting the node packages dir and reinstalling fixes a weird issue, but that’s happened very rarely for me on the python side. Also, having to comb through a bunch of node packages of dubious quality to find some sort of standard approach happens way too often. Like take pytho…

I'd say that Node's package management has run laps around Python's, to the point where it's pretty embarrassing. Since it's relatively new, it was able to hit the ground running with best practices: 1. Declarative package manifests. Python's ecosystem is still a mess of various approaches, and the fact that you have to run the setup.py script to determine dependencies is a nightmare. Because of this, running depende…

Respectfully, this reads like you're using outdated Python tools.

Give Poetry [1] a shot, it has all the things you've listed here. Just as Node.js has come a long way in the last 5 years, Python has, too. Albeit, in a fashion that was much less centralized, arguably to Python's detriment.

[1]: https://python-poetry.org/history/#100---2019-12-12, released 1.0 in 2019/12.

Re: How to improve Python packaging

#112

Post by Godel_unicode »

samwillis wrote: post #71
woodruffw wrote: post #11

It's not very well documented, but the PyPA tools do provide a unified experience when used correctly. Here's a PyPA project (FD: one I work on) that uses a single pyproject.toml to handle all aspects of packaging (and most non-packaging tool configuration, to boot)[1]. With a single file like that, the only thing you need to do to start a local development environment is: python -m venv env && . env/bin/activate pyt…

Ultimately it needs to be "Python.org" that endorses the tool, not PyPA, no one in the scheme of things know who PyPA are and if it's the "one true way". If you go to Python.org and follow through to the beginners guide [0] this is what's suggested: > There are several methods to install additional Python packages: > Packages can be installed via the standard Python distutils mode (python setup.py install). > Many pa…

The future is here, it’s just inconsistently distributed. There’s also the problem of python.org not wanting to pick winners, i.e. not promoting activestate vs conda vs pypa.

https://packaging.python.org/en/latest/tutorials/installing-...

https://packaging.python.org/en/latest/tutorials/packaging-p...

Re: How to improve Python packaging

#113

Post by nmstoker »

noisy_boy wrote: post #89
samwillis wrote: post #13

This post is prompted by the survey of Python users, their feedback, and a current thread [0] on the Python forms discussing a way forward. I read the thread the other day, it's long, and there are a lot of opinions. Towards the end of this post there is an interesting observation: > Discourse, the platform that the discussion was held on, shows the number of times a link was clicked. Granted, this count might not be…

> Ultimately they need one system that is front and centre, that is the "officially endorsed" tool, and very clear messaging on that. I think this is important; otherwise there is a risk behind switching to a new packaging tool and then it being subject to neglect/lack of resources - that makes people averse to switching to it. That is why virtualenv/pip is the lowest common denominator - everyone knows that worst ca…

Yes, there was some questionable comms around pipenv being "officially recommdnded" which upon closer inspection seemed to have come from the people who wrote it and not really been that official so far as I could see! That seems to have been walked back after a while but not before it gained traction and randoms bring up that it's official even now.

Re: How to improve Python packaging

#114

Post by TigerTeamX »

I must admit I just gave up on all these tools. Instead I just do

    pip install -r requirements.txt -t .lib
and I have

    import sys
    import os

    sys.path.append(f"{os.getcwd()}/.lib")
in the top of my script. Some will tell me this is silly, but it just works. Rememer to at .lib to your .gitignore. Else you'll have lot of fun.

Re: How to improve Python packaging

#115

Post by TigerTeamX »

TigerTeamX wrote: post #114

I must admit I just gave up on all these tools. Instead I just do pip install -r requirements.txt -t .lib and I have import sys import os sys.path.append(f"{os.getcwd()}/.lib") in the top of my script. Some will tell me this is silly, but it just works. Rememer to at .lib to your .gitignore. Else you'll have lot of fun.

This is similar to how node is doing. I just basically copied that idea.

Re: How to improve Python packaging

#116

Post by gen220 »

imglorp wrote: post #18
tempest_ wrote: post #12

I've pretty much settled on poetry for most things at this point. It still has a ton of rough edges like error messages still closer to a stack trace than something actionable and it's slow as hell resolving but it does the job 90 percent of the time without issue. Also running everything in docker now anyway which side steps some other problems.

That's where we are as well, but yes the problems you didn't sidestep with poetry+docker are still there: the pypi fecosystem out there does not stop quantum fluctuations just because you want it to. If you pin too much you're not getting security fixes and if you don't pin enough, you get bitten every time a tertiary dep changes that you never asked for. Oh yeah and there are now trojans upstream almost daily.

Does any FOSS package ecosystem have a good solution to this particular set of problems?

I know Go has historically done something interesting with selecting the "minimum viable version number" for each dependency, but that's the only relevant idea that comes to mind.

With PyPi, at least it's relatively trivial to self-host a pypi instance and self-manage dependency updates.

In the Python ecosystem (vs e.g. Node), at least the total set of first+second-order dependencies for most normal projects is quite small (e.g. medium double digits to low triple digits). It doesn't feel too painful to manage that magnitude of upgrades on a monthly or quarterly basis.

Re: How to improve Python packaging

#117

Post by agumonkey »

woodruffw wrote: post #73
agumonkey wrote: post #72

Earlier quoted context omitted.

where is the install .[dev] format/syntax defined ? I was trying to find what was possible and how to make sense of it in setup.cfg tools I found one mention in the docs but no more.

It’s a pip-ism; as far as I know, it’s not defined in any PEP. It should be in their documentation, however.

gonna dig deeper there then, thanks

Re: How to improve Python packaging

#118

Post by gen220 »

odiroot wrote: post #63
tempest_ wrote: post #12

I've pretty much settled on poetry for most things at this point. It still has a ton of rough edges like error messages still closer to a stack trace than something actionable and it's slow as hell resolving but it does the job 90 percent of the time without issue. Also running everything in docker now anyway which side steps some other problems.

Poetry still doesn't seem to support PEP621, instead requiring its custom, vendor-specific shape of pyproject.toml.

Have they spoken about why they don't support it? Or plans to support it in the future?

It seems like the shortest path to a universal package management tool for Python is to add this capability to Poetry, rather than to build something new.

Re: How to improve Python packaging

#119

Post by carapace »

woodruffw wrote: post #81
carapace wrote: post #58

Earlier quoted context omitted.

I think you're right. At risk of making a "lowbrow dismissal" the very length of TFA is a clear symptom of the problem: Design-by-committee (The PyPA (Python Package Authority) reminds me of the movie Brazil . Don't get me started.) - - - - Someone should do for packaging what pathlib did for file I/O: make a Pythonic model/API for it. Then deployment becomes simple, scriptable, testable, repeatable, etc...

This might be a fun thought to entertain, but the PyPA doesn't really form design committees (at least, I haven't been on one). You can see exactly how Python packaging standards are made: they're done with PEPs[1], exactly the same as with every other Python standardization effort. Indeed, most packaging PEPs start out exactly the way you've laid out: a tool or service writes a Pythonic API or model, and it gets sta…

> Don't get me started.

Re: How to improve Python packaging

#120

Post by phailhaus »

gen220 wrote: post #111
phailhaus wrote: post #52

Earlier quoted context omitted.

I'd say that Node's package management has run laps around Python's, to the point where it's pretty embarrassing. Since it's relatively new, it was able to hit the ground running with best practices: 1. Declarative package manifests. Python's ecosystem is still a mess of various approaches, and the fact that you have to run the setup.py script to determine dependencies is a nightmare. Because of this, running depende…

Respectfully, this reads like you're using outdated Python tools. Give Poetry [1] a shot, it has all the things you've listed here. Just as Node.js has come a long way in the last 5 years, Python has, too. Albeit, in a fashion that was much less centralized, arguably to Python's detriment. [1]: https://python-poetry.org/history/#100---2019-12-12 , released 1.0 in 2019/12.

Oh yeah, I've used Poetry. I'm talking about Python out of the box. The dependency management ecosystem is super fractured and relies on 3rd party tools developing their own standards, and even then, they can't overcome fundamental limitations. For example, Poetry is great but locking/installation still takes way longer than Node because there are no declarative manifests.
Post reply on HN