Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Thoughts on the Python packaging ecosystem

pradyunsg.me

Post reply on HN
Print view View original

91–100 of 121 posts

Re: Thoughts on the Python packaging ecosystem

#91

Post by 411111111111111 »

bombolo wrote: post #90
numbsafari wrote: post #82

Earlier quoted context omitted.

Any multi step process is guaranteed to produce random assortments of workflow tools to manage those steps which will seek to replace the original process as the one true process. If there two steps, one part of the community will insist they belong in: a makefile, bash script, python script, lambda network service, bazel, pants, scons, terraform, ansible, npm …

So we should have 1 single shell command to do literally everything?

before poetry and pipenv was a thing, pretty much everyone was recommending the virtualenvwrapper script instead of using venv directly... so yes, thats what it ultimetaly always ends up as.

Re: Thoughts on the Python packaging ecosystem

#92

Post by Hackbraten »

LtWorf wrote: post #44
lmm wrote: post #32

Earlier quoted context omitted.

Shipping with python is a disadvantage; any real workflow tool needs to be able to manage multiple python installations, and if you try to use a part of the python installation to do that you've got a circular dependency problem. Plus the whole "where modules go to die" thing.

If you have multiple versions of python installed you can just do python3.9 -m pip python3.10 -m pip The standard library is the reason why python is popular to begin with.

To me, that’d be a UX nightmare. Even if I tried to, I’m not going to memorize which of my 20 checked-out Python projects are requiring which Python invocation.

IMHO it’s absolutely the toolchain’s job to manage that for me. I’d never adopt a toolchain that wouldn’t.

Re: Thoughts on the Python packaging ecosystem

#93

Post by woodruffw »

dmr_92 wrote: post #81
meandmycode wrote: post #75

Earlier quoted context omitted.

Poetry ended up being mostly a disappointment having used it for the last 18 months. First time installs were eventually an hour long to resolve, and lock files were platform specific, making using them for reproducibility almost useless. The wider python ecosystem is frankly painful to productionize even today and it really does need some focus and governance.

Are you sure the lockfiles are platform-specific? I spent a long time migrating a project to use poetry. One of the reasons I opted for poetry over others was that the lockfile retained all of the environment markers in the packaging metadata, so that the lockfile could support multiple interpreters and interpreter versions, multiple platforms, etc.

I can't speak for Poetry directly, but knowing how Python dependency resolution works: I don't think Poetry can make lockfiles not platform-specific, since package source distributions are allowed to (and regularly do) run platform-specific code for their own dependency selection logic.

For example, your package might depend on `foo`, which in turn could sniff the host OS and select the appropriate subdependency. You'd then end up pinning that subdependency, which would be incorrect on a different host OS.

(Similarly for Python versions: a subdependency might be required on < 3.7, so re-installing from a lockfile generated from an older Python could produce a spurious runtime dependency.)

Re: Thoughts on the Python packaging ecosystem

#94

Post by numbsafari »

bombolo wrote: post #90
numbsafari wrote: post #82

Earlier quoted context omitted.

Any multi step process is guaranteed to produce random assortments of workflow tools to manage those steps which will seek to replace the original process as the one true process. If there two steps, one part of the community will insist they belong in: a makefile, bash script, python script, lambda network service, bazel, pants, scons, terraform, ansible, npm …

So we should have 1 single shell command to do literally everything?

Not what I would or wouldn’t do personally, but what happens generally in practice.

Re: Thoughts on the Python packaging ecosystem

#95

Post by Hackbraten »

Too wrote: post #87

I'm convinced 99% of all issues with pip can be solved with 3 simple aliases. It already has everything else, it just the UX that isn't opinionated enough. There wouldn't be a need for Pipenv, Poetry or yet another project-definition or lockfile format, if only these few fundamental commands with sane defaults were included out of the box. 1. Installation. Just install everything in a bloody application-specific virt…

> Now if you were to do this just a little more seriously its of course a bit more effort but still within the realms of weeks/months, not years.

My feeling is that that approach would cause dozens of itches to pop up. Things like, “my coworker is on Windows and I’m struggling to come up with step-by-step instructions for her to set up the project.” Those are valid UX issues but likely just the beginning of a rabbit hole.

I think that tackling those issues would inevitably lead to yet another Poetry/Pipenv clone – because with UX, the devil is in the details.

Re: Thoughts on the Python packaging ecosystem

#96

Post by wheelerof4te »

Kamq wrote: post #35
wheelerof4te wrote: post #21

There really needs to be one tool that combines installing, building, testing and running Python projects. This tool should be officially endorsed by core developers of Python and gradually added to the standard distribution model. The good starting point for this would be Pip. It is de facto standard tool for installing packages. It can be extended to do more. Also, the standard should favor pure-Python packages, in…

> Also, the standard should favor pure-Python packages, in order to untangle Python from it's legacy as a glue language for modules written in C(++). Never gonna happen. Python's explosion has been through data science and ml, which is based entirely around calling Fortran/C/C++/etc. > Otherwise, no progress will be made at the language level and Python will die out once C becomes legacy language, replaced by safer s…

"Never gonna happen. Python's explosion has been through data science and ml, which is based entirely around calling Fortran/C/C++/etc."

What's Django, then?

"I assume you mean Rust here. Do you think python can't already call rust? It's been able to do that for the better part of a decade."

Python can only call C code. The fact that Rust itself (or Fortran, btw) has good C ABI is on Rust, not Python. Also, by calling Rust's C ABI, you are making a double indirection, thus severely reducing performance.

That's why I said we need to optimize Python, and even make progress towards making Python - in Python itself. Just like Go can compile itself, Python should be able to run itself. But, that is all for the long run.

Re: Thoughts on the Python packaging ecosystem

#97

Post by Hackbraten »

solarkraft wrote: post #40

I don't know, I'm pretty happy with Pipenv. It's the standard among the people I talk to and very good at being "pip, but better". I've tried poetry but don't really find it good enough to justify being so different, maybe unless you already have a pyproject.toml anyway. There's a lot of talk about Python packaging but as a semi-casual user I've actually had very few problems with it.

Former heavy Pipenv user here.

Moved all my projects to Poetry for reasons. But in hindsight, I must say that Pipenv used to work just fine for me, as does Poetry today. No major issues so far with either tool.

What I do miss is being able to `pipenv run myscript`. That UX has degraded slightly for me, having to say `poetry run poe myscript` now. Barely an inconvenience though.

Re: Thoughts on the Python packaging ecosystem

#98

Post by bombolo »

411111111111111 wrote: post #91
bombolo wrote: post #90

Earlier quoted context omitted.

So we should have 1 single shell command to do literally everything?

before poetry and pipenv was a thing, pretty much everyone was recommending the virtualenvwrapper script instead of using venv directly... so yes, thats what it ultimetaly always ends up as.

One might invest 20 seconds to learn how to use it, but to each their own :)

Re: Thoughts on the Python packaging ecosystem

#99

Post by Kamq »

wheelerof4te wrote: post #96
Kamq wrote: post #35

Earlier quoted context omitted.

> Also, the standard should favor pure-Python packages, in order to untangle Python from it's legacy as a glue language for modules written in C(++). Never gonna happen. Python's explosion has been through data science and ml, which is based entirely around calling Fortran/C/C++/etc. > Otherwise, no progress will be made at the language level and Python will die out once C becomes legacy language, replaced by safer s…

"Never gonna happen. Python's explosion has been through data science and ml, which is based entirely around calling Fortran/C/C++/etc." What's Django, then? "I assume you mean Rust here. Do you think python can't already call rust? It's been able to do that for the better part of a decade." Python can only call C code. The fact that Rust itself (or Fortran, btw) has good C ABI is on Rust, not Python. Also, by callin…

> What's Django, then?

Pretty rare compared to data work, at least among python projects I've worked on.

> Python can only call C code. The fact that Rust itself (or Fortran, btw) has good C ABI is on Rust, not Python. Also, by calling Rust's C ABI, you are making a double indirection, thus severely reducing performance.

That's a bit of an implementation detail, but fair. There is some overhead. But every language since C has shied away from committing to a stable abi (or at least the C ecosystem is willing to commit to one, the standard mentions nothing about it).

That's only overhead at the boundaries though, and the common pattern of using python as an orchestrator with the vast majority of logic happening inside native code really does minimize the overhead you actually experience. At least you can architect an application that way (same pattern works with things like pyspark as well).

> That's why I said we need to optimize Python

By all means, optimize python. I think you'll run into the GIL pretty fast, but I wasn't objecting to that. I was objecting to making things like pyspark or tensorflow second class citizens. Doing that is going to destroy python.

Post reply on HN