Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

What the Heck Is Pyproject.toml?

snarky.ca

Post reply on HN
Print view View original

21–30 of 199 posts

Re: What the Heck Is Pyproject.toml?

#21

Post by aprdm »

tych0 wrote: post #14
fermienrico wrote: post #4

Python desperately needs a better packaging solution than pip from Python Software Foundation. Stop all new features until this - the biggest pain point for Python across all expertise levels from newbies to experienced Python programmers - is officially solved. No pipenv, no poetry, no conda, etc from third party devs who sometimes get tired of the pressure [1]. It is immense. Combine all this virtualenv stuff with…

FWIW, the issue you linked to was closed in favor of https://github.com/pypa/pipenv/issues/3369 . Looks like Canonical is paying him now.

I had some really bad experiences with pipenv. I don't think it is the way forward. It also seems like abandonware at the moment so I am glad there's at least work happening.

Re: What the Heck Is Pyproject.toml?

#22

Post by harikb »

fermienrico wrote: post #4

Python desperately needs a better packaging solution than pip from Python Software Foundation. Stop all new features until this - the biggest pain point for Python across all expertise levels from newbies to experienced Python programmers - is officially solved. No pipenv, no poetry, no conda, etc from third party devs who sometimes get tired of the pressure [1]. It is immense. Combine all this virtualenv stuff with…

I teach Python to some middle school kids. Pip vs Pip3 and PYTHONPATH wasted entirety of two class sessions.

By the way, Python should go beyond Go. It should single binary the someone can install without root privileges. The fact that I have hundreds of Chromebooks at my school and I can’t use any of them - school IT, correctly, doesn’t want me mucking with advanced settings.

Re: What the Heck Is Pyproject.toml?

#23

Post by brettcannon »

no_wizard wrote: post #11
fermienrico wrote: post #4

Python desperately needs a better packaging solution than pip from Python Software Foundation. Stop all new features until this - the biggest pain point for Python across all expertise levels from newbies to experienced Python programmers - is officially solved. No pipenv, no poetry, no conda, etc from third party devs who sometimes get tired of the pressure [1]. It is immense. Combine all this virtualenv stuff with…

I agree. This really seems like high impact too. A first party, supported solution would solve this mess. Getting the Python dependency build story under control I would include in this category. In this department, I think Conda shows the way. They leverage Azure Pipelines to build much of the c dependencies ahead of time, and just send binaries down the pipe (at least, that's the high level overview as I understand…

> I can't see a world where the Python Software Foundation can't get support for this, I think they have the contacts at the big companies to get it funded.

You might be surprised at how hard it is to get money donated for this sort of thing. Large companies often have their own build infrastructure and thus build things for themselves from scratch, to the point of building their own build toolchains (e.g. look at Google and bazel, Facebook and buck). So the way they handle packaging means how the overall packaging ecosystem functions does not impact them like it does individuals or smaller companies.

This is also why companies only started funding PyPI security work about two years ago; downloading source securely from PyPI is important to big companies as that's where they grab the source that they compile. But funding pip improvements only just happened in Decembe, starting with the new pip resolver work got money (https://pyfound.blogspot.com/2020/03/new-pip-resolver-to-rol...). And all that money came from Mozilla and the Chan Zuckerberg Initiative; two huge non-profits, not companies.

Add on to the fact that the PSF is projected to have a $627,000 USD loss this year due to PyCon US 2020 being cancelled and it makes finding funding really hard (https://pyfound.blogspot.com/2020/03/psfs-projected-2020-fin...).

Python overall very much runs on volunteers, but the packaging system especially. The core team of Python itself might generously have a total of 3 devs/week being paid to work on CPython spread across all contributors (IOW it is nowhere near as impactful as 3 actual full-time devs who aren't context-switching constantly between "work work" and "Python work"). But for Python packaging in general? If you don't count conda I think packaging has less than even 3.

But if anyone wants to donate to try and rectify this, please go to https://psfmember.org/civicrm/contribute/transact?reset=1&id... and donate to the packaging WG!

Re: What the Heck Is Pyproject.toml?

#24

Post by brettcannon »

dralley wrote: post #8
fermienrico wrote: post #4

Python desperately needs a better packaging solution than pip from Python Software Foundation. Stop all new features until this - the biggest pain point for Python across all expertise levels from newbies to experienced Python programmers - is officially solved. No pipenv, no poetry, no conda, etc from third party devs who sometimes get tired of the pressure [1]. It is immense. Combine all this virtualenv stuff with…

None of that is "necessary". Rust doesn't have a BDFL and they've gotten along fine. But I agree that the current situation is a mess. The "official" solutions are suboptimal and none of the unofficial solutions, which are pretty good, have enough clout to meaningfully simplify the ecosystem (held hostage by the majority of users with just pip). So the ecosystem is kind of frozen in place. PyPA doesn't seem to have t…

It's all about capacity, not will. The PyPA as a whole knows what is lacking, but when you're all just a bunch of volunteers it's hard to tackle massive projects that take potentially years to complete and then even longer to see gain traction in the community when you are doing it all in your spare time. And that's if you don't burn out before you finish (e.g. calling projects "garbage" is not motivating to the few people volunteering their time to try and fix the situation while keeping what is there afloat and running).

Re: What the Heck Is Pyproject.toml?

#25

Post by brettcannon »

harikb wrote: post #22
fermienrico wrote: post #4

Python desperately needs a better packaging solution than pip from Python Software Foundation. Stop all new features until this - the biggest pain point for Python across all expertise levels from newbies to experienced Python programmers - is officially solved. No pipenv, no poetry, no conda, etc from third party devs who sometimes get tired of the pressure [1]. It is immense. Combine all this virtualenv stuff with…

I teach Python to some middle school kids. Pip vs Pip3 and PYTHONPATH wasted entirety of two class sessions. By the way, Python should go beyond Go. It should single binary the someone can install without root privileges. The fact that I have hundreds of Chromebooks at my school and I can’t use any of them - school IT, correctly, doesn’t want me mucking with advanced settings.

People would love to see that happen, but it requires time and effort from people to build such a solution. Some people are trying, e.g. https://pyoxidizer.readthedocs.io/ and https://briefcase.readthedocs.io/ as well as stalwarts like PyInstaller and such.

But as I said, it takes time and effort and there's only so much of that when it's being done by volunteers in their evenings and weekends.

And as an aside, I would suggest teaching them `python -m pip` over `pip3`: https://snarky.ca/why-you-should-use-python-m-pip/.

Re: What the Heck Is Pyproject.toml?

#26

Post by Waterluvian »

Is toml even supported in the standard library? This seems just bizarre. Python's really overtaking node for massive churn. I feel like an old guy when peers say, "don't use virtual environment use pipenv." then someone says no no don't even use that, use poetry. Etc.

I think about 50% of why I enjoy, in my soul, about Rust when I'm hacking away and personal goofs is that the dependency and project management system is all first class and pretty decent and I sure hope they don't go changing it much.

Re: What the Heck Is Pyproject.toml?

#27

Post by kapilvt »

joshvm wrote: post #16
nvrspyx wrote: post #6

Earlier quoted context omitted.

I haven't looked much into it myself, but I've heard good things about Poetry [0]. 0: https://python-poetry.org/

Poetry is good, but you should be aware that people with legacy versions of pip (I think before 19, see PEP 517) won't be able to pip install from your repository directly. That means anyone using pip installed with their package manager (up to the lastest version of Debian/Ubuntu in fact, which only provide v18). This doesn't affect installs from PyPI of course. So either you just accept this, or you also maintain a…

having just converted a large project/monorepo to poetry, its pretty easy to generate setup.py/requirements from pyproject.toml https://github.com/cloud-custodian/cloud-custodian/blob/mast...

note most of this just working around poetry for multiple packages in a single repo, with interdependencies. flip side, our pip/setuptools users building trunk from ci are unaffected.

that said there are open questions on poetry (governance/dev community), but the experience is overall pretty solid with a some minor rough edges, and the fact that it has a usable api makes those pretty minor. in contrast the only api on pip is cli arguments for subprocess/fork.

Re: What the Heck Is Pyproject.toml?

#28

Post by altoidaltoid »

j88439h84 wrote: post #12
fermienrico wrote: post #4

Python desperately needs a better packaging solution than pip from Python Software Foundation. Stop all new features until this - the biggest pain point for Python across all expertise levels from newbies to experienced Python programmers - is officially solved. No pipenv, no poetry, no conda, etc from third party devs who sometimes get tired of the pressure [1]. It is immense. Combine all this virtualenv stuff with…

Poetry is a modern packaging system for Python. It's really good. I hope everyone converges on it. http://python-poetry.org/

Where is that XKCD about new standards...

Ah yes: https://xkcd.com/927/

Re: What the Heck Is Pyproject.toml?

#29

Post by throwaway894345 »

brettcannon wrote: post #23
no_wizard wrote: post #11

Earlier quoted context omitted.

I agree. This really seems like high impact too. A first party, supported solution would solve this mess. Getting the Python dependency build story under control I would include in this category. In this department, I think Conda shows the way. They leverage Azure Pipelines to build much of the c dependencies ahead of time, and just send binaries down the pipe (at least, that's the high level overview as I understand…

> I can't see a world where the Python Software Foundation can't get support for this, I think they have the contacts at the big companies to get it funded. You might be surprised at how hard it is to get money donated for this sort of thing. Large companies often have their own build infrastructure and thus build things for themselves from scratch, to the point of building their own build toolchains (e.g. look at Go…

It would probably go a long way if they advertised a roadmap of things they’d like to work on given donations. Maybe they have this out there and I just haven’t seen it, but I imagine it would be easy enough for me to persuade my small employer to throw a few thousand USD their way each year until some sane packaging solution is developed.

Re: What the Heck Is Pyproject.toml?

#30

Post by no_wizard »

brettcannon wrote: post #23
no_wizard wrote: post #11

Earlier quoted context omitted.

I agree. This really seems like high impact too. A first party, supported solution would solve this mess. Getting the Python dependency build story under control I would include in this category. In this department, I think Conda shows the way. They leverage Azure Pipelines to build much of the c dependencies ahead of time, and just send binaries down the pipe (at least, that's the high level overview as I understand…

> I can't see a world where the Python Software Foundation can't get support for this, I think they have the contacts at the big companies to get it funded. You might be surprised at how hard it is to get money donated for this sort of thing. Large companies often have their own build infrastructure and thus build things for themselves from scratch, to the point of building their own build toolchains (e.g. look at Go…

First of all, as an aside, thank you for all the work you do on Python! I actually got started pretty early on in my professional career with Python (3.5+ too, which I am thankful for in hindsight). I was using VS Code and the Python Extension before MS even acquired its developer (is Don still there? I hope he is. Such a genuinely down to earth guy). Without the work you guys do I don't know if I would have taken the programming the same way, or at that the time I did in my life.

I will say, I didn't realize the funding situation was _that_ dire, especially since on Talk Python I remember specifically Michael and some others talking about the massive influx of money into Python and its ecosystem in the last year and half or so. Its not billions of dollars but it sounded like the situation had a positive uptick substantially, so, in short, I was wrong!

Has the PSF ever considered having an actual business? Like say, consulting and/or having developers work on features that businesses pay to have implemented? (A loose example of this would be Igalia[0], which makes surprisingly good amounts of money to implement features in web browsers on behalf of the major browser vendors)

I've always thought this would be a pretty natural way to have a steady source of revenue and the profits could all feed back into the PSF.

[0]https://www.igalia.com/

Post reply on HN