Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

PyPI Blog: Project Quarantine

blog.pypi.org

Post reply on HN
Print view View original

51–60 of 62 posts

Re: PyPI Blog: Project Quarantine

#51

Post by LtWorf »

woodruffw wrote: post #44
LtWorf wrote: post #42

Earlier quoted context omitted.

They have a say in that if it doesn't behave like they want they can point the documentation to something that does. If pip is the tool linked in the documentation it's the official one that has the PSF's blessing, clearly.

We're going in circles. PSF can't unilaterally change any documentation of particular relevance here; the most immediately relevant docs would be controlled by PyPA and PyPI itself. The former has a standards/community review process, and the latter is particular to PyPI. But again: this has nothing to do with blessings or not. The fact that pip is the official installer and PyPI is the official index does not mean t…

Who controls the python.org domain that I linked? Am I wrong to think that PSF controls it?

Re: PyPI Blog: Project Quarantine

#52

Post by IshKebab »

woodruffw wrote: post #50
IshKebab wrote: post #49

Earlier quoted context omitted.

The simplest way to fix this problem is to support namespaces, which PyPI absolutely does need to be aware of.

This would not be meaningfully addressed by namespaces, since there's no authoritative, authenticated unique name system across indices. Two separate indices can (and will, based on what ecosystems like piwheels do[1]) advertise `foo/*`-namespaced packages, leaving installers where they are today. (I think namespacing is a good idea regardless, if only because it eliminates artificial scarcity in a one-level namespac…

This absolutely would be meaningfully addressed by namespaces because the typical use case is PyPI + a private repo you control. Register the namespace in both repos and you're done.

If you disagree I would love to hear of a concrete way that solution would be vulnerable.

Re: PyPI Blog: Project Quarantine

#53

Post by woodruffw »

IshKebab wrote: post #52
woodruffw wrote: post #50

Earlier quoted context omitted.

This would not be meaningfully addressed by namespaces, since there's no authoritative, authenticated unique name system across indices. Two separate indices can (and will, based on what ecosystems like piwheels do[1]) advertise `foo/*`-namespaced packages, leaving installers where they are today. (I think namespacing is a good idea regardless, if only because it eliminates artificial scarcity in a one-level namespac…

This absolutely would be meaningfully addressed by namespaces because the typical use case is PyPI + a private repo you control. Register the namespace in both repos and you're done. If you disagree I would love to hear of a concrete way that solution would be vulnerable.

> Register the namespace in both repos and you're done.

That’s the operative part of “authoritative.” It’s a distributed trust problem, and there’s no particular guarantee that your namespace on one index will be honored by another. Namespacing is great for eliminating scarcity on one index at a time; I don’t think it helps much with this kind of cross-index security.

Re: PyPI Blog: Project Quarantine

#54

Post by woodruffw »

LtWorf wrote: post #51
woodruffw wrote: post #44

Earlier quoted context omitted.

We're going in circles. PSF can't unilaterally change any documentation of particular relevance here; the most immediately relevant docs would be controlled by PyPA and PyPI itself. The former has a standards/community review process, and the latter is particular to PyPI. But again: this has nothing to do with blessings or not. The fact that pip is the official installer and PyPI is the official index does not mean t…

Who controls the python.org domain that I linked? Am I wrong to think that PSF controls it?

PSF controls that domain. But that domain doesn’t host PyPI (anymore) or the PyPA docs, so I’m not sure what connection you’re making there.

Re: PyPI Blog: Project Quarantine

#55

Post by oblvious-earth »

IshKebab wrote: post #10

The still don't even have a way to avoid dependency confusion attacks when using private package repos (other than also registering every single private package name you use on pypi.org). Blows my mind.

If you're concerned about dependency confusion attacks you should host your own index and vet what goes on to it.

But there is a better solution coming, PEP 708 was developed for this and is in prototype on pypi.org, so it's an overstatement to say "don't even have a way to avoid dependency confusion attacks ".

It is, however, a non-trivial problem, and more solutions will likely come over the years, many Python packaging tools like uv and poetry (and likely others) have way to name indexes and pin specific packages to indexes, which appears to be a promising UX.

Re: PyPI Blog: Project Quarantine

#56

Post by IshKebab »

woodruffw wrote: post #53
IshKebab wrote: post #52

Earlier quoted context omitted.

This absolutely would be meaningfully addressed by namespaces because the typical use case is PyPI + a private repo you control. Register the namespace in both repos and you're done. If you disagree I would love to hear of a concrete way that solution would be vulnerable.

> Register the namespace in both repos and you're done. That’s the operative part of “authoritative.” It’s a distributed trust problem, and there’s no particular guarantee that your namespace on one index will be honored by another. Namespacing is great for eliminating scarcity on one index at a time; I don’t think it helps much with this kind of cross-index security.

Right, so it would provide a solution to the specific problem I'm taking about.

Re: PyPI Blog: Project Quarantine

#57

Post by LtWorf »

woodruffw wrote: post #54
LtWorf wrote: post #51

Earlier quoted context omitted.

Who controls the python.org domain that I linked? Am I wrong to think that PSF controls it?

PSF controls that domain. But that domain doesn’t host PyPI (anymore) or the PyPA docs, so I’m not sure what connection you’re making there.

https://docs.python.org/3/installing/index.html

so it's PSF's decision to document using pip, right? And they might decide to change that page without asking for anyone's consent right?

So they also don't get to complain about pip's shortcomings because it is their decision to point users towards pip. Correct?

Have I been explicit enough for you now?

Re: PyPI Blog: Project Quarantine

#58

Post by woodruffw »

LtWorf wrote: post #57
woodruffw wrote: post #54

Earlier quoted context omitted.

PSF controls that domain. But that domain doesn’t host PyPI (anymore) or the PyPA docs, so I’m not sure what connection you’re making there.

https://docs.python.org/3/installing/index.html so it's PSF's decision to document using pip, right? And they might decide to change that page without asking for anyone's consent right? So they also don't get to complain about pip's shortcomings because it is their decision to point users towards pip. Correct? Have I been explicit enough for you now?

You've pointed to a stub document that basically explains Python packaging and why pip comes with Python (which, notably, distributions are fond of breaking).

This doesn't somehow imply that Python has a top-down authority structure where the PSF dictates the development flow of PyPI, pip, or any other official/semi-official/blessed tooling or infrastructure. That's not how Python works as a community.

The PSF is also not complaining here about pip's shortcomings. I'm the only one here and I don't represent the PSF, nor am I complaining: I think pip is great. I'm trying to explain (apparently unsuccessfully) why pip (and PyPI's) behavior isn't always 100% congruous with a single stream of development thinking. As mentioned above there are more optimal structures given different kinds of governance and community, but this one aligns with the (IMO good) values that the Python community espouses.

Re: PyPI Blog: Project Quarantine

#59

Post by LtWorf »

woodruffw wrote: post #58
LtWorf wrote: post #57

Earlier quoted context omitted.

https://docs.python.org/3/installing/index.html so it's PSF's decision to document using pip, right? And they might decide to change that page without asking for anyone's consent right? So they also don't get to complain about pip's shortcomings because it is their decision to point users towards pip. Correct? Have I been explicit enough for you now?

You've pointed to a stub document that basically explains Python packaging and why pip comes with Python (which, notably, distributions are fond of breaking). This doesn't somehow imply that Python has a top-down authority structure where the PSF dictates the development flow of PyPI, pip, or any other official/semi-official/blessed tooling or infrastructure. That's not how Python works as a community. The PSF is als…

I think the PSF has the means and opportunity to say "please don't use xxx, use yyy instead". Which is why conda and linux distributions are not seen on the same footing at all.

Re: PyPI Blog: Project Quarantine

#60

Post by ryan29 »

woodruffw wrote: post #50
IshKebab wrote: post #49

Earlier quoted context omitted.

The simplest way to fix this problem is to support namespaces, which PyPI absolutely does need to be aware of.

This would not be meaningfully addressed by namespaces, since there's no authoritative, authenticated unique name system across indices. Two separate indices can (and will, based on what ecosystems like piwheels do[1]) advertise `foo/*`-namespaced packages, leaving installers where they are today. (I think namespacing is a good idea regardless, if only because it eliminates artificial scarcity in a one-level namespac…

> since there's no authoritative, authenticated unique name system across indices

Domains provide a globally unique namespace and ownership can be verified automatically with domain validation. Bluesky did an ok job of it, but they didn't do anything to account for domain ownership changes and re-validation is non-existent, which is disappointing to see from the first big adopter since the oversight will eventually invite criticism.

I've wanted domain validated namespaces for 5+ years. Here's a comment I made about using domain validated namespaces in package managers a couple of years ago [1]:

---

I think one possible solution to that would be to assume namespaces can have their ownership changed and build something that works with that assumption.

Think along the lines of having 'pypi.org/example.com' be a redirect to an immutable organization; 'pypi.org/abcd1234'. If a new domain owner wants to take over the namespace they won't have access to the existing account and re-validating to take ownership would force them to use a different immutable organization; 'pypi.org/ef567890'.

If you have a package locking system (like NPM), it would lock to the immutable organization and any updates that resolve to a new organization could throw a warning and require explicit approval. If you think of it like an organization lock:

    v1:

        pypi.org/example.com --> pypi.org/abcd1234

    v2:

        pypi.org/example.com --> pypi.org/ef123456
If you go from v1 to v2 you know there was an ownership change or, at the very least, an event that you need to investigate.

Losing control of a domain would be recoverable because existing artifacts wouldn't be impacted and you could use the immutable organization to publish the change since that's technically the source of truth for the artifacts. Put another way, the immutable organization has a pointer back the current domain validated namespace:

    v1:

        pypi.org/abcd1234 --> example.com

    v2:

        pypi.org/abcd1234 --> example.net
If you go from v1 to v2 you know the owner of the artifacts you want has moved from the domain example.com to example.net. The package manager could give a warning about this and let an artifact consumer approve it, but it's less risky than the change above because the owner of 'abcd1234' hasn't changed and you're already trusting them.

---

1. https://news.ycombinator.com/item?id=32754029

Post reply on HN