Live data from Hacker News

Pypi.org is running a survey on the state of Python packaging

pypi.org

61–70 of 193 posts

Re: Pypi.org is running a survey on the state of Python packaging

#61
post #36

Earlier quoted context omitted.

I think you'd probably need some buy in from the domain registries and ICANN to make it really solid. Ideally, domains would have something similar to public certificate transparency logs where domain expirations would be recorded. I even think it would be reasonable to log registrant changes (legal registrant, not contact info). In both cases, it wouldn't need to include any identifiable info, just a simple expired/…

You've already talked about Sigstore (which is an excellent technology for this space), so we can consider developers holding keys that are stored in an append-only log. Then it doesn't matter if the domain expires and someone re-registers it, since they don't have the developer's private keys. Of course there are going to be complexities involving key-rollover and migrating to a different domain, but a sufficiently…

> Then it doesn't matter if the domain expires and someone re-registers it, since they don't have the developer's private keys.

That's a good point in terms of invalidation, but a new domain registrant should be able to claim the namespace and start using it.

I think one possible solution to that would be to assume namespaces can have their ownership changed and build something that works with that assumption.

Think along the lines of having 'pypi.org/example.com' be a redirect to an immutable organization; 'pypi.org/abcd1234'. If a new domain owner wants to take over the namespace they won't have access to the existing account and re-validating to take ownership would force them to use a different immutable organization; 'pypi.org/ef567890'.

If you have a package locking system (like NPM), it would lock to the immutable organization and any updates that resolve to a new organization could throw a warning and require explicit approval. If you think of it like an organization lock:

v1:

    pypi.org/example.com --> pypi.org/abcd1234
v2:

    pypi.org/example.com --> pypi.org/ef123456
If you go from v1 to v2 you know there was an ownership change or, at the very least, an event that you need to investigate.

Losing control of a domain would be recoverable because existing artifacts wouldn't be impacted and you could use the immutable organization to publish the change since that's technically the source of truth for the artifacts. Put another way, the immutable organization has a pointer back the current domain validated namespace:

v1:

    pypi.org/abcd1234 --> example.com
v2:

    pypi.org/abcd1234 --> example.net
If you go from v1 to v2 you know the owner of the artifacts you want has moved from the domain example.com to example.net. The package manager could give a warning about this and let an artifact consumer approve it, but it's less risky than the change above because the owner of 'abcd1234' hasn't changed and you're already trusting them.

I think that's a reasonably effective way of solving attacks that rely on registering expired domains to take over a namespace and it also makes it fairly trivial for namespace owners to point artifact consumers to a new domain if needed.

Think of the validated domain as more of a vanity pointer than an actual artifact repository. In fact, thinking about it like that, you don't actually need any cooperation or buy in from the domain registries.

> The hard part is how to issue a new key if you lose the old one, since that probably requires social vouching and a reputation system.

It's actually really hard because as you increase the value of a key, I think you decrease the security practices around handling them. For example, some people will simply drop their keys into OneDrive if there's any inconvenience associated with losing them.

I would really like to have something where I can use a key generated on a tamper proof device like a YubiKey and not have to worry about losing it. Ideally, I could register a new key without any friction.

Re: Pypi.org is running a survey on the state of Python packaging

#62

I have a terrible admission to make: one of the reason I like Python is its huge standard library, and I like that because I just ... despise looking for libraries, trying to install them, evaluating their fitness, and so on. I view dependencies outside of the standard library as a kind of technical debt, not because I suffer from Not Invented Here and want to code it myself, no, I look and think, "Why isn't this in…

> despise looking for libraries, trying to install them, evaluating their fitness, and so on. This is exactly why I prefer the larger opinionated web frameworks (Django, Vue.js) to the smaller more composable frameworks (Flask, React). I don’t what to make decisions every time I need a new feature, I want something that “just works”. Python and Django just work, and brilliantly at that!

Currently dealing with Flask and it makes me sad from the endless decision fatigue. Enormous variations in quality of code, documentation, SO answers, etc. To not even consider the potential for supply side attacks.

With Django there is a happy path answer for most everything. If I run into a problem, I know I'm not the first.

Re: Pypi.org is running a survey on the state of Python packaging

#63
post #17

My wishlist: We need a way to configure an ordered list of indexes pip searches for packages. —extra-index-url or using a proxy index is not the solution. Also namespaces and not based on a domain. So for example: pip install apache:parquet Also some logic either in the pip client or index server to minimize typosquatting Also pip should adopt a lock file similar to npm/yarn. Instead of requirements.txt And also “pip…

If you are using poetry you can add something to the pyproject.toml to handle the indexes, though I am not sure if they are ordered or not

[[tool.poetry.source]] name = "my-pypi" url = "https://my-pypi-index.wherever" secondary = true

Re: Pypi.org is running a survey on the state of Python packaging

#64
post #15

I just wish that PyPI would enforce binary wheels going forward (at least for linux x64/arm64 for people who use Docker, but ideally for all common platforms). They already supply the cibuildwheel tool to automate their builds, so it shouldn't be that hard for library developers... Software developers shouldn't need to figure out what build-time dependencies their libraries need...

Should PyPI kick my project off because it don't support MS Windows?

My package uses C and Cython extensions. While I support macOS and Linux-based OSes, I don't know how to develop on or support MS Windows.

I've tried to be careful about the ILP32 vs LP64 differences, but I suspect there's going to be many places where I missed up.

I also use "/dev/stdin" to work-around my use of a third-party library that has no way to read from stdin. As far as I can tell, there's no equivalent in Windows, so I'll have to raise an exception for that case, and modify my tests cases.

Re: Pypi.org is running a survey on the state of Python packaging

#65
post #56
post #38

I like poetry for its simplicity but I can’t tell how “official” it is in the python ecosystem. I hope it doesn’t die out. I think it’s the simplest possible way to maintain deps and publish to PyPI if you don’t have any weird edge cases.

Couldn't agree more. Poetry is fantastic and provides that 'just works' experience for most cases. It's not official (although possibly should be adopted) but has gained ground by virtue of its quality. Fortunately it's very actively developed so will hopefully stick around.

I like poetry. It still has a way to go though since it is slow as all hell doing almost anything and its error messages are closer to a stack trace than something actionable.

Re: Pypi.org is running a survey on the state of Python packaging

#66
post #51

I just wish they'd change their name so that my students stop snickering. (The name is pronounced like the French word for "piss".)

Isn't PyPI pronounced like: pie (food/π) + pea + eye ().

that is different from the french "pipi": pea + pea.

Re: Pypi.org is running a survey on the state of Python packaging

#67
post #57
post #41

For improvements I commented: Remove setup.py files and mandate wheels. This is the root cause of a lot of the evil in the ecosystem. Next on the list would be pypi namespaces, but there are good reasons why that is very hard. The mission statement they are proposing, “a packaging ecosystem for all”, completely misses the mark. How about a “packaging ecosystem that works” first? I spent a bunch of time recently fixin…

> Remove setup.py files and mandate wheels What alternative is there for me? My package has a combination of hand-built C extensions and Cython extensions, as well as a code generation step during compilation. These are handled through a subclass of setuptools.command.build_ext.build_ext. Furthermore, I have compile-time options to enable/disable certain configuration options, like enabling/disabling support for Open…

[deleted]

Re: Pypi.org is running a survey on the state of Python packaging

#68
post #58
post #48

Earlier quoted context omitted.

...and ensure _all_ package metadata required to perform dependency resolution can be retrieved through an API (in other words without downloading wheels).

What if I have a dependency on a commercial third-party Python package which is on Conda but not on PyPI?

you are placing open code in a vendor lock-in, to start

Re: Pypi.org is running a survey on the state of Python packaging

#69
Python lacks the flexible universal binary distribution solution which nearly all of the new comers has. Consider golang, rust or docker images. Most probably docker image distribution is the only available solution for now and volume management is the worst problem on that front.

Re: Pypi.org is running a survey on the state of Python packaging

#70
post #41

For improvements I commented: Remove setup.py files and mandate wheels. This is the root cause of a lot of the evil in the ecosystem. Next on the list would be pypi namespaces, but there are good reasons why that is very hard. The mission statement they are proposing, “a packaging ecosystem for all”, completely misses the mark. How about a “packaging ecosystem that works” first? I spent a bunch of time recently fixin…

> For improvements I commented: Remove setup.py files and mandate wheels.

This would make most C extensions impossible to install on anything other than x86_64-pc-linux-gnu (or arm-linux-gnueabihf/aarch64-linux-gnu if you are lucky) because developers don't want to bother building wheels for them.

Post reply on HN