Live data from Hacker News

Pypi.org is running a survey on the state of Python packaging

pypi.org

41–50 of 193 posts

Re: Pypi.org is running a survey on the state of Python packaging

#41
For improvements I commented: Remove setup.py files and mandate wheels. This is the root cause of a lot of the evil in the ecosystem.

Next on the list would be pypi namespaces, but there are good reasons why that is very hard.

The mission statement they are proposing, “a packaging ecosystem for all”, completely misses the mark. How about a “packaging ecosystem that works” first?

I spent a bunch of time recently fixing our internal packaging repo (nexus) because the switch from md5 hashes to sha256 hashes broke everything, and re-locking a bajillion lock files would take literally months of man hours time.

I’ve been a Python user for the last 17 years, so I’m sympathetic of how we got to the current situation and aware that we’ve actually come quite far.

But every time I use Cargo I am insanely jealous, impressed and sad that we don’t have something like it. Poetry is closest, but it’s a far cry.

Re: Pypi.org is running a survey on the state of Python packaging

#42
post #31
post #29

I didn't take the survey because I've never packaged anything for PyPI, but I wish all of the package managers would have an option for domain validated namespaces. If I own example.com, I should be able to have 'pypi.org/example.com/package'. The domain can be tied back to my (domain verified) GitHub profile and it opens up the possibility of using something like 'example.com/.well-known/pypi/' for self-managed sign…

That's a really interesting idea, but I worry about what happens when a domain name expires and is re-registered (potentially even maliciously) by someone else.

CT: Certificate Transparency logs log creation and revocation events.

The Google/trillian database which supports Google's CT logs uses Merkle trees but stores the records in a centralized data store - meaning there's at least one SPOF Single Point of Failure - which one party has root on and sole backup privileges for.

Keybase, for example, stores their root keys - at least - in a distributed, redundantly-backed-up blockchain that nobody has root on; and key creation and revocation events are publicly logged similarly to now-called "CT logs".

You can link your Keybase identity with your other online identities by proving control by posting a cryptographic proof; thus adding an edge to a WoT Web of Trust.

While you can add DNS record types like CERT, OPENPGPKEY, SSHFP, CAA, RRSIG, NSEC3; DNSSEC and DoH/DoT/DoQ cannot be considered to be universally deployed across all TLDs. Should/do e.g. ACME DNS challenges fail when a TLD doesn't support DNSSEC, or hasn't secured root nameservers to a sufficient baseline, or? DNS is not a trustless system.

EDNS (Ethereum DNS) is a trustless system. Reading EDNS records does not cost EDNS clients any gas/particles/opcodes/ops/money.

Blockcerts is designed to issue any sort of credential, and allow for signing of any RDF graph like JSON-LD.

List_of_DNS_record_types: https://en.wikipedia.org/wiki/List_of_DNS_record_types

Blockcerts: https://www.blockcerts.org/ https://github.com/blockchain-certificates :

> Blockcerts is an open standard for creating, issuing, viewing, and verifying blockchain-based certificates

W3C VC-DATA-MODEL: https://w3c.github.io/vc-data-model/ :

> Credentials are a part of our daily lives; driver's licenses are used to assert that we are capable of operating a motor vehicle, university degrees can be used to assert our level of education, and government-issued passports enable us to travel between countries. This specification provides a mechanism to express these sorts of credentials on the Web in a way that is cryptographically secure, privacy respecting, and machine-verifiable

W3C VC-DATA-INTEGRITY: "Verifiable Credential Data Integrity 1.0" https://w3c.github.io/vc-data-integrity/#introduction :

> This specification describes mechanisms for ensuring the authenticity and integrity of Verifiable Credentials and similar types of constrained digital documents using cryptography, especially through the use of digital signatures and related mathematical proofs. Cryptographic proofs enable functionality that is useful to implementors of distributed systems. For example, proofs can be used to: Make statements that can be shared without loss of trust,

W3C TR DID (Decentralized Identifiers) https://www.w3.org/TR/did-core/ :

> Decentralized identifiers (DIDs) are a new type of identifier that enables verifiable, decentralized digital identity. A DID refers to any subject (e.g., a person, organization, thing, data model, abstract entity, etc.) as determined by the controller of the DID. In contrast to typical, federated identifiers, DIDs have been designed so that they may be decoupled from centralized registries, identity providers, and certificate authorities. Specifically, while other parties might be used to help enable the discovery of information related to a DID, the design enables the controller of a DID to prove control over it without requiring permission from any other party. DIDs are URIs that associate a DID subject with a DID document allowing trustable interactions associated with that subject.

> Each DID document can express cryptographic material, verification methods, or services, which provide a set of mechanisms enabling a DID controller to prove control of the DID. Services enable trusted interactions associated with the DID subject. A DID might provide the means to return the DID subject itself, if the DID subject is an information resource such as a data model.

Re: Pypi.org is running a survey on the state of Python packaging

#43
post #31

Earlier quoted context omitted.

That's a really interesting idea, but I worry about what happens when a domain name expires and is re-registered (potentially even maliciously) by someone else.

CT: Certificate Transparency logs log creation and revocation events. The Google/trillian database which supports Google's CT logs uses Merkle trees but stores the records in a centralized data store - meaning there's at least one SPOF Single Point of Failure - which one party has root on and sole backup privileges for. Keybase, for example, stores their root keys - at least - in a distributed, redundantly-backed-up…

Sigstore uses Trillian for its transparency log, Rekor.

Re: Pypi.org is running a survey on the state of Python packaging

#44
post #36

Earlier quoted context omitted.

I think you'd probably need some buy in from the domain registries and ICANN to make it really solid. Ideally, domains would have something similar to public certificate transparency logs where domain expirations would be recorded. I even think it would be reasonable to log registrant changes (legal registrant, not contact info). In both cases, it wouldn't need to include any identifiable info, just a simple expired/…

You've already talked about Sigstore (which is an excellent technology for this space), so we can consider developers holding keys that are stored in an append-only log. Then it doesn't matter if the domain expires and someone re-registers it, since they don't have the developer's private keys. Of course there are going to be complexities involving key-rollover and migrating to a different domain, but a sufficiently…

> Then it doesn't matter if the domain expires and someone re-registers it, since they don't have the developer's private keys.

A principal reason to use sigstore is to get out of the business of handling private keys entirely. It turns a key management problem into an identity problem, the latter being much easier to solve at scale.

Re: Pypi.org is running a survey on the state of Python packaging

#45
post #29

I didn't take the survey because I've never packaged anything for PyPI, but I wish all of the package managers would have an option for domain validated namespaces. If I own example.com, I should be able to have 'pypi.org/example.com/package'. The domain can be tied back to my (domain verified) GitHub profile and it opens up the possibility of using something like 'example.com/.well-known/pypi/' for self-managed sign…

Maven Central requires validation of a domain name in order to use a reverse-domain package[0].

It's not without problems. One is that folks often don't control the domain (consider Go's charming habit of conflating version control with package namespacing). Another is what was noted below: resurrection attacks on domains can be quite trivial and already happen in other forms (eg registering lapsed domains for user accounts and performing a reset).

[0] https://central.sonatype.org/faq/how-to-set-txt-record/

Re: Pypi.org is running a survey on the state of Python packaging

#46
post #31

Earlier quoted context omitted.

That's a really interesting idea, but I worry about what happens when a domain name expires and is re-registered (potentially even maliciously) by someone else.

CT: Certificate Transparency logs log creation and revocation events. The Google/trillian database which supports Google's CT logs uses Merkle trees but stores the records in a centralized data store - meaning there's at least one SPOF Single Point of Failure - which one party has root on and sole backup privileges for. Keybase, for example, stores their root keys - at least - in a distributed, redundantly-backed-up…

For another example of how Ethereum might be useful for certificate transparency, there's a fascinating paper from 2016 called "EthIKS: Using Ethereum to audit a CONIKS key transparency log" which is probably way ahead of its time.

Abstract: https://link.springer.com/chapter/10.1007/978-3-662-53357-4_...

PDF: https://jbonneau.com/doc/B16b-BITCOIN-ethiks.pdf

Re: Pypi.org is running a survey on the state of Python packaging

#47
post #31
post #29

I didn't take the survey because I've never packaged anything for PyPI, but I wish all of the package managers would have an option for domain validated namespaces. If I own example.com, I should be able to have 'pypi.org/example.com/package'. The domain can be tied back to my (domain verified) GitHub profile and it opens up the possibility of using something like 'example.com/.well-known/pypi/' for self-managed sign…

That's a really interesting idea, but I worry about what happens when a domain name expires and is re-registered (potentially even maliciously) by someone else.

Needs compensating controls to get it right.

* Dependencies are managed in a similar way to Go - where hashes of installed packages are stored and compared client side. This means that a hijacker could only serve up the valid versions of packages that I’ve already installed.

* This is still a “centralized” model where a certain level of trust is placed in PyPi - a mode of operation where the “fingerprint” of the TLS key is validated would assist here. However it comes with a few constraints.

Of course the above still comes with the caveat that you have to trust pypi. I’m not saying that this is an unreasonable ask. It’s just how it is.

Re: Pypi.org is running a survey on the state of Python packaging

#48
post #41

For improvements I commented: Remove setup.py files and mandate wheels. This is the root cause of a lot of the evil in the ecosystem. Next on the list would be pypi namespaces, but there are good reasons why that is very hard. The mission statement they are proposing, “a packaging ecosystem for all”, completely misses the mark. How about a “packaging ecosystem that works” first? I spent a bunch of time recently fixin…

...and ensure _all_ package metadata required to perform dependency resolution can be retrieved through an API (in other words without downloading wheels).

Re: Pypi.org is running a survey on the state of Python packaging

#49
post #48
post #41

For improvements I commented: Remove setup.py files and mandate wheels. This is the root cause of a lot of the evil in the ecosystem. Next on the list would be pypi namespaces, but there are good reasons why that is very hard. The mission statement they are proposing, “a packaging ecosystem for all”, completely misses the mark. How about a “packaging ecosystem that works” first? I spent a bunch of time recently fixin…

...and ensure _all_ package metadata required to perform dependency resolution can be retrieved through an API (in other words without downloading wheels).

Yeah, that’s sort of what I meant by my suggestion. Requirements that can only be resolved by downloading and executing code is a huge burden on tooling

Re: Pypi.org is running a survey on the state of Python packaging

#50

Earlier quoted context omitted.

I meant this for dependency not the package I am installing.

I'm not sure I understand what you mean -- `pip` should not be reinstalling transitive dependencies. If you install A and B and both depend on C, C should only be installed once.

I think they're referring to requirements files. I've seen the same behavior - on day 1, pip installs packages A and B, then on day 2 when someone else modified the requirements file it installs C and reinstalls B even though B hasn't changed.

This one I've seen when the dependency doesn't specify an exact version and you included "--upgrade".

There's a second case that I think was fixed in pip 21 or 22, where two transitive dependencies overlap - A and B depend on C, but with different version ranges. If A allows a newer version of C than B allows, C can get installed twice.

Post reply on HN