Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

PyPI in 2025: A Year in Review

blog.pypi.org

Post reply on HN
Print view View original

31–40 of 43 posts

Re: PyPI in 2025: A Year in Review

#31

Post by woodruffw »

bastawhiz wrote: post #29
woodruffw wrote: post #18

Earlier quoted context omitted.

The searchable context for a distribution on PyPI is unbounded in the general case, assuming the goal is to allow search over READMEs, distribution metadata, etc. (Which isn’t to say I disagree with you about scale not being the main issue, just to offer some nuance. Another piece of nuance is the fact that distributions are the source of metadata but users think in terms of projects/releases.)

> assuming the goal is to allow search over READMEs, distribution metadata, etc. Why would you build a dedicated tool for this instead of just using a search engine? If I'm looking for a specific keyword in some project's very long README I'm searching kagi, not npm. I'd expect that the most you should be indexing is the data in the project metadata (setup.py). That could be unbounded but I can't think of a compellin…

You would definitely use a search engine. I was just responding to a specific design constraint.

(Note PyPI can’t index metadata from a `setup.py` however, since that would involve running arbitrary code. PyPI needs to be given structured metadata, and not all distributions provide that.)

Re: PyPI in 2025: A Year in Review

#32

Post by ptx »

firesteelrain wrote: post #17
froh wrote: post #16

Earlier quoted context omitted.

I wonder how a PyPi search index could be statically served and locally evaluated on `pip search`?

PyPI servers would have to be constantly rebuilding a central index and making it available for download. Seems inefficient

Debian is somehow able to manage it for apt.

Re: PyPI in 2025: A Year in Review

#33

Post by firesteelrain »

ptx wrote: post #32
firesteelrain wrote: post #17

Earlier quoted context omitted.

PyPI servers would have to be constantly rebuilding a central index and making it available for download. Seems inefficient

Debian is somehow able to manage it for apt.

1. Debian is local first via client side cache

2. apt repositories are cryptographically signed, centrally controlled, and legally accountable.

3. apt search is understood to be approximate, distro-scoped, and slow-moving. Results change slowly and rarely break scripts. PyPI search rankings change frequently by necessity

4. Turning PyPI search into an apt-like experience would require distributing a signed, periodically refreshed global metadata corpus to every client. At PyPI’s scale, that is nontrivial in bandwidth, storage, and governance terms

5. apt search works because the repository is curated, finite, and opinionated

Re: PyPI in 2025: A Year in Review

#34

Post by coldtea »

woodruffw wrote: post #18
bastawhiz wrote: post #10

Earlier quoted context omitted.

Pypi has fewer than one million projects. The searchable content for each package is what? 300 bytes? That's a 200mb index. You don't even need fancy full text search, you could literally split the query by word and do a grep over a text file. No need for elasticsearch or anything fancy. And anyway, hit rates are going to be pretty good. You're not taking arbitrary queries, the domain is pretty narrow. Half the queri…

The searchable context for a distribution on PyPI is unbounded in the general case, assuming the goal is to allow search over READMEs, distribution metadata, etc. (Which isn’t to say I disagree with you about scale not being the main issue, just to offer some nuance. Another piece of nuance is the fact that distributions are the source of metadata but users think in terms of projects/releases.)

>The searchable context for a distribution on PyPI is unbounded in the general case, assuming the goal is to allow search over READMEs, distribution metadata, etc.

Even including those, it's what? Sub-20-30GB.

Re: PyPI in 2025: A Year in Review

#35

Post by froh »

firesteelrain wrote: post #17
froh wrote: post #16

Earlier quoted context omitted.

I wonder how a PyPi search index could be statically served and locally evaluated on `pip search`?

PyPI servers would have to be constantly rebuilding a central index and making it available for download. Seems inefficient

that depends on how it can be downloaded incrementally.

Re: PyPI in 2025: A Year in Review

#36

Post by froh »

firesteelrain wrote: post #33
ptx wrote: post #32

Earlier quoted context omitted.

Debian is somehow able to manage it for apt.

1. Debian is local first via client side cache 2. apt repositories are cryptographically signed, centrally controlled, and legally accountable. 3. apt search is understood to be approximate, distro-scoped, and slow-moving. Results change slowly and rarely break scripts. PyPI search rankings change frequently by necessity 4. Turning PyPI search into an apt-like experience would require distributing a signed, periodica…

isn't this an incrementally updatable tree that is managed with a Merkle tree? git-like, essentially?

Re: PyPI in 2025: A Year in Review

#37

Post by firesteelrain »

froh wrote: post #36
firesteelrain wrote: post #33

Earlier quoted context omitted.

1. Debian is local first via client side cache 2. apt repositories are cryptographically signed, centrally controlled, and legally accountable. 3. apt search is understood to be approximate, distro-scoped, and slow-moving. Results change slowly and rarely break scripts. PyPI search rankings change frequently by necessity 4. Turning PyPI search into an apt-like experience would require distributing a signed, periodica…

isn't this an incrementally updatable tree that is managed with a Merkle tree? git-like, essentially?

The install side is basically Merkle-friendly (immutable artifacts, append-only metadata, hashes, mirrors). Search isn’t. Search results are derived, subjective, and frequently rewritten (ranking tweaks, spam/malware takedowns, popularity signals). That’s more like constantly rebasing than appending commits.

You can Merklize “what files exist”; you can’t realistically Merklize “what should rank for this query today” without freezing semantics and turning CLI search into a hard API contract.

Re: PyPI in 2025: A Year in Review

#38

Post by musicale »

> More than 3.9 million new files published

> More than 130,000 new projects created

Is there any way to prevent PyPI from becoming a morass of supply chain attacks like NPM etc.? The cited security measures (though some of them like domain resurrection protection are probably very good ideas) seem like they won't, but it also seems like a very hard problem to solve, given the vast scale as well as core issues like malicious (but seemingly innocuous) upstream commits.

Re: PyPI in 2025: A Year in Review

#39

Post by froh »

firesteelrain wrote: post #37
froh wrote: post #36

Earlier quoted context omitted.

isn't this an incrementally updatable tree that is managed with a Merkle tree? git-like, essentially?

The install side is basically Merkle-friendly (immutable artifacts, append-only metadata, hashes, mirrors). Search isn’t. Search results are derived, subjective, and frequently rewritten (ranking tweaks, spam/malware takedowns, popularity signals). That’s more like constantly rebasing than appending commits. You can Merklize “what files exist”; you can’t realistically Merklize “what should rank for this query today”…

are you saying PyPi search is spammed o-O ?

Re: PyPI in 2025: A Year in Review

#40

Post by firesteelrain »

froh wrote: post #39
firesteelrain wrote: post #37

Earlier quoted context omitted.

The install side is basically Merkle-friendly (immutable artifacts, append-only metadata, hashes, mirrors). Search isn’t. Search results are derived, subjective, and frequently rewritten (ranking tweaks, spam/malware takedowns, popularity signals). That’s more like constantly rebasing than appending commits. You can Merklize “what files exist”; you can’t realistically Merklize “what should rank for this query today”…

are you saying PyPi search is spammed o-O ?

Yes, it was subject to abuse so they had to shutdown the XML-RPC API
Post reply on HN