> This is a much more substantial critique
Thanks! FWIW, I engaged because I think there's the basis for a good approach here and I want to see if we can expose what works through discussion.
> why is cgit scanning 340 MB to answer this question? Is cgit not using good indexes?
That's the critical question (ugh, as I write this I realize I'm starting to sound like an AI, sorry). It comes down (I think, I don't work on kernel.org) to a few things:
- cgit is ubiquitous and easy to deploy.
- Cgit doesn't have a database, or indexes. It's a fairly thin wrapper over git itself. Git itself is optimized for performance of local filesystem operations, which is nice, but means it can be written with e.g. the page cache in mind for performance, and doesn't need to spend a ton of time thinking about the number of files it accesses--after all, they're all small and locally-available, and thus probably cached.
- Things like cgit often have to combine (the C equivalents of) multiple chained git commands to render certain views.
- Cgit has caching, but it's simultaneously too naïve and too specific to handle use-cases like this. It's too naive in that it's caching content blobs via the filesystem (and using sendfile to serve them) without awareness of e.g. shared cache blocks for deduplication or anything, so it can't cache intermediate states used for rendering e.g. diffs. It's too specific in that it's only caching the results of certain underlying git operations and specified renders, so it can't prioritize e.g. cross-application LRU/frecency patterns when deciding what to cache and what to evict.
In short, what you're proposing is probably best implemented server-side (and hopefully already exists): it's an indexed database of Git's objects that has behavior parity with Git's rendering of the various views/questions people can make of the server. That's a nontrivial undertaking: git doesn't have a VFS layer you could plug a database into the way something like SQLite does; it very strongly prefers procedural file-based accesses to its data. If you could use something like that, you could engineer it like a typical webapp: content-unaware caching layers and indexes to make certain operations fast. Not easy to build, and not easy to swap in in place of cgit (which you also probably have to keep a lot of behavior parity with), though.