The technology behind GitHub’s new code search

Post by jpgvm » Tue, Feb 07, 2023, 4:41 AM UTC

100k wrote: post #7

ezekg wrote: post #3

I use their new code search a lot to grok how people use certain features, or implement certain things. But I do wish there was a way to filter out forks. Sometimes I search a string and just get a bunch of forks all with the same result. For example, searching a common class in a Rails app often just shows a bunch of rails/rails forks, which is a lot of noise to sift through when you're trying to see how devs common…

Thanks for the feedback! That's coming, we've been prioritizing scaling the index and ingest process and haven't had a chance got add that yet. There are a bunch of value-add features like this I am looking forward to knocking out soon.

Post by semiquaver » Tue, Feb 07, 2023, 5:11 AM UTC

PaulHoule wrote: post #68

My beef with GitHub's code search is that it doesn't distinguish between the definition of a symbol and the uses of the symbol, so you need to wade through 5 pages of results to get the one result you're looking for. I would contrast that to my IDE which usually scores a direct hit if I enter a search in the right box. The indexing they talk about in that article seems like rearranging the deck chairs on the Titanic…

    /func.+MyFunc/

    /\.MyFunc/

Post by nickysielicki » Tue, Feb 07, 2023, 5:43 AM UTC

100k wrote: post #83

andrewmcwatters wrote: post #76

> Just use grep? First though, let’s explore the brute force approach to the problem. We get this question a lot: “Why don’t you just use grep?” To answer that, let’s do a little napkin math using ripgrep on that 115 TB of content. On a machine with an eight core Intel CPU, ripgrep can run an exhaustive regular expression query on a 13 GB file cached in memory in 2.769 seconds, or about 0.6 GB/sec/core. But you don't…

(I work on this.) If you check out our prior blog post, "A brief history of code search at GitHub" ( https://github.blog/2021-12-15-a-brief-history-of-code-searc... ), you can learn a bit about the evolution of this feature. And, in fact, we used to use git grep to search repositories. This doesn't work well at GitHub's scale. We have 100M users and over 200M repositories in a multi-tenant environment. Your git grep…

Post by __float » Tue, Feb 07, 2023, 6:34 AM UTC

Scaevolus wrote: post #20

As a comparison to Sourcegraph: Sourcegraph shards and indexes a repository at a time, and uses trigrams and bloom filters (to skip shards). Github shards and indexes individual files according to their hashes. It also uses variable length ngrams (neat!). This makes horizontal scaling simpler, but also means more of the index needs to be scanned for org/repo-scoped queries ("Due to our sharding strategy, a query requ…

Post by __float » Tue, Feb 07, 2023, 6:45 AM UTC

jeffbee wrote: post #28

tonymet wrote: post #21

This is a great intro / overview of full-text search for those wondering how to build your own search engine. It's a great 101-level exercise to write an inverted index implementation you can do it in an afternoon , and then expand to a leaf /aggregator in follow-up exercises.

It is indeed Information Retrieval 101 level stuff which leads to the question of why this is the best GitHub can do with all the resources of Microsoft behind them. It's almost useless, at least for C++. It can't tell the difference between foo(int) and foo(double) or this::foo vs. that::foo. If I wanted the kind of search engine I can get a teenager to write in 16 weeks why would I expect my org to be paying $$$ fo…