Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The technology behind GitHub’s new code search

github.blog

Post reply on HN
Print view View original

111–120 of 187 posts

Re: The technology behind GitHub’s new code search

#111

Post by sqs »

lern_too_spel wrote: post #42
imadethis wrote: post #2

Sourcegraph should’ve accepted that offer from GitHub.

Sourcegraph had to have known GitHub would do this if they didn't accept the offer. Since this should be expected, the launch of this feature shouldn't change what their decision should have been.

Sourcegraph CEO here. Just to be clear, so internet rumors don’t get started, there was no “offer” here. We started Sourcegraph with the intent of remaining independent because building really good code search and intelligence means working across all code (not just on GitHub), all devs, and all code intelligence sources (code nav plus every dev tool you use that knows stuff about code, not just the ones in the GitHub/Microsoft suite bundle). We’ve never entertained any kind of acquisition interest for this reason.

We don’t think any of today’s code host vendors with their current strategies can make truly great code search and intelligence because they’ll be biased toward their own bundled tools and limited to the subset of code hosted on that instance. It’d be kind of like Encyclopedia Britannica or The NY Times building a web search engine: helpful, but so much more limited compared to what the independent Google became.

And yes, none of this was a surprise. GitHub’s new code search has been out for 14 months now.

OK, hope this puts an internet rumor to rest!

Re: The technology behind GitHub’s new code search

#112

Post by andrewmcwatters »

IshKebab wrote: post #95
andrewmcwatters wrote: post #76

> Just use grep? First though, let’s explore the brute force approach to the problem. We get this question a lot: “Why don’t you just use grep?” To answer that, let’s do a little napkin math using ripgrep on that 115 TB of content. On a machine with an eight core Intel CPU, ripgrep can run an exhaustive regular expression query on a 13 GB file cached in memory in 2.769 seconds, or about 0.6 GB/sec/core. But you don't…

Actually I've been using https://grep.app for ages and while I agree on GitHub I basically only search the repo I'm in, that's mainly because Github's existing search sucks. On grep.app I regularly search all repos. It's very useful for finding out how to use APIs or where APIs from dependencies are defined. So I suspect you don't want it because subconsciously you know that Github's "search all" feature won't return…

Thank you for sharing. This is basically what I want.

Re: The technology behind GitHub’s new code search

#113

Post by j1elo »

Will this allow for a happy closure of this question about searching partial words? [0]

Like searching for "OPTION" and getting "-DOPTION=TRUE" among the results. Very commonly needed to find all usages of a flag, even instances where the flag is being passed to (at least, that I know of) CMake and Meson.

[0]: https://stackoverflow.com/questions/43891605/search-partial-...

Re: The technology behind GitHub’s new code search

#114

Post by damidekronik »

ezekg wrote: post #3

I use their new code search a lot to grok how people use certain features, or implement certain things. But I do wish there was a way to filter out forks. Sometimes I search a string and just get a bunch of forks all with the same result. For example, searching a common class in a Rails app often just shows a bunch of rails/rails forks, which is a lot of noise to sift through when you're trying to see how devs common…

When trying out some C++ I was surprised by how many projects bundled the library source code, essentially finding any actual client code a pain.

Re: The technology behind GitHub’s new code search

#115

Post by dymk »

slimsag wrote: post #79
dymk wrote: post #38

Earlier quoted context omitted.

I think that there should be some sort of standardized, language-agnostic metadata format for semantically indexing a codebase. It could include e.g type information for expressions or declared variables (for languages that infer types), and an index of symbols and how they're connected. This metadata file would be generated by a language-specific tool. For instance, Cargo could generate it for Rust projects, ctags/c…

This is pretty much exactly what we've built at Sourcegraph. Microsoft had introduced (but pretty much abandoned before it even started) LSIF, a static index format for LSP servers which encodes in detail all possible LSP requests/responses, effectively. We took that torch and carried it forward, building the spiritual successor called SCIP[0]. It's language agnostic, we have indexers for quite a few languages alread…

This is pretty much exactly what I'd imagined, very cool. Is the plan to open source all SCIP indexers?

Re: The technology behind GitHub’s new code search

#116

Post by saurik »

damidekronik wrote: post #114
ezekg wrote: post #3

I use their new code search a lot to grok how people use certain features, or implement certain things. But I do wish there was a way to filter out forks. Sometimes I search a string and just get a bunch of forks all with the same result. For example, searching a common class in a Rails app often just shows a bunch of rails/rails forks, which is a lot of noise to sift through when you're trying to see how devs common…

When trying out some C++ I was surprised by how many projects bundled the library source code, essentially finding any actual client code a pain.

As a C++ developer, I maybe-ironically have this same complaint with JavaScript projects ;P. I particularly hate it when people embed minified versions of libraries, as line-based search and display then will experience a hit for essentially every query on the one giant line that is tens of kilocharacters long.

Re: The technology behind GitHub’s new code search

#117

Post by xavdid »

DontchaKnowit wrote: post #105
PaulHoule wrote: post #68

My beef with GitHub's code search is that it doesn't distinguish between the definition of a symbol and the uses of the symbol, so you need to wade through 5 pages of results to get the one result you're looking for. I would contrast that to my IDE which usually scores a direct hit if I enter a search in the right box. The indexing they talk about in that article seems like rearranging the deck chairs on the Titanic…

Thank you. This is SUCH an obvious feature that seems super trivial to implement.

I'm not sure how to read the post and come away thinking any of this is "super trivial", but I'd imagine if it were that easy, they would have done it already.

Re: The technology behind GitHub’s new code search

#118

Post by saurik »

tiagod wrote: post #92
andrewmcwatters wrote: post #76

> Just use grep? First though, let’s explore the brute force approach to the problem. We get this question a lot: “Why don’t you just use grep?” To answer that, let’s do a little napkin math using ripgrep on that 115 TB of content. On a machine with an eight core Intel CPU, ripgrep can run an exhaustive regular expression query on a 13 GB file cached in memory in 2.769 seconds, or about 0.6 GB/sec/core. But you don't…

I use github-wide searches all the time to see how people are using certain APIs, to find libraries used in some blob from the strings I find there, to find people working with the same data I'm about to attempt to work it, and the list goes on. What you use github search for doesn't require all this engineering, but what I use it for does. Why wouldn't they build something that satisfies both our necessities well?

Same. I'll go even further and say that the narrower the search the less useful it is, as if I want to search a single repository I can and will just download it as local grep is so much better than remote grep it is painful. They could make a really amazing remote search for a repository--with a find-as-you-type low-latency display that supported language-specific syntax tree queries--but it would require way too many resources and so they are never going to bother. The result: if I am searching on GitHub at all, it is probably a full-site query, and I absolutely do use full-organization searches to figure out what repository to download as so many people now do extremely fine-grained linked repositories (though, in my experience, GitHub's search for this purpose sucks... maybe it is better now).

Re: The technology behind GitHub’s new code search

#119

Post by muchospandas »

andrewmcwatters wrote: post #76

> Just use grep? First though, let’s explore the brute force approach to the problem. We get this question a lot: “Why don’t you just use grep?” To answer that, let’s do a little napkin math using ripgrep on that 115 TB of content. On a machine with an eight core Intel CPU, ripgrep can run an exhaustive regular expression query on a 13 GB file cached in memory in 2.769 seconds, or about 0.6 GB/sec/core. But you don't…

If I want to search locally, I'll use some tool to search my checked-out code.

I used to search my org's code on a daily basis and all of GitHub a few times a week at least, back when I used git for work.

I think you may be overestimating how common your specific use-case is.

Re: The technology behind GitHub’s new code search

#120

Post by 100k »

ZephyrBlu wrote: post #90
100k wrote: post #80

Earlier quoted context omitted.

I agree! Search is so cool. It's interesting that GitHub seems to have quite a few shards. Algolia basically has a monolithic architecture with 3 different hosts I used to work at an Algolia competitor. I don't know for sure, but my guess is that Algolia shards their indices by customer. Algolia does not provide global search. GitHub code search does. That, and the desire to deduplicate data, is what led us to our cu…

I'm not familiar with production search systems at scale (Very curious about them though). How do you think Algolia shards their data given that architecture? Based on their description it seems like the search engine itself is monolithic. Maybe they're running a 3-node cluster with a monolithic index for each customer? Interesting, do you keep a copy of the index document form of repos or is that done on the fly dur…

How do you think Algolia shards their data given that architecture?

My guess is that Algolia's indices are sharded by customer and each cluster probably has multiple customer indices.

do you keep a copy of the index document form of repos or is that done on the fly during indexing?

As mentioned in the post, the index contains the full content. Our ingest process essentially flattens git repos (which are stored as DAGs) into a list of documents to index (the prior state is diffed for changes).

Is your custom index format a binary format? I have no idea whether that's standard practice, or just a compressed text format is enough. I guess that non-binary formats would be enormous though, and given that an index is by definition relatively unique it probably wouldn't compress that well.

Binary formats are normal, posting lists are giant sorted blocks of numbers, so there are a lot of techniques that can be used to compress them. Lucene's index format is pretty well documented if you're interested in learning more (interestingly, Lucene has a text format for debugging: https://lucene.apache.org/core/8_6_3/codecs/org/apache/lucen...).

Scale up in terms of what? Is the current system not indexing all of GitHub, or you mean you want to index on more things (E.g. commits, PRs, etc)?

It's not indexing all of GitHub yet, nor do all users have access yet. Those are the things we are focusing on now. In the future, we want to support indexing branches.

Post reply on HN