Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The technology behind GitHub’s new code search

github.blog

Post reply on HN
Print view View original

151–160 of 187 posts

Re: The technology behind GitHub’s new code search

#151

Post by typesanitizer »

dymk wrote: post #115
slimsag wrote: post #79

Earlier quoted context omitted.

This is pretty much exactly what we've built at Sourcegraph. Microsoft had introduced (but pretty much abandoned before it even started) LSIF, a static index format for LSP servers which encodes in detail all possible LSP requests/responses, effectively. We took that torch and carried it forward, building the spiritual successor called SCIP[0]. It's language agnostic, we have indexers for quite a few languages alread…

This is pretty much exactly what I'd imagined, very cool. Is the plan to open source all SCIP indexers?

All of our SCIP indexers are open-source: scip-java (for Java, Kotlin and Scala), scip-typescript (for TypeScript and JavaScript), scip-python, scip-ruby, scip-go and scip-clang (for C and C++).

There are also some community-maintained OSS SCIP indexers. - rust-analyzer: https://sourcegraph.com/github.com/rust-lang/rust-analyzer/-... - scip-zig: https://github.com/zigtools/scip-zig

Re: The technology behind GitHub’s new code search

#152

Post by purkka »

My biggest feature request would be sorting or filtering by code/commit/repo age, or even repo stars.

Most often I end up using code search for figuring out where a piece of code originated, just to find thousands of random projects that have also copied the same code verbatim. Sorting for "relevance" or "latest/oldest indexed" are equally useless.

Re: The technology behind GitHub’s new code search

#153

Post by kgeist »

Zuiii wrote: post #136
xavdid wrote: post #117

Earlier quoted context omitted.

I'm not sure how to read the post and come away thinking any of this is "super trivial", but I'd imagine if it were that easy, they would have done it already.

It's simple enough that most IDEs implement it. Take from that what you will.

GitHub hosts 28 million public repositories. Do you think your IDE can open 28 million projects at once and search there "trivially" without hanging? Unless you're talking about searching inside a single repository?

Re: The technology behind GitHub’s new code search

#156

Post by [deleted] »

100k wrote: post #83
andrewmcwatters wrote: post #76

> Just use grep? First though, let’s explore the brute force approach to the problem. We get this question a lot: “Why don’t you just use grep?” To answer that, let’s do a little napkin math using ripgrep on that 115 TB of content. On a machine with an eight core Intel CPU, ripgrep can run an exhaustive regular expression query on a 13 GB file cached in memory in 2.769 seconds, or about 0.6 GB/sec/core. But you don't…

(I work on this.) If you check out our prior blog post, "A brief history of code search at GitHub" ( https://github.blog/2021-12-15-a-brief-history-of-code-searc... ), you can learn a bit about the evolution of this feature. And, in fact, we used to use git grep to search repositories. This doesn't work well at GitHub's scale. We have 100M users and over 200M repositories in a multi-tenant environment. Your git grep…

[deleted]

Re: The technology behind GitHub’s new code search

#157

Post by therealdrag0 »

boyter wrote: post #47

The sparse grams solution to deal with stupidly common ngrams such as for or tes is very interesting. I’d love to see more discussion on how they are dealing with the false positives though. It looks like a positional index is being used to achieve this, but that usually blows out your index size. Additional information about deduplication would be especially interesting to me as well. It seems to solve this quite we…

> What I find really interesting about all the code search engines I know of is that each one implemented its own index

I mean GH got a long way using ElasticSearch until now.

Re: The technology behind GitHub’s new code search

#159

Post by Manfred »

colin353 wrote: post #26

Hey everyone, I'm Colin from GitHub's code search team: happy to answer any questions people have about it. Also, you can sign up to get access here: https://github.com/features/code-search

Have you considered using an index directly on language tokens (eg. the abstract language tree representing the file) instead of ngrams on the source text?

Re: The technology behind GitHub’s new code search

#160

Post by bluehatbrit »

ZephyrBlu wrote: post #4

I really hope they release this soon and that it’s actually good. The current search sucks ass, you can’t find anything. I was trying to search for something in the WebKit source the other day and I had to use Sourcegraph because the GitHub search gave me zero results.

I've been using it since it entered beta a few months back. It's a huge improvement, still not perfect but it's actually usable now, and I don't just give up with it ever time.
Post reply on HN