Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The technology behind GitHub’s new code search

github.blog

Post reply on HN
Print view View original

171–180 of 187 posts

Re: The technology behind GitHub’s new code search

#171

Post by keegancsmith »

PaulHoule wrote: post #170
kgeist wrote: post #153

Earlier quoted context omitted.

GitHub hosts 28 million public repositories. Do you think your IDE can open 28 million projects at once and search there "trivially" without hanging? Unless you're talking about searching inside a single repository?

I am talking about searching in a single repository, who would expect to get useful results otherwise? I have no idea how you’re going to rank 28 million repos in a way that matches my perception of relevance. To be specific, I was looking for the definition of one method in Highcharts so I could understand what it does and override it, GitHub gave 6 pages of results. I was able to find the function immediately in my…

Sourcegraph engineer here. I'd be interested to know what you were searching for and what your expectation for top result was. We already do things like boost class name matches higher than functions (GitHub's new search does the same) amongst other possible signals.

Re: The technology behind GitHub’s new code search

#172

Post by colin353 »

Royaljj wrote: post #146
colin353 wrote: post #26

Hey everyone, I'm Colin from GitHub's code search team: happy to answer any questions people have about it. Also, you can sign up to get access here: https://github.com/features/code-search

Hi Colin, I’m curious as to how you search repeated letters through ngram index? I understand the example search with the string “limits” (find intersection of “lim”, “imi”, “mit” and “its”). However, if the user wants to search the string “aaaaa” how would you go about searching that?

Good question. We still construct ngrams for it, exactly the same way. So for example, we might extract `aaa`, `aaa`, and `aaa`. Or we may extract `aaaa` and `aaaa`, or perhaps `aaaaa`. Then we deduplicate to find the unique ngrams and look them up in the index.

So it's possible that a document containing `aaa` might match our ngram search, but we double check after retrieving them and exclude them from the result set.

Re: The technology behind GitHub’s new code search

#173

Post by colin353 »

Manfred wrote: post #159
colin353 wrote: post #26

Hey everyone, I'm Colin from GitHub's code search team: happy to answer any questions people have about it. Also, you can sign up to get access here: https://github.com/features/code-search

Have you considered using an index directly on language tokens (eg. the abstract language tree representing the file) instead of ngrams on the source text?

We have not done this yet, but we do intend to.

Actually, our search engine is so fast that syntax highlighting the search results is often slower than finding them... so if we store the language tokens directly in the index, we'll be able to directly emit syntax highlighted snippets and make it even faster.

It may also enable some interesting search capabilities in the future, like searching within comments or by code structure.

Re: The technology behind GitHub’s new code search

#174

Post by PaulHoule »

keegancsmith wrote: post #171
PaulHoule wrote: post #170

Earlier quoted context omitted.

I am talking about searching in a single repository, who would expect to get useful results otherwise? I have no idea how you’re going to rank 28 million repos in a way that matches my perception of relevance. To be specific, I was looking for the definition of one method in Highcharts so I could understand what it does and override it, GitHub gave 6 pages of results. I was able to find the function immediately in my…

Sourcegraph engineer here. I'd be interested to know what you were searching for and what your expectation for top result was. We already do things like boost class name matches higher than functions (GitHub's new search does the same) amongst other possible signals.

I am searching this repository

https://github.com/highcharts/highcharts

for Series.drawPoint and expecting a direct hit for

https://github.com/highcharts/highcharts/blob/29d2a83a5a997b...

practically I tried "Series" and "drawPoint" also.

Re: The technology behind GitHub’s new code search

#175

Post by 100k »

Cian911 wrote: post #169

> Shard by Git blob object ID which gives us a nice way of evenly distributing documents between the shards while avoiding any duplication. There won’t be any hot servers due to special repositories and we can easily scale the number of shards as necessary. What exactly do they mean by "special repositories" here?

(I edited this post.) Maybe that could have been worded more clearly... What we mean is repositories with atypical activity patterns, for example a busy monorepo with lots of files and continuous pushes. If we sharded by repository, a single shard would be responsible for processing all the updated documents for this busy repository.

Re: The technology behind GitHub’s new code search

#176

Post by orf »

bspammer wrote: post #164
joshuawarner32 wrote: post #131

Earlier quoted context omitted.

Oooh that sounds really useful, would you mind dropping a link?

Sorry for the cyber-stalking orf, but I'm guessing it's this: https://github.com/orf/pypi-data

It's not - it's something that builds upon that. It's not really ready for prime time yet, but it will be here: https://github.com/orf/pypi-code-repo

if it works, which I really hope it does, we should have a repo containing every file (sub-25mb) published to pypi.

it's not really useful to clone this, but the git packfile + index seems to be pretty ideal for storing + querying this data, reducing 14tb of compressed packages down to like 100gb or so.

this should enable large scale exploration of the contents of pypi from only your laptop, which is useful if you want to look at how the python language is evolving (how many packages use f-strings over time?).

the rust tool i've built to do the heavy lifting is here: https://github.com/orf/pypi-import-test. i'm learning a lot about git internals, it's quite fun

Re: The technology behind GitHub’s new code search

#177

Post by interactivecode »

anitil wrote: post #109
tonymet wrote: post #21

This is a great intro / overview of full-text search for those wondering how to build your own search engine. It's a great 101-level exercise to write an inverted index implementation you can do it in an afternoon , and then expand to a leaf /aggregator in follow-up exercises.

I did this for our organization using sqlite's FTS module and datasette and boy was it fast. Unfortunately I did get (temporarily) banned from the organisations github account, but it was definitely worth it. Even now I find myself using it despite the index being a few months out of date.

I would love to see how to do something like this. Are you open to sharing your implementation?

Re: The technology behind GitHub’s new code search

#178

Post by shurane »

Xeophon wrote: post #99
IshKebab wrote: post #95

Earlier quoted context omitted.

Actually I've been using https://grep.app for ages and while I agree on GitHub I basically only search the repo I'm in, that's mainly because Github's existing search sucks. On grep.app I regularly search all repos. It's very useful for finding out how to use APIs or where APIs from dependencies are defined. So I suspect you don't want it because subconsciously you know that Github's "search all" feature won't return…

Recently discovered grep.app and it is incredibly useful. Wish I’ve knew it earlier.

This is pretty neat. Feels like real time full text search. The user interface really reminds me of https://hn.algolia.com as well.

Re: The technology behind GitHub’s new code search

#179

Post by tjoff »

dcreager wrote: post #166
tjoff wrote: post #36

Earlier quoted context omitted.

The rise and popularity of LSP and projects such as treesitter are a superb foundation for features such as this. Both support a wealth of languages, it is still and will be quite hard to assume the toolchain and settings required for producing accurate information though. But this could be tied into CI, especially for projects utilizing runners for building the code. So the barrier to entry now is orders of magnitud…

> But this could be tied into CI, especially for projects utilizing runners for building the code. One often overlooked drawback of generating this data during CI is that you, the project owner, are now paying for the compute. Of course, maybe you qualify for a free tier. And if not, if you're already running a CI job for testing/linting/etc, the marginal cost of also generating code nav symbols might not be too bad.…

By necessity I already do it many times a day on my workstation. It would be neat for the cloud to start utilizing the endless compute available on client machines. A back-channel when doing a git push to publish some related artifacts?

Re: The technology behind GitHub’s new code search

#180

Post by robertlagrant »

dcreager wrote: post #168
robertlagrant wrote: post #67

Earlier quoted context omitted.

The other big thing that works well is being able to jump directly into the source of an open source library from your code. That is powerful, but again, possibly doesn't need a giant ultra search. Just some clever linking.

This sounds like Cross-Repo Code Navigation [1], which we do support, though only for Python at the moment. And you're right that it does not use the same search index under the covers — OP is specifically describing the index for "search box"-style code search across a large set of repositories (org-scoped, global-scoped, me-scoped, etc). For Code Navigation, we have a _different_ set of interesting and bespoke inde…

Thanks! That's a really fascinating search graph.
Post reply on HN