Live data from Hacker News

The technology behind GitHub’s new code search

github.blog

21–30 of 187 posts

Re: The technology behind GitHub’s new code search

#21
This is a great intro / overview of full-text search for those wondering how to build your own search engine.

It's a great 101-level exercise to write an inverted index implementation you can do it in an afternoon , and then expand to a leaf /aggregator in follow-up exercises.

Re: The technology behind GitHub’s new code search

#22
post #13

Sourcegraph should’ve accepted that offer from GitHub.

I don't think Sourcegraph is in big trouble here. Their whole play is enterprises, who likely have code spread across many different hosts. On top of that, their code search is still miles ahead of GitHub's.

The new code search includes private repos. Miles ahead doesn’t matter if you get the search at GitHub for free. GitHub actions started out pretty terrible but they are now dominating hosted CI.

SourceGraph likely has challenging times ahead considering the valuation.

Re: The technology behind GitHub’s new code search

#23
post #3

I use their new code search a lot to grok how people use certain features, or implement certain things. But I do wish there was a way to filter out forks. Sometimes I search a string and just get a bunch of forks all with the same result. For example, searching a common class in a Rails app often just shows a bunch of rails/rails forks, which is a lot of noise to sift through when you're trying to see how devs common…

As an open source library maintainer I've been using it for that too - it's fantastic for answering questions like "is anyone using this API method I wrote?" and "how much of a mess would it likely cause people if I deprecated this function?"

Re: The technology behind GitHub’s new code search

#24
post #15

I really like the new search. Though sometimes it is a bit deceptive. I.e. when searching for a function name by clicking on a piece of code and suddenly you are in an entitely different code base with an unrelated function though it shares the name. It feels like github code browsing is a step between a full editor with lsp and a static site. I Hope they work out the Kinks and make it more smooth

Supporting jump-to-definition natively seems like something that will be table stakes for any code hosting site in the future.

Re: The technology behind GitHub’s new code search

#25
post #24
post #15

I really like the new search. Though sometimes it is a bit deceptive. I.e. when searching for a function name by clicking on a piece of code and suddenly you are in an entitely different code base with an unrelated function though it shares the name. It feels like github code browsing is a step between a full editor with lsp and a static site. I Hope they work out the Kinks and make it more smooth

Supporting jump-to-definition natively seems like something that will be table stakes for any code hosting site in the future.

I can only think of one hosted repo service that provides a working go-to-definition feature and it is not github or sourcegraph. What makes you think this will suddenly become widespread? Github spent years doing this and their new thing is strictly non-semantic. It doesn't have the faintest idea where the name is defined, or if there's even a difference between a function name, a parameter name, or a word in a comment.

Re: The technology behind GitHub’s new code search

#27
I really appreciate that this includes details about how search permissions work - how they ensure that search results include data from my private repos.

I'd always wondered how they implemented that: it turns out they add extra internal filters to their searches along the lines of "RepoIDs(...) or PublicRepo".

Question for the team: Do you have an additional permission check in the view layer before the results are shown to the end-user? I worry that if I switch a repo from public to private it may take a while for the code search index to catch up to the new permissions.

Re: The technology behind GitHub’s new code search

#28
post #21

This is a great intro / overview of full-text search for those wondering how to build your own search engine. It's a great 101-level exercise to write an inverted index implementation you can do it in an afternoon , and then expand to a leaf /aggregator in follow-up exercises.

It is indeed Information Retrieval 101 level stuff which leads to the question of why this is the best GitHub can do with all the resources of Microsoft behind them. It's almost useless, at least for C++. It can't tell the difference between foo(int) and foo(double) or this::foo vs. that::foo.

If I wanted the kind of search engine I can get a teenager to write in 16 weeks why would I expect my org to be paying $$$ for the service?

Re: The technology behind GitHub’s new code search

#29
post #7
post #3

I use their new code search a lot to grok how people use certain features, or implement certain things. But I do wish there was a way to filter out forks. Sometimes I search a string and just get a bunch of forks all with the same result. For example, searching a common class in a Rails app often just shows a bunch of rails/rails forks, which is a lot of noise to sift through when you're trying to see how devs common…

Thanks for the feedback! That's coming, we've been prioritizing scaling the index and ingest process and haven't had a chance got add that yet. There are a bunch of value-add features like this I am looking forward to knocking out soon.

Well done
Post reply on HN