Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The technology behind GitHub’s new code search

github.blog

Post reply on HN
Print view View original

31–40 of 187 posts

Re: The technology behind GitHub’s new code search

#31

Post by 100k »

jeffbee wrote: post #28
tonymet wrote: post #21

This is a great intro / overview of full-text search for those wondering how to build your own search engine. It's a great 101-level exercise to write an inverted index implementation you can do it in an afternoon , and then expand to a leaf /aggregator in follow-up exercises.

It is indeed Information Retrieval 101 level stuff which leads to the question of why this is the best GitHub can do with all the resources of Microsoft behind them. It's almost useless, at least for C++. It can't tell the difference between foo(int) and foo(double) or this::foo vs. that::foo. If I wanted the kind of search engine I can get a teenager to write in 16 weeks why would I expect my org to be paying $$$ fo…

Have you tried the new search? Thanks to the variable length ngram indexing mentioned in the post, it can handle all of those cases. Sign up here to try it: https://github.com/features/code-search

Symbol extraction for C and C++ is currently disabled because we were having problems with the performance of the tree-sitter queries we were using, but we are planning to bring that back.

Re: The technology behind GitHub’s new code search

#32

Post by burntsushi »

jeffbee wrote: post #28
tonymet wrote: post #21

This is a great intro / overview of full-text search for those wondering how to build your own search engine. It's a great 101-level exercise to write an inverted index implementation you can do it in an afternoon , and then expand to a leaf /aggregator in follow-up exercises.

It is indeed Information Retrieval 101 level stuff which leads to the question of why this is the best GitHub can do with all the resources of Microsoft behind them. It's almost useless, at least for C++. It can't tell the difference between foo(int) and foo(double) or this::foo vs. that::foo. If I wanted the kind of search engine I can get a teenager to write in 16 weeks why would I expect my org to be paying $$$ fo…

What a shit take. The article itself is perhaps a nice light overview of 101-ish level concepts, although knowing how and when to apply them in a real engineering context is not something I would consider 101 level. And certainly, building something that is actually at the scale of GitHub Search is nowhere near 101 level.

This is what a 101-level inverted index implementation looks like: https://github.com/BurntSushi/imdb-rename

In other words, absolutely nothing like what GitHub built. Nowhere close.

Re: The technology behind GitHub’s new code search

#33

Post by tclem »

simonw wrote: post #27

I really appreciate that this includes details about how search permissions work - how they ensure that search results include data from my private repos. I'd always wondered how they implemented that: it turns out they add extra internal filters to their searches along the lines of "RepoIDs(...) or PublicRepo". Question for the team: Do you have an additional permission check in the view layer before the results are…

Yes, we never fully trust the search index so before anything is displayed to the user there are a number of final checks performed to make sure you're actually allowed to see that content.

Another fun example is that your SSO session might have expired. While you technically have access to view the result, we can't show it until you go through the refresh dance to get another valid token.

Re: The technology behind GitHub’s new code search

#34

Post by slimsag »

jeffbee wrote: post #25
kibwen wrote: post #24

Earlier quoted context omitted.

Supporting jump-to-definition natively seems like something that will be table stakes for any code hosting site in the future.

I can only think of one hosted repo service that provides a working go-to-definition feature and it is not github or sourcegraph. What makes you think this will suddenly become widespread? Github spent years doing this and their new thing is strictly non-semantic. It doesn't have the faintest idea where the name is defined, or if there's even a difference between a function name, a parameter name, or a word in a comm…

FWIW Sourcegraph has fully precise/semantic go-to-definition, find-references, etc. We use SCIP code indexers (a spiritual successor to LSIF, the Microsoft standard for indexing LSP servers)

Re: The technology behind GitHub’s new code search

#35

Post by jeffbee »

100k wrote: post #31
jeffbee wrote: post #28

Earlier quoted context omitted.

It is indeed Information Retrieval 101 level stuff which leads to the question of why this is the best GitHub can do with all the resources of Microsoft behind them. It's almost useless, at least for C++. It can't tell the difference between foo(int) and foo(double) or this::foo vs. that::foo. If I wanted the kind of search engine I can get a teenager to write in 16 weeks why would I expect my org to be paying $$$ fo…

Have you tried the new search? Thanks to the variable length ngram indexing mentioned in the post, it can handle all of those cases. Sign up here to try it: https://github.com/features/code-search Symbol extraction for C and C++ is currently disabled because we were having problems with the performance of the tree-sitter queries we were using, but we are planning to bring that back.

Sorry, it cannot handle any of those cases. You're talking about the ability to find the literal `this::foo` but that's not how it would normally appear. It normally will appear anywhere inside a `namespace this` scope, which cs.github does not grok. And cs.github cannot address finding the definition related to a given call site. It doesn't even try.

Re: The technology behind GitHub’s new code search

#36

Post by tjoff »

jeffbee wrote: post #25
kibwen wrote: post #24

Earlier quoted context omitted.

Supporting jump-to-definition natively seems like something that will be table stakes for any code hosting site in the future.

I can only think of one hosted repo service that provides a working go-to-definition feature and it is not github or sourcegraph. What makes you think this will suddenly become widespread? Github spent years doing this and their new thing is strictly non-semantic. It doesn't have the faintest idea where the name is defined, or if there's even a difference between a function name, a parameter name, or a word in a comm…

The rise and popularity of LSP and projects such as treesitter are a superb foundation for features such as this. Both support a wealth of languages, it is still and will be quite hard to assume the toolchain and settings required for producing accurate information though.

But this could be tied into CI, especially for projects utilizing runners for building the code.

So the barrier to entry now is orders of magnitude less and with github and others pushing for codespaces it could be the final piece to tie everything together.

Re: The technology behind GitHub’s new code search

#37

Post by 100k »

jeffbee wrote: post #35
100k wrote: post #31

Earlier quoted context omitted.

Have you tried the new search? Thanks to the variable length ngram indexing mentioned in the post, it can handle all of those cases. Sign up here to try it: https://github.com/features/code-search Symbol extraction for C and C++ is currently disabled because we were having problems with the performance of the tree-sitter queries we were using, but we are planning to bring that back.

Sorry, it cannot handle any of those cases. You're talking about the ability to find the literal `this::foo` but that's not how it would normally appear. It normally will appear anywhere inside a `namespace this` scope, which cs.github does not grok. And cs.github cannot address finding the definition related to a given call site. It doesn't even try.

You are correct, as I mentioned, we do not analyze symbols for C and C++ at this time.

Re: The technology behind GitHub’s new code search

#38

Post by dymk »

kibwen wrote: post #24
kjuulh wrote: post #15

I really like the new search. Though sometimes it is a bit deceptive. I.e. when searching for a function name by clicking on a piece of code and suddenly you are in an entitely different code base with an unrelated function though it shares the name. It feels like github code browsing is a step between a full editor with lsp and a static site. I Hope they work out the Kinks and make it more smooth

Supporting jump-to-definition natively seems like something that will be table stakes for any code hosting site in the future.

I think that there should be some sort of standardized, language-agnostic metadata format for semantically indexing a codebase. It could include e.g type information for expressions or declared variables (for languages that infer types), and an index of symbols and how they're connected.

This metadata file would be generated by a language-specific tool. For instance, Cargo could generate it for Rust projects, ctags/cmake for C / C++, Sorbet for Ruby.

Then a service like Github / Gitlab / your own homegrown code viewer could provide things like "Show type", "Jump to source" without ever needing to build a language-specific parser or interpreter (which seems arbitrarily difficult, most build systems provide escape hatches, so you can't assume much about project structure).

Basically, LSP as a static file in a standard format that tools can read to understand a codebase, without needing to model the language's semantics.

Re: The technology behind GitHub’s new code search

#39

Post by qabqabaca »

ezekg wrote: post #3

I use their new code search a lot to grok how people use certain features, or implement certain things. But I do wish there was a way to filter out forks. Sometimes I search a string and just get a bunch of forks all with the same result. For example, searching a common class in a Rails app often just shows a bunch of rails/rails forks, which is a lot of noise to sift through when you're trying to see how devs common…

Sourcegraph[1] does this better and has done for a couple of years now. I use it for this reason all the time.

[1] https://sourcegraph.com

Re: The technology behind GitHub’s new code search

#40

Post by hobofan »

jeffbee wrote: post #25
kibwen wrote: post #24

Earlier quoted context omitted.

Supporting jump-to-definition natively seems like something that will be table stakes for any code hosting site in the future.

I can only think of one hosted repo service that provides a working go-to-definition feature and it is not github or sourcegraph. What makes you think this will suddenly become widespread? Github spent years doing this and their new thing is strictly non-semantic. It doesn't have the faintest idea where the name is defined, or if there's even a difference between a function name, a parameter name, or a word in a comm…

> It doesn't have the faintest idea where the name is defined, or if there's even a difference between a function name, a parameter name, or a word in a comment.

I don't think what you are saying is actually true for stack-graphs[0][1].

[0]: https://github.com/github/stack-graphs

[1]: https://github.blog/2021-12-09-introducing-stack-graphs/

Post reply on HN