Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The technology behind GitHub’s new code search

github.blog

Post reply on HN
Print view View original

51–60 of 187 posts

Re: The technology behind GitHub’s new code search

#51

Post by Arnavion »

qabqabaca wrote: post #39
ezekg wrote: post #3

I use their new code search a lot to grok how people use certain features, or implement certain things. But I do wish there was a way to filter out forks. Sometimes I search a string and just get a bunch of forks all with the same result. For example, searching a common class in a Rails app often just shows a bunch of rails/rails forks, which is a lot of noise to sift through when you're trying to see how devs common…

Sourcegraph[1] does this better and has done for a couple of years now. I use it for this reason all the time. [1] https://sourcegraph.com

Yes, just change the URL from https://github.com/foo/bar to https://sourcegraph.com/github.com/foo/bar to be dropped in to a code search for that GH repo.

Re: The technology behind GitHub’s new code search

#52

Post by mnutt »

ezekg wrote: post #3

I use their new code search a lot to grok how people use certain features, or implement certain things. But I do wish there was a way to filter out forks. Sometimes I search a string and just get a bunch of forks all with the same result. For example, searching a common class in a Rails app often just shows a bunch of rails/rails forks, which is a lot of noise to sift through when you're trying to see how devs common…

I run into this a lot where I'd like to see some real world use cases of a method in FooClass, but get a hundred pages of forks of the FooClass.h header file. In some cases I've been successful adding "-filename:FooClass.h" to filter out that header file. If the method is used a lot in the primary project it can be a game of whack-a-mole but it often eventually works.

Re: The technology behind GitHub’s new code search

#53

Post by ZephyrBlu »

boyter wrote: post #47

The sparse grams solution to deal with stupidly common ngrams such as for or tes is very interesting. I’d love to see more discussion on how they are dealing with the false positives though. It looks like a positional index is being used to achieve this, but that usually blows out your index size. Additional information about deduplication would be especially interesting to me as well. It seems to solve this quite we…

Implementing your own index gives you more control over it. I think at this scale you probably want to tweak things specifically to your product rather than using a generic solution. I would guess that what you're indexing on (E.g. language, file, repo, etc) and sharding strategy affects the structure of your index as well.

Re: The technology behind GitHub’s new code search

#55

Post by slimsag »

jeffbee wrote: post #41
slimsag wrote: post #34

Earlier quoted context omitted.

FWIW Sourcegraph has fully precise/semantic go-to-definition, find-references, etc. We use SCIP code indexers (a spiritual successor to LSIF, the Microsoft standard for indexing LSP servers)

Not for C++. To test my recollection I navigated to abseil-cpp/strings/str_split.h, clicked on the declaration of absl::ByString::Find, and clicked "Go to definition". I was presented with every function in Abseil named "Find" regardless of its scope or parameter types. That's not "precise code intelligence"!

In the top right corner of the tooltip it will say either "Search-based" or "Precise" - in this case, you're right, we don't have the abseil-cpp repo indexed so it falls back to search-based as you describe.

We do have a C++ code indexer in beta, https://github.com/sourcegraph/lsif-clang - it is based on clang but C++ indexing is notably harder to do automatically/without-setup due to the varying build systems that need to be understood in order to invoke the compiler.

Re: The technology behind GitHub’s new code search

#56

Post by boyter »

ZephyrBlu wrote: post #53
boyter wrote: post #47

The sparse grams solution to deal with stupidly common ngrams such as for or tes is very interesting. I’d love to see more discussion on how they are dealing with the false positives though. It looks like a positional index is being used to achieve this, but that usually blows out your index size. Additional information about deduplication would be especially interesting to me as well. It seems to solve this quite we…

Implementing your own index gives you more control over it. I think at this scale you probably want to tweak things specifically to your product rather than using a generic solution. I would guess that what you're indexing on (E.g. language, file, repo, etc) and sharding strategy affects the structure of your index as well.

Believe me I am aware. I am one of those who implemented their own index for a code search engine :) I did it for my own learning, but find it interesting because something like elastic with trigrams can get you very close, albeit at a far greater cost.

Re: The technology behind GitHub’s new code search

#57

Post by saagarjha »

jeffbee wrote: post #25
kibwen wrote: post #24

Earlier quoted context omitted.

Supporting jump-to-definition natively seems like something that will be table stakes for any code hosting site in the future.

I can only think of one hosted repo service that provides a working go-to-definition feature and it is not github or sourcegraph. What makes you think this will suddenly become widespread? Github spent years doing this and their new thing is strictly non-semantic. It doesn't have the faintest idea where the name is defined, or if there's even a difference between a function name, a parameter name, or a word in a comm…

I can think of zero. Doing go-to-definition statically is very difficult.

Re: The technology behind GitHub’s new code search

#58

Post by 100k »

Daffodils wrote: post #54

Was looking for more details on the data structure 'Geometric filter' mentioned in the footnotes. Couldn't find anything (a few unrelated papers in object recognition aside). If anybody can share anything that would be great !

We hope to share more on this soon!

Re: The technology behind GitHub’s new code search

#59

Post by dymk »

maxov wrote: post #50
dymk wrote: post #38

Earlier quoted context omitted.

I think that there should be some sort of standardized, language-agnostic metadata format for semantically indexing a codebase. It could include e.g type information for expressions or declared variables (for languages that infer types), and an index of symbols and how they're connected. This metadata file would be generated by a language-specific tool. For instance, Cargo could generate it for Rust projects, ctags/c…

SemanticDB ( https://scalameta.org/docs/semanticdb/guide.html ) is a protobuf-based file format that does almost exactly this for JVM languages, primarily Scala (I was a contributor a while back). It is used to build an intelligent online code browser, as the backend for a language server, and to do intelligent refactorings. I think a language-agnostic semantic metadata format is a good idea, but requires a lot of co…

I think simple things like "go to reference" or "show type" would be sufficient for 95% of usecases. But if you split languages up into a few different categories (maybe along the lines of Algol-like vs Lisp-like), and were flexible with extensions, I'd imagine we'd see some common patterns emerging, and clients would take advantage of that. Best effort is probably good enough to greatly improve the ergonomics of search.

Re: The technology behind GitHub’s new code search

#60

Post by napsterbr »

debdut wrote: post #30

https://grep.app

I use this one almost daily. It's great to find real world examples of APIs/contracts being used. Also, instant results!

The underlying data may be limited (I have no idea how large it is, I doubt it has indexed every public repository out there), but I never failed to find examples of what I was looking for.

Post reply on HN