Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The technology behind GitHub’s new code search

github.blog

Post reply on HN
Print view View original

181–187 of 187 posts

Re: The technology behind GitHub’s new code search

#181

Post by lern_too_spel »

sqs wrote: post #111
lern_too_spel wrote: post #42

Earlier quoted context omitted.

Sourcegraph had to have known GitHub would do this if they didn't accept the offer. Since this should be expected, the launch of this feature shouldn't change what their decision should have been.

Sourcegraph CEO here. Just to be clear, so internet rumors don’t get started, there was no “offer” here. We started Sourcegraph with the intent of remaining independent because building really good code search and intelligence means working across all code (not just on GitHub), all devs, and all code intelligence sources (code nav plus every dev tool you use that knows stuff about code, not just the ones in the GitHu…

GitHub's code search product looks limited compared to what Sourcegraph provides, but I don't think being limited to code on their instance is a problem. Even if a company doesn't want to use the full GitHub source code management stack, if GH search becomes good enough, people could mirror their repos onto GitHub or GitHub Enterprise Server just to use its search functionality, and Microsoft will go after that segment if they see a market there.

Re: The technology behind GitHub’s new code search

#182

Post by WorldMaker »

andrewmcwatters wrote: post #76

> Just use grep? First though, let’s explore the brute force approach to the problem. We get this question a lot: “Why don’t you just use grep?” To answer that, let’s do a little napkin math using ripgrep on that 115 TB of content. On a machine with an eight core Intel CPU, ripgrep can run an exhaustive regular expression query on a 13 GB file cached in memory in 2.769 seconds, or about 0.6 GB/sec/core. But you don't…

Have you tried using github.dev for your remote grep? Press . in a code repo (while logged in) and it loads a remote VS Code for you in that repo. I don't entirely understand the complex interplay between how github.dev uses both the GitHub search indexes and in-browser in-memory grep, but being as it tries its best reproduce the local VS Code ripgrep experience, it may still be more optimized for your use cases.

Re: The technology behind GitHub’s new code search

#183

Post by bobbylarrybobby »

rlt wrote: post #140
Zuiii wrote: post #136

Earlier quoted context omitted.

It's simple enough that most IDEs implement it. Take from that what you will.

The fact that most IDEs implement it doesn’t mean it’s trivial. I believe most IDEs leverage “language servers” written ~once per language and used by many IDEs. It’s possible GitHub could also leverage those language servers (which would be super cool) but doing it at GitHub scale is certainly not trivial. Poor man’s version using the new GitHub search would be to construct a regex that matches one but not the other…

I think running a language server over a repo would be a reasonable addition to the current index. Should also be pretty small compared to the current index because the number of n grams far exceeds the number of symbols/items.

Re: The technology behind GitHub’s new code search

#185

Post by masklinn »

therealdrag0 wrote: post #157
boyter wrote: post #47

The sparse grams solution to deal with stupidly common ngrams such as for or tes is very interesting. I’d love to see more discussion on how they are dealing with the false positives though. It looks like a positional index is being used to achieve this, but that usually blows out your index size. Additional information about deduplication would be especially interesting to me as well. It seems to solve this quite we…

> What I find really interesting about all the code search engines I know of is that each one implemented its own index I mean GH got a long way using ElasticSearch until now.

> I mean GH got a long way using ElasticSearch until now.

I'm not sure that's true. It used ES for a long time, but the search was also terrible, at the edge of complete uselessness.

Re: The technology behind GitHub’s new code search

#186

Post by saagarjha »

colin353 wrote: post #70
saagarjha wrote: post #64

I’ve been using the new code search for a couple of months and I like it, but the UI is kind of antagonistic to how I typically want to search for things. For one, the new experience doesn’t actually load code onto the page, it does some sort of lazy loading thing as you scroll around, so ⌘F doesn’t work. I understand that there’s a custom search box to try to get around this but it’s pretty slow and fiddly and I don…

Hey saagarjha, thanks for the feedback. It's our goal to make the experience as good as possible, and we're aware of shortcomings with cmd+F and `t`, among other things. We're working on it, and your feedback helps us a lot. We read all the feedback on the forum here: https://github.com/orgs/community/discussions/38692 , so please keep providing it. Videos and screenshots are super helpful too. Thanks for bearing wit…

Ah, I see. I'll file future feedback there, then. Thanks!

Re: The technology behind GitHub’s new code search

#187

Post by yazaddaruvala »

loginatnine wrote: post #5

I'm curious if they'll open source Blackbird, it does not seem mentioned in the post.

From the description, it is just ElasticSearch re-built on top of a Kafka topic. Its not clear what benefit it would brings to open source it.
Post reply on HN