Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The technology behind GitHub’s new code search

github.blog

Post reply on HN
Print view View original

101–110 of 187 posts

Re: The technology behind GitHub’s new code search

#101

Post by zellyn »

emidoots wrote: post #72
mxstbr wrote: post #13

Earlier quoted context omitted.

I don't think Sourcegraph is in big trouble here. Their whole play is enterprises, who likely have code spread across many different hosts. On top of that, their code search is still miles ahead of GitHub's.

I've worked alongside the CEO/CTO of Sourcegraph for the past 8 years, everyone else is at our company offsite so I figured I'd chime in :) nobody asked me to write this (nor did I ask) :) The article is a top-notch technical write-up, the devs on GitHub code search should be proud of what they've achieved so far! Honestly, we're rooting for GitHub to improve their code search, viewing them as a close peer-not a comp…

Thanks for the information! I came to the hackernews discussion because SourceGraph was so curiously absent from the article and the "brief history of codesearch" article it linked to, and I wanted to see what others thought!

Re: The technology behind GitHub’s new code search

#102

Post by sqs »

imemyself wrote: post #94
mxstbr wrote: post #13

Earlier quoted context omitted.

I don't think Sourcegraph is in big trouble here. Their whole play is enterprises, who likely have code spread across many different hosts. On top of that, their code search is still miles ahead of GitHub's.

Sourcegraph has become pretty obscenely expensive in the last couple years (with borderline hostile sales folks to boot). I know at least one company who would love to be able to cancel that contract if GH search is "good enough" now.

Sourcegraph CEO here. We definitely need a cheaper tier for smaller companies or those who don’t need our entire feature set. I agree! What do you think that should be?

Overall, we’re building what our customers need, and our product goes way beyond what GitHub can offer. Sourcegraph indexes all the code and increasingly all the code intelligence (including code nav but also code ownership and other metadata in the future from your other dev tools). We charge based on active usage, so we make money when devs at customers /choose/ to use us over the alternatives. We’re trying to do this the right way, and tons of customers agree. (If anyone reading this disagrees, please let me know!)

Re: your comment about our sales team, I’m really sorry to hear that and want to understand more so I can fix the problem. Can you please email me at sqs@sourcegraph.com?

Re: The technology behind GitHub’s new code search

#103

Post by rtuin »

tiagod wrote: post #92
andrewmcwatters wrote: post #76

> Just use grep? First though, let’s explore the brute force approach to the problem. We get this question a lot: “Why don’t you just use grep?” To answer that, let’s do a little napkin math using ripgrep on that 115 TB of content. On a machine with an eight core Intel CPU, ripgrep can run an exhaustive regular expression query on a 13 GB file cached in memory in 2.769 seconds, or about 0.6 GB/sec/core. But you don't…

I use github-wide searches all the time to see how people are using certain APIs, to find libraries used in some blob from the strings I find there, to find people working with the same data I'm about to attempt to work it, and the list goes on. What you use github search for doesn't require all this engineering, but what I use it for does. Why wouldn't they build something that satisfies both our necessities well?

Same here. I find doing a quick org-wide code search for a specific call a great starting point for impact analysis for API changes. Using it multiple times a week!

Re: The technology behind GitHub’s new code search

#104

Post by kirillbobyrev »

This is exciting! I see a lot of familiar pieces here that propagated from Google's Code Search and I know few people from Code Search went to GitHub, probably specifically to work on this. I always wondered why GitHub didn't invest into a decent code searching features, but I'm happy it finally gets to the State of the Art one step at a time. Some of the folks going to GitHub to work on this I know are just incredible and I have no doubt GitHub's code search will be amazing.

I also worked on something similar to the search engine that is described here for the purposes of making auto-complete fast for C++ in Clangd. That was my intern project back in 2018 and it was very successful in reducing the delays and latencies in the auto-complete pipeline. That project was a lot of fun and was also based on Russ Cox's original Google Code Search trigram index. My implementation of the index is still largely untouched and is a hot path of Clangd. I made a huge effort to document it as much as I can and the code is, I believe, very readable (although I'm obviously very biased because I spent a loot of time with it).

Here is the implementation:

https://github.com/llvm/llvm-project/tree/main/clang-tools-e...

I also wrote a... very long design document about how exactly this works, so if you're interested in understanding the internals of a code search engine, you can check it out:

https://docs.google.com/document/d/1C-A6PGT6TynyaX4PXyExNMiG...

Re: The technology behind GitHub’s new code search

#105

Post by DontchaKnowit »

PaulHoule wrote: post #68

My beef with GitHub's code search is that it doesn't distinguish between the definition of a symbol and the uses of the symbol, so you need to wade through 5 pages of results to get the one result you're looking for. I would contrast that to my IDE which usually scores a direct hit if I enter a search in the right box. The indexing they talk about in that article seems like rearranging the deck chairs on the Titanic…

Thank you. This is SUCH an obvious feature that seems super trivial to implement.

Re: The technology behind GitHub’s new code search

#106

Post by MapleWalnut »

drcongo wrote: post #65

With current search, I can search [0] the Django repo for a class that definitely exists [1] in Django, there are 0 code results. Zero. GitHub search is mystifyingly bad, I hope this is a LOT better. [0] https://github.com/django/django/search?q=DeleteView&type=co... [1] https://github.com/django/django/blob/main/django/views/gene...

With the new GitHub code search, the first result is the DeleteView class: https://github.com/django/django/blob/bd366ca2aeffa869b7dbc0...

Re: The technology behind GitHub’s new code search

#108

Post by sqs »

the_mitsuhiko wrote: post #22
mxstbr wrote: post #13

Earlier quoted context omitted.

I don't think Sourcegraph is in big trouble here. Their whole play is enterprises, who likely have code spread across many different hosts. On top of that, their code search is still miles ahead of GitHub's.

The new code search includes private repos. Miles ahead doesn’t matter if you get the search at GitHub for free. GitHub actions started out pretty terrible but they are now dominating hosted CI. SourceGraph likely has challenging times ahead considering the valuation.

Sourcegraph CEO here. “Challenging times” is how it should be all the time. That means competition is forcing both GitHub and us to build better stuff for devs. Devs win.

But to be clear, as a company we are doing well and growing nicely inside customers, with a ton of cash in the bank, an awesome team, and a huge opportunity ahead of us. GitHub’s new code search has been out for 14 months now, so this is nothing new.

It’s a big market and there’s way more room for differentiation and dev choice in code search/intelligence than in CI. There’s a lot of code intelligence that GitHub won’t support (precise code nav for more languages, comprehensive code ownership, metadata from other dev tools that know things about code outside the GitHub/Microsoft suite, etc.), there’s a need for the ability to fix (with our Batch Changes) not just find, and even in the core search workflow there’s so much room for improvement with AI fine-tuned on your own code, etc.

But talk is cheap and only shipping matters. So, watch what we ship, and send any feedback and requests our way!

Re: The technology behind GitHub’s new code search

#109

Post by anitil »

tonymet wrote: post #21

This is a great intro / overview of full-text search for those wondering how to build your own search engine. It's a great 101-level exercise to write an inverted index implementation you can do it in an afternoon , and then expand to a leaf /aggregator in follow-up exercises.

I did this for our organization using sqlite's FTS module and datasette and boy was it fast. Unfortunately I did get (temporarily) banned from the organisations github account, but it was definitely worth it.

Even now I find myself using it despite the index being a few months out of date.

Re: The technology behind GitHub’s new code search

#110

Post by chatmasta »

sqs wrote: post #88
chatmasta wrote: post #84

Earlier quoted context omitted.

Perhaps it could benefit from something like a "dissimilarity" filter, which ranks the current result set by returning the most unique hits first. You wouldn't always want this, because sometimes you're searching for how something is typically used, and with many duplicative results you can confirm that's the preferred pattern. But other times you're looking for more estoteric usage of a certain function, and it woul…

Re: Sourcegraph, we're working on improving that, and sorry you couldn't get the results you wanted. We primarily build for the code within customers, where this particular problem is less common than across all open-source repositories. But we want it to work really well in every case. Our new ranking ( https://about.sourcegraph.com/blog/new-search-ranking ) should help a lot here, and it's live on https://sourcegra…

Sure! I will try it next time I'm using code search. I can't remember the specific queries I've tried, but I only ever used it when looking for "how have people used this function," so I remember the limited depth of results being particularly annoying.

Thanks for the link to your blog post. I didn't realize Steve Yegge had joined your team - congratulations, that's quite an endorsement!

I've always liked Sourcegraph just because our company names are so similar (founder of Splitgraph here)... we might have even gotten a few candidates because of that initial confusion. :)

How has GitHub Code Search impacted your product direction? Do you see it as an opportunity to focus more on the internal use case, or do you have plans for some other differentiation? It's always unfortunate when a big company introduces a product so similar to the core product of a startup, but I'm sure there is a silver lining there, especially when you have a talented team and a mature codebase (for example, Fly.io has been able to carve out a niche for itself despite Cloudflare moving to compete in the same areas). Either way, best of luck to you from a fellow S-grapher!

Post reply on HN