Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The technology behind GitHub’s new code search

github.blog

Post reply on HN
Print view View original

131–140 of 187 posts

Re: The technology behind GitHub’s new code search

#131

Post by joshuawarner32 »

orf wrote: post #89
100k wrote: post #86

Earlier quoted context omitted.

Sorry, there are size limits on the number of files in a repositories that we'll index. That is probably why your repository wasn't indexed. Out of curiosity, what are you storing in this repository?

Sequences of code from Python package releases from pypi, as an experiment I’m working on. They compress quite nicely as the deltas between releases are fairly small. I thought it would be nice to be able to search through them, but I guess the file limit was reached. That’s a shame.

Oooh that sounds really useful, would you mind dropping a link?

Re: The technology behind GitHub’s new code search

#132

Post by userbinator »

andrewmcwatters wrote: post #76

> Just use grep? First though, let’s explore the brute force approach to the problem. We get this question a lot: “Why don’t you just use grep?” To answer that, let’s do a little napkin math using ripgrep on that 115 TB of content. On a machine with an eight core Intel CPU, ripgrep can run an exhaustive regular expression query on a 13 GB file cached in memory in 2.769 seconds, or about 0.6 GB/sec/core. But you don't…

or about 0.6 GB/sec/core.

That seems very slow by today's standards. There's a rather... eccentric guy who easily beat that almost 10 years ago with his implementations of string search: https://news.ycombinator.com/item?id=6954298

Re: The technology behind GitHub’s new code search

#134

Post by runlevel1 »

saurik wrote: post #116
damidekronik wrote: post #114

Earlier quoted context omitted.

When trying out some C++ I was surprised by how many projects bundled the library source code, essentially finding any actual client code a pain.

As a C++ developer, I maybe-ironically have this same complaint with JavaScript projects ;P. I particularly hate it when people embed minified versions of libraries, as line-based search and display then will experience a hit for essentially every query on the one giant line that is tens of kilocharacters long.

If your language lacks a de facto way to install dependencies, people will commit vendored code. See: C, C++.

If your language's package manager puts your deps within the repo directory by default, people will commit the vendored code. See: Node, Go (since go.mod).

As long as we can all agree that committing compiled code is a crime punishable by 24 hours in the shame cube.

Re: The technology behind GitHub’s new code search

#136

Post by Zuiii »

xavdid wrote: post #117
DontchaKnowit wrote: post #105

Earlier quoted context omitted.

Thank you. This is SUCH an obvious feature that seems super trivial to implement.

I'm not sure how to read the post and come away thinking any of this is "super trivial", but I'd imagine if it were that easy, they would have done it already.

It's simple enough that most IDEs implement it. Take from that what you will.

Re: The technology behind GitHub’s new code search

#137

Post by exikyut »

100k wrote: post #37
jeffbee wrote: post #35

Earlier quoted context omitted.

Sorry, it cannot handle any of those cases. You're talking about the ability to find the literal `this::foo` but that's not how it would normally appear. It normally will appear anywhere inside a `namespace this` scope, which cs.github does not grok. And cs.github cannot address finding the definition related to a given call site. It doesn't even try.

You are correct, as I mentioned, we do not analyze symbols for C and C++ at this time.

Grimoire (I see that in the HTML and HTTP requests for cs.chromium.org and cs.android.com, so I presume that's what it's called) is really cool, although sadly obviously not OSS. It completely falls apart when faced with JS, which is increasingly being checked in as part of frontend and Mojo glue code, so it's (from the perspective of an outsider trying to get their feet wet) a bit creaky, but being able to click around in C++, which I don't really understand at this point, and learn something new almost every time, is really cool, and IMO representative of at least one concrete beneficial outcome whenever you do get to this.

I wonder if it would be possible to leverage LSP as a kind of tokenization generalization framework, or even piggyback off of the existing effort by incorporating search-friendly/-helpful metadata into future versions of the protocol spec.

Re: The technology behind GitHub’s new code search

#140

Post by rlt »

Zuiii wrote: post #136
xavdid wrote: post #117

Earlier quoted context omitted.

I'm not sure how to read the post and come away thinking any of this is "super trivial", but I'd imagine if it were that easy, they would have done it already.

It's simple enough that most IDEs implement it. Take from that what you will.

The fact that most IDEs implement it doesn’t mean it’s trivial. I believe most IDEs leverage “language servers” written ~once per language and used by many IDEs.

It’s possible GitHub could also leverage those language servers (which would be super cool) but doing it at GitHub scale is certainly not trivial.

Poor man’s version using the new GitHub search would be to construct a regex that matches one but not the other.

Post reply on HN