Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Towards Natural Language Semantic Code Search at GitHub

githubengineering.com

Post reply on HN
Print view View original

31–40 of 55 posts

Re: Towards Natural Language Semantic Code Search at GitHub

#31

Post by Deimorz »

psychometry wrote: post #22
dgreensp wrote: post #14

It is a real cultural problem how engineers get more excited about machine learning than basic usability. GitHub search can't even search for a literal string, let alone a regex. It can't search a subdirectory. Ranking is indistinguishable from random. It's been this way for years. How about building an actual, usable, basic code search and then getting all fancy with your machine learning? I almost built my own "onl…

You also can't search a forked repository, which is pathetic.

This limitation is especially frustrating when a fork becomes the "primary" repo for a project for some reason. It's probably not a common occurrence overall, but I've run into it at least a couple of times.

A good example is that GitHub's own repo for their CommonMark implementation isn't searchable, because it's a fork of cmark: https://github.com/github/cmark/

Re: Towards Natural Language Semantic Code Search at GitHub

#32

Post by neongreen »

dgreensp wrote: post #14

It is a real cultural problem how engineers get more excited about machine learning than basic usability. GitHub search can't even search for a literal string, let alone a regex. It can't search a subdirectory. Ranking is indistinguishable from random. It's been this way for years. How about building an actual, usable, basic code search and then getting all fancy with your machine learning? I almost built my own "onl…

I'm building one! https://codesearch.aelve.com

Currently it runs on a fairly slow machine, so regex-heavy requests will take some time on big package repositories like Rubygems, but I plan to get a nicer machine soon.

If you know Scala, you can even contribute (wink wink), just ping me. A lot of tasks we have at this stage are pretty basic.

Re: Towards Natural Language Semantic Code Search at GitHub

#33

Post by karmakaze »

dgreensp wrote: post #14

It is a real cultural problem how engineers get more excited about machine learning than basic usability. GitHub search can't even search for a literal string, let alone a regex. It can't search a subdirectory. Ranking is indistinguishable from random. It's been this way for years. How about building an actual, usable, basic code search and then getting all fancy with your machine learning? I almost built my own "onl…

In full agreement. Every now and then I'll expect a search to work. My solution has been to run etsy/hound [0] for my active reps.

  [0] https://github.com/etsy/hound

Re: Towards Natural Language Semantic Code Search at GitHub

#34

Post by nerdponx »

boyter wrote: post #30
amelius wrote: post #28

Earlier quoted context omitted.

But people use grep on their code all the time ...

People use grep on their local code repository, which is generally less than 2 gigabytes of source. A tool like ripgrep can process that in under a second on any modern machine with a warm disk cache. Its when you get to hundreds of repositories or 10's of gigabytes of code that local tools cannot run fast enough. They are not designed for this use-case, and usually rely on the files being searched hitting the disk c…

Nobody is asking for cross-repo search. Literally just let us run "git grep" on Github.

Re: Towards Natural Language Semantic Code Search at GitHub

#36

Post by derefr »

nerdponx wrote: post #34
boyter wrote: post #30

Earlier quoted context omitted.

People use grep on their local code repository, which is generally less than 2 gigabytes of source. A tool like ripgrep can process that in under a second on any modern machine with a warm disk cache. Its when you get to hundreds of repositories or 10's of gigabytes of code that local tools cannot run fast enough. They are not designed for this use-case, and usually rely on the files being searched hitting the disk c…

Nobody is asking for cross-repo search. Literally just let us run "git grep" on Github.

On your local machine, you can get away with a linear scan (as "git grep" does), because nothing else is contending for time with your search. "git grep" is actually very costly in terms of CPU and disk IO, but you don't notice, because you're only running one "git grep" at once, and so nothing is contending for those resources.

On GitHub, tens of thousands of searches will be happening at once on the same search cluster. If they were all literally doing a "git grep" (a linear scan of the associated repo data), the disk caches would thrash back and forth between queries, and nothing could be answered in less than 30 seconds.

The only way for GitHub to respond to code searches at scale in a reasonable time, is to have a pre-built index.

If the index was per repo, that'd be a kind of partitioned index; and there's no DBMS that I know of that can handle a partitioned object having 58 million partitions. It makes much more sense to have an unpartitioned index... which effectively implies "cross-repo search" (because, once you have an unpartitioned index built up over all your repos, it costs nothing to enable someone to search that entire index at once.)

Re: Towards Natural Language Semantic Code Search at GitHub

#37

Post by bradleyjg »

dgreensp wrote: post #14

It is a real cultural problem how engineers get more excited about machine learning than basic usability. GitHub search can't even search for a literal string, let alone a regex. It can't search a subdirectory. Ranking is indistinguishable from random. It's been this way for years. How about building an actual, usable, basic code search and then getting all fancy with your machine learning? I almost built my own "onl…

Exactly this. I don’t understand why the-thing-I-searched-for.java is so rarely on the first page of results. Doesn’t that seem like an obvious thing I might be interested in?!?

Re: Towards Natural Language Semantic Code Search at GitHub

#38

Post by [deleted] »

boyter wrote: post #30
amelius wrote: post #28

Earlier quoted context omitted.

But people use grep on their code all the time ...

People use grep on their local code repository, which is generally less than 2 gigabytes of source. A tool like ripgrep can process that in under a second on any modern machine with a warm disk cache. Its when you get to hundreds of repositories or 10's of gigabytes of code that local tools cannot run fast enough. They are not designed for this use-case, and usually rely on the files being searched hitting the disk c…

[deleted]

Re: Towards Natural Language Semantic Code Search at GitHub

#39

Post by welder »

kornish wrote: post #16
dgreensp wrote: post #14

It is a real cultural problem how engineers get more excited about machine learning than basic usability. GitHub search can't even search for a literal string, let alone a regex. It can't search a subdirectory. Ranking is indistinguishable from random. It's been this way for years. How about building an actual, usable, basic code search and then getting all fancy with your machine learning? I almost built my own "onl…

Agreed. Luckily, we as a community have tools like Sourcegraph which are based on battle-tested pragmatic systems from places like Google. Disclaimer: no affiliation, just love the team and product.

At first I thought this would replace Sourcegraph, but looks like it's just an experiment with NLP... Thank goodness we have Sourcegraph for searching GH but especially for searching GHEnterprise in an SOA environment where it's impossible to have every repo cloned locally for ripgrep.

P.S. I'm not affiliated, we just use Sourcegraph at a company I work for.

Post reply on HN