Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Towards Natural Language Semantic Code Search at GitHub

githubengineering.com

Post reply on HN
Print view View original

51–55 of 55 posts

Re: Towards Natural Language Semantic Code Search at GitHub

#51

Post by bitL »

dgreensp wrote: post #14

It is a real cultural problem how engineers get more excited about machine learning than basic usability. GitHub search can't even search for a literal string, let alone a regex. It can't search a subdirectory. Ranking is indistinguishable from random. It's been this way for years. How about building an actual, usable, basic code search and then getting all fancy with your machine learning? I almost built my own "onl…

So what? Here their only mistake is not to license/buy some search engine instead of wasting years on developing another "meh" one. What they are doing here with semantic search is the future and their chance to make all existing code search engines obsolete. Use your favorite Internet search engine to find GitHub's snippets of code instead. Those won't give you semantic code search though.

Re: Towards Natural Language Semantic Code Search at GitHub

#52

Post by nh2 »

neongreen wrote: post #32
dgreensp wrote: post #14

It is a real cultural problem how engineers get more excited about machine learning than basic usability. GitHub search can't even search for a literal string, let alone a regex. It can't search a subdirectory. Ranking is indistinguishable from random. It's been this way for years. How about building an actual, usable, basic code search and then getting all fancy with your machine learning? I almost built my own "onl…

I'm building one! https://codesearch.aelve.com Currently it runs on a fairly slow machine, so regex-heavy requests will take some time on big package repositories like Rubygems, but I plan to get a nicer machine soon. If you know Scala, you can even contribute (wink wink), just ping me. A lot of tasks we have at this stage are pretty basic.

Nice! Next index all public Github repos?

Re: Towards Natural Language Semantic Code Search at GitHub

#54

Post by daze42 »

amelius wrote: post #49
derefr wrote: post #36

Earlier quoted context omitted.

On your local machine, you can get away with a linear scan (as "git grep" does), because nothing else is contending for time with your search. "git grep" is actually very costly in terms of CPU and disk IO, but you don't notice, because you're only running one "git grep" at once, and so nothing is contending for those resources. On GitHub, tens of thousands of searches will be happening at once on the same search clu…

When I press "download" on GitHub, it also does a linear scan ...

Of the files, maybe, but not the characters in those files. It's orders of magnitude more difficult.

Re: Towards Natural Language Semantic Code Search at GitHub

#55

Post by dgreensp »

derefr wrote: post #36
nerdponx wrote: post #34

Earlier quoted context omitted.

Nobody is asking for cross-repo search. Literally just let us run "git grep" on Github.

On your local machine, you can get away with a linear scan (as "git grep" does), because nothing else is contending for time with your search. "git grep" is actually very costly in terms of CPU and disk IO, but you don't notice, because you're only running one "git grep" at once, and so nothing is contending for those resources. On GitHub, tens of thousands of searches will be happening at once on the same search clu…

While that's basically true, git grep doesn't scale...

With ever-cheaper RAM, ever-faster SSDs, and cloud computing, it could actually make sense, now or soon, to scan through an entire repo, either on "disk" or in RAM. I started building an app that would suck down any GitHub repo by simulating a git clone in the back-end, in the time it took you to type in your query, and hold it in RAM across queries. As a user, I'm sure I would gladly pay however many cents it costs to take up this much RAM when I'm on the search page over the course of a month.

Even if a linear scan isn't feasible, regex search at scale is not an unsolved problem, and GitHub has access to world-class engineers. Google Code used trigrams to do regex search with an index (https://swtch.com/~rsc/regexp/regexp4.html). Sourcegraph offers regex search.

Post reply on HN