Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How Google Code Search Worked

swtch.com

Post reply on HN
Print view View original

11–20 of 47 posts

Re: How Google Code Search Worked

#11

Post by dennisgorelik »

ajasmin wrote: post #9
dennisgorelik wrote: post #6

It's a nice design (in particular Trigram Index), but overall product still failed. My guess is that regular expression search is not as useful as full-text search that general Google Search does.

Finding code with the regular Google Search is nearly impossible though.

I'm finding code with regular Google Search all the time.

Yes, General Google Search is missing some neat features, but overall these features are not as important as convenience of using familiar general search queries, search speed, and the size of general google search index.

BTW, do you have your own explanation of why Google Code Search was cancelled?

Re: How Google Code Search Worked

#12

Post by fasdg »

arnsholt wrote: post #8

To be fair, the reason Perl, Python and PCRE (which all use pretty much the same regex syntax) don't use the linear-time algorithms is because they can't. Features like look-around and matching what you've already matched (/(\w+) \1/ to find repeated words for example) give you more expressivity than regular languages, but also takes away linear time algorithms.

As Russ points out in his earlier re2-related blog posts, these regex engines still perform non-linearly on inputs which don't involve look-around, look-behind, etc. There's plenty of room for improvement even if they want to keep these features.

Re: How Google Code Search Worked

#13

Post by inerte »

gruseom wrote: post #10

Google made a mistake in killing code search. Indexing the world's source code and making it searchable is so obviously part of their core mission that I wonder how this decision even got made. Yeah, code search is a niche market numerically speaking, but intellectually and economically (considering the economic impact of software) it is vital. Google was doing so much better a job of it than anybody else that they c…

Did it stop working internally? Do Google employees still get to use it?

Re: How Google Code Search Worked

#14

Post by snprbob86 »

fasdg wrote: post #12
arnsholt wrote: post #8

To be fair, the reason Perl, Python and PCRE (which all use pretty much the same regex syntax) don't use the linear-time algorithms is because they can't. Features like look-around and matching what you've already matched (/(\w+) \1/ to find repeated words for example) give you more expressivity than regular languages, but also takes away linear time algorithms.

As Russ points out in his earlier re2-related blog posts, these regex engines still perform non-linearly on inputs which don't involve look-around, look-behind, etc. There's plenty of room for improvement even if they want to keep these features.

Seems like the default should be linear runtime and you should have to explicitly ask for the richer feature set (and opt into the assertion that you trust the input not to DOS your process).

Could easily be added as a modifier (see `man perlre`), but should be implemented as two to enable explicit behavior and toggling the default. Randomly picking the letter N:

    /(\w+) \1/n    # Error: look-behind is incompatible with linear runtime RE engine
    /(\w+) \1/N    # Works!
    /(\w+) \1/     # Preferable works for backwards compat, maybe overridden by an ENV var

Re: How Google Code Search Worked

#15

Post by toddc »

Russ's articles are an excellent write-up and explanation.

However, many finite-state automata regex implementations have existed for years (e.g. Java http://cs.au.dk/~amoeller/automaton) without the backtracking feature, of course. Also of interest is the benchmark data at: http://tusker.org/regex/regex_benchmark.html

Re: How Google Code Search Worked

#16

Post by pshc »

gruseom wrote: post #10

Google made a mistake in killing code search. Indexing the world's source code and making it searchable is so obviously part of their core mission that I wonder how this decision even got made. Yeah, code search is a niche market numerically speaking, but intellectually and economically (considering the economic impact of software) it is vital. Google was doing so much better a job of it than anybody else that they c…

From Steve Yegge's most recent blog rant:

Now, as it happens, I am in fact working on a very cool project at Google. [...] a project that aims to turn source code -- ALL source code -- from plain text into Wikipedia.

Re: How Google Code Search Worked

#17

Post by gruseom »

pshc wrote: post #16
gruseom wrote: post #10

Google made a mistake in killing code search. Indexing the world's source code and making it searchable is so obviously part of their core mission that I wonder how this decision even got made. Yeah, code search is a niche market numerically speaking, but intellectually and economically (considering the economic impact of software) it is vital. Google was doing so much better a job of it than anybody else that they c…

From Steve Yegge's most recent blog rant: Now, as it happens, I am in fact working on a very cool project at Google. [...] a project that aims to turn source code -- ALL source code -- from plain text into Wikipedia.

Ah. Well, the decision to kill Code Search would make sense if it were in favour of something better. But then why kill it now and leave nothing for any length of time? Also, there's no guarantee the new thing will turn out to actually be better.

For a company that succeeded partly by leveraging the economic value of hackers in a way that hadn't been been done before, this decision is disturbingly out of character. It feels like something that must have happened for inward-facing political reasons - in other words, a sign of rot.

I get that Steve told Larry to focus, but "code" and "search" almost define focus in their case.

Edit: It seems the service is still available under a different URL. Weird, but I'll happily take it! http://news.ycombinator.com/item?id=3487950

Re: How Google Code Search Worked

#18

Post by rachelbythebay »

gruseom wrote: post #17
pshc wrote: post #16

Earlier quoted context omitted.

From Steve Yegge's most recent blog rant: Now, as it happens, I am in fact working on a very cool project at Google. [...] a project that aims to turn source code -- ALL source code -- from plain text into Wikipedia.

Ah. Well, the decision to kill Code Search would make sense if it were in favour of something better. But then why kill it now and leave nothing for any length of time? Also, there's no guarantee the new thing will turn out to actually be better. For a company that succeeded partly by leveraging the economic value of hackers in a way that hadn't been been done before, this decision is disturbingly out of character. I…

Historically, there were two types of projects at Google: the one that's deprecated and the one that doesn't work yet. It seems they have amended that slightly so now it's break-before-make during a migration instead of the other way around.

Re: How Google Code Search Worked

#19

Post by 178 »

inerte wrote: post #13
gruseom wrote: post #10

Google made a mistake in killing code search. Indexing the world's source code and making it searchable is so obviously part of their core mission that I wonder how this decision even got made. Yeah, code search is a niche market numerically speaking, but intellectually and economically (considering the economic impact of software) it is vital. Google was doing so much better a job of it than anybody else that they c…

Did it stop working internally? Do Google employees still get to use it?

http://googlesystem.blogspot.com/2012/01/google-code-search-... —» http://code.google.com/codesearch

Re: How Google Code Search Worked

#20

Post by gruseom »

178 wrote: post #19
inerte wrote: post #13

Earlier quoted context omitted.

Did it stop working internally? Do Google employees still get to use it?

http://googlesystem.blogspot.com/2012/01/google-code-search-... —» http://code.google.com/codesearch

Wow. If this really is the same service that was available before - which it looks like it is - then thanks!

p.s. You should submit this as a story in its own right.

Post reply on HN