Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Improvements to searching for special characters in programming languages

blog.google

Post reply on HN
Print view View original

11–20 of 30 posts

Re: Improvements to searching for special characters in programming languages

#11

Post by chimprich »

binarymax wrote: post #10
chimprich wrote: post #7

Earlier quoted context omitted.

Genuine question: why should this be any more difficult than searching for any other type of character? I've long found it hard to understand why Google is so bad at searching for non-alphanumeric characters.

When indexing documents (and querying for them), there is a process the terms go through, to split them up and then normalize them so they can be found easier. A trivial example is you want to find "can't" when someone searches for "cant". Typically special characters are removed for several reasons: the vocabulary of terms becomes smaller and saves space and time, you can ignore punctuation (like searching things th…

OK, thanks - that sounds plausible on the face of it, but why wouldn't you store special characters and then ignore them when matching patterns? You could then make an exception for strings in quotes (or some other option for activating a more precise search).

Maybe Google hasn't previously thought the extra space/complexity was worth the special treatment but given the relative quantity of data they already index and the usefulness of this feature I'm surprised.

Re: Improvements to searching for special characters in programming languages

#12

Post by nostrademons »

chimprich wrote: post #11
binarymax wrote: post #10

Earlier quoted context omitted.

When indexing documents (and querying for them), there is a process the terms go through, to split them up and then normalize them so they can be found easier. A trivial example is you want to find "can't" when someone searches for "cant". Typically special characters are removed for several reasons: the vocabulary of terms becomes smaller and saves space and time, you can ignore punctuation (like searching things th…

OK, thanks - that sounds plausible on the face of it, but why wouldn't you store special characters and then ignore them when matching patterns? You could then make an exception for strings in quotes (or some other option for activating a more precise search). Maybe Google hasn't previously thought the extra space/complexity was worth the special treatment but given the relative quantity of data they already index an…

[ex-Googler, used to work on search, this issue came up repeatedly during my tenure then].

The storage cost was prohibitive. Search engines rely on a data structure known as an inverted index; it's basically a list, for each token, of every document that contains the token, and for a context-aware search engine like Google it usually contains the position within the document of the token as well. Single-character punctuation marks like periods, commas, parentheses, dashes etc. appear in literally every sentence. That means that the inverted index for periods or commas would have to contain an entry for literally every single sentence on the web.

There's a similar problem for common words like 'a', 'the', prepositions, etc, but these are usually already solved by stopwording.

That's why this announcement only covers groups of punctuation with 2-3 characters. These don't appear in ordinary text, and so you can generate posting lists for them that are reasonably-sized. (I suspect that the economics of the index have changed as well, making storage costs cheaper, but this work happened after I left and so I don't know details.)

Re: Improvements to searching for special characters in programming languages

#13

Post by JoshTriplett »

binarymax wrote: post #10
chimprich wrote: post #7

Earlier quoted context omitted.

Genuine question: why should this be any more difficult than searching for any other type of character? I've long found it hard to understand why Google is so bad at searching for non-alphanumeric characters.

When indexing documents (and querying for them), there is a process the terms go through, to split them up and then normalize them so they can be found easier. A trivial example is you want to find "can't" when someone searches for "cant". Typically special characters are removed for several reasons: the vocabulary of terms becomes smaller and saves space and time, you can ignore punctuation (like searching things th…

> A trivial example is you want to find "can't" when someone searches for "cant".

In that case, I think you want to just store "can't" and treat "cant" the way you would any other potential near-miss spelling of a more common word.

Re: Improvements to searching for special characters in programming languages

#14

Post by tomsmeding »

rspeer wrote: post #3

This will be extremely helpful next time I have to use a Haskell library that decides to implement everything as infix operators named "~ " and ".~=" and stuff.

Hoogle is the way to go, man. https://www.haskell.org/hoogle/

Re: Improvements to searching for special characters in programming languages

#17

Post by kristianp »

Animats wrote: post #5

I can now find the C+@ programming language. So it's not heavily special cased for common programming languages. Google Code Search (2006-2013) [1] was more useful. I miss that. Its search allowed regular expressions. [1] https://en.wikipedia.org/wiki/Google_Code_Search

It doesn't seem to work perfectly. Doing a verbatim search for "C+@" programming language produces a lot of results without the "C+@" on the page.

Re: Improvements to searching for special characters in programming languages

#18

Post by rspeer »

tomsmeding wrote: post #14
rspeer wrote: post #3

This will be extremely helpful next time I have to use a Haskell library that decides to implement everything as infix operators named "~ " and ".~=" and stuff.

Hoogle is the way to go, man. https://www.haskell.org/hoogle/

I know Hoogle exists, but that just searches one kind of documentation. Despite the cute name, it's not Google. You can't Hoogle an error message and see if anyone else got it.

Re: Improvements to searching for special characters in programming languages

#19

Post by johnfn »

TACIXAT wrote: post #8

This is great. I feel Google has slowly become too user friendly. My mobile results are always way less technical than my desktop results. If I'm in the car (passenger) and want to look up a problem I'm having while programming, I get mostly related queries that are a simplified version of what I'm looking for. I really believe that the technical crowd drives what becomes popular (app recommendations for family and f…

Google tailors its results to the kind of person it thinks you are. For example, if you immediately search "python", you will get results about snakes. But if you search for programming first, and then python second, it will now give back programming results on the second search. This continues to apply if you searched "programming" last week.

This behavior is actually very nuanced and impressive to watch, once you understand what's going on.

I don't think google is becoming more user friendly at the expense of being technical. It certainly isn't for me. What your problem sounds like is that it's built two separate profiles for you - one of which is what you're likely to search of desktop, and the other for what you're likely to search on mobile.

Post reply on HN