Live data from Hacker News

Stop Me If You've Seen This Word Before

codinghorror.com

11–19 of 19 posts

Re: Stop Me If You've Seen This Word Before

#11
Way back in 2004, I ran a little experiment with Google -- over a period of a week, I searched for an entire dictionary of ~110k individual English words and recorded how many hits Google returned for each.

Of course, a word can appear on a page multiple times. That's why, I think, folks used to ignore the stopwords. They introduced noise when trying to access the content words. Now, with span constraints, you can incorporate them into the analysis. So "a matrix" and "the matrix" returns very different results, even without quotes.

Re: Stop Me If You've Seen This Word Before

#12
It makes sense that low-information terms would have a lower preference when searching without any context. If your index models the context around terms, you can get better results from a low-information search.

I think...I'm kind of shooting from the hip here relating it to context modeling in lossless compression schemes like CABAC and PPM.

Could you overcome stop words with some sort of Bayesian phrase matching over some learned hidden states?

Re: Stop Me If You've Seen This Word Before

#13
it is about how to sort with stop word. Tranditional tf-idf method didn't work well as it didn't contain any information about each word relative location in its context. a simple method is to index "the the", the word group instead of single "the". I guess it is what Google does now with "to be or not to be". However, the word grouping tech is a common method in CJK full text search.

Re: Stop Me If You've Seen This Word Before

#14
post #8

I ran into a similar problem teaching myself perl a couple years ago by doing a short tutorial then foolishly jumping into a co-worker's code. "What the hell is '$.'? Hmm, well I'm sure Google can help me. What? No matching documents?!? What is this crazy s.o.b. doing here??"

http://www.google.com/codesearch?hl=en&lr=&q=lang%3A...

Re: Stop Me If You've Seen This Word Before

#15
post #8

I ran into a similar problem teaching myself perl a couple years ago by doing a short tutorial then foolishly jumping into a co-worker's code. "What the hell is '$.'? Hmm, well I'm sure Google can help me. What? No matching documents?!? What is this crazy s.o.b. doing here??"

Glad to hear I'm not the only one who's had trouble with this. I remember reading some C code once and I saw some strange operators I had never seen before... ?, if I remember correctly. I eventually figured out they were GCC extensions for "min" and "max", but only after a painful experience trying to get more info out of Google than "here's how you do addition and multiplication in C!"

http://www.google.com/codesearch?hl=en&lr=&q=lang%3A...

Re: Stop Me If You've Seen This Word Before

#16
post #8

I ran into a similar problem teaching myself perl a couple years ago by doing a short tutorial then foolishly jumping into a co-worker's code. "What the hell is '$.'? Hmm, well I'm sure Google can help me. What? No matching documents?!? What is this crazy s.o.b. doing here??"

Glad to hear I'm not the only one who's had trouble with this. I remember reading some C code once and I saw some strange operators I had never seen before... ?, if I remember correctly. I eventually figured out they were GCC extensions for "min" and "max", but only after a painful experience trying to get more info out of Google than "here's how you do addition and multiplication in C!"

[deleted]

Re: Stop Me If You've Seen This Word Before

#19
post #8

I ran into a similar problem teaching myself perl a couple years ago by doing a short tutorial then foolishly jumping into a co-worker's code. "What the hell is '$.'? Hmm, well I'm sure Google can help me. What? No matching documents?!? What is this crazy s.o.b. doing here??"

$. -> "perldoc perlvar" -> $INPUT_LINE_NUMBER -> http://www.google.com/search?q=%24INPUT_LINE_NUMBER+perl
Post reply on HN