Information Retrieval papers you need to read • Hacker News Board

Re: Information Retrieval papers you need to read

#11

Post by mahmud » Wed, Apr 08, 2009, 7:28 AM UTC

GeneralMaximus wrote: post #10

mahmud wrote: post #5

Earlier quoted context omitted.

I'm doing the same right now. Massive scraping plus "gisting" or document summarization. You're pretty much on the right track; half of those papers are industry standards (my browser marked them as "visited" automatically :-)

So which of those would be a good starting point for someone who has no idea about IR?

start with LSA: laten semantic indexing, if you want information gisting/summarization.

i'm not worried about IR because I use a good search-engine library: Montezuma, and it's in Common Lisp. It's Java clone is called Lucene :-)

Re: Information Retrieval papers you need to read

#12

Post by gfodor » Wed, Apr 08, 2009, 12:32 PM UTC

wheels wrote: post #6

A couple other seminal favorites: Authoritative Sources in a Hyperlinked Environment http://www.cs.cornell.edu/home/kleinber/auth.pdf Probabilistic Latent Semantic Analysis http://www.cs.brown.edu/~th/papers/Hofmann-UAI99.pdf Kleinberg, author of the first paper, is quite possibly my favorite computer scientist. He's been on the leading edge of a lot of largely disjoint areas of research on the information structure…

The "rebel king" is also a phenominal lecturer and a favorite among all cornell CS students.

Re: Information Retrieval papers you need to read

#13

Post by GeneralMaximus » Wed, Apr 08, 2009, 3:19 PM UTC

mahmud wrote: post #11

GeneralMaximus wrote: post #10

Earlier quoted context omitted.

So which of those would be a good starting point for someone who has no idea about IR?

start with LSA: laten semantic indexing, if you want information gisting/summarization. i'm not worried about IR because I use a good search-engine library: Montezuma, and it's in Common Lisp. It's Java clone is called Lucene :-)

A Google search gave me this: http://www.cs.utk.edu/~lsi/

Is that what you're talking about?

Re: Information Retrieval papers you need to read

#14

Post by [deleted] » Wed, Apr 08, 2009, 3:28 PM UTC

[deleted]

Re: Information Retrieval papers you need to read

#15

Post by wheels » Wed, Apr 08, 2009, 3:36 PM UTC

GeneralMaximus wrote: post #10

mahmud wrote: post #5

Earlier quoted context omitted.

I'm doing the same right now. Massive scraping plus "gisting" or document summarization. You're pretty much on the right track; half of those papers are industry standards (my browser marked them as "visited" automatically :-)

So which of those would be a good starting point for someone who has no idea about IR?

Honestly, I don't think LSI is a good place to start. It'd be better to start a few steps back with basic IR algorithms and data structures like tries, inverted indexes, stemming, string search algorithms like KMP and Boyer-Moore.

When I got started with IR I used Baeza-Yate's "Modern Information Retrieval"; Manning's new "Introduction to Information Retrieval" looks great, though I haven't gotten my hands on a copy yet. Here are the Wikipedia pages on the concepts above:

http://en.wikipedia.org/wiki/Trie

http://en.wikipedia.org/wiki/Inverted_index

http://en.wikipedia.org/wiki/Stemming

http://en.wikipedia.org/wiki/Knuth–Morris–Pratt_algorithm

http://en.wikipedia.org/wiki/Boyer–Moore_string_search_algor...