What tools do you recommend for text mining?

#1

Post by andresmh » Mon, Jun 28, 2010, 12:20 AM UTC

I have a text file with online discussions from a website (about 54 million words) and I would like to do some analysis on it. I have done some basic word frequency counts but I am interested in doing things like clustering to find what are the words that appear together more often. Something like this: http://jcmc.indiana.edu/vol8/issue4/rosen.html#sixth

I'm looking for simple free tools that can allow me to do some basic analysis on the text that can give me a basic understanding of the content of the text. I'm familiar with perl and python primarily.

Thanks.

Re: What tools do you recommend for text mining?

#2

Post by aitoehigie » Mon, Jun 28, 2010, 12:47 AM UTC

Beautiful soup

Re: What tools do you recommend for text mining?

#3

Post by andresmh » Mon, Jun 28, 2010, 12:52 AM UTC

aitoehigie wrote: post #2

Beautiful soup

Isn't Beautiful soup primarily to do screen scraping? I already scraped the content and stored it in a DB. Now I want to do some analysis on the text.

Re: What tools do you recommend for text mining?

#4

Post by edwtjo » Mon, Jun 28, 2010, 1:08 AM UTC

I only know of one, in java, http://www.ontopia.net.

Re: What tools do you recommend for text mining?

#5

Post by andresmh » Mon, Jun 28, 2010, 1:21 AM UTC

edwtjo wrote: post #4

I only know of one, in java, http://www.ontopia.net .

Looks interesting. What have you done with it?

Re: What tools do you recommend for text mining?

#6

Post by Absolute0 » Mon, Jun 28, 2010, 2:43 AM UTC

NLTK is the best resource if you know python. And nothing beats it!

Re: What tools do you recommend for text mining?

#7

Post by brisance » Mon, Jun 28, 2010, 3:11 AM UTC

A lot of resources are in Java.

http://www.cs.waikato.ac.nz/ml/weka/ http://rapid-i.com/

Re: What tools do you recommend for text mining?

#8

Post by andresmh » Mon, Jun 28, 2010, 3:44 AM UTC

Absolute0 wrote: post #6

NLTK is the best resource if you know python. And nothing beats it!

This looks great. It even has a text book!

Re: What tools do you recommend for text mining?

#9

Post by apurva » Mon, Jun 28, 2010, 5:24 AM UTC

andresmh wrote: post #8

Absolute0 wrote: post #6

NLTK is the best resource if you know python. And nothing beats it!

This looks great. It even has a text book!

+1 for nltk... its very very neat!

Re: What tools do you recommend for text mining?

#10

Post by irahul » Mon, Jun 28, 2010, 5:32 AM UTC

andresmh wrote: post #3

aitoehigie wrote: post #2

Beautiful soup

Isn't Beautiful soup primarily to do screen scraping? I already scraped the content and stored it in a DB. Now I want to do some analysis on the text.

"Programming collective intelligence" is a good book if you are willing to invest time in it. If your goal is just to get this task done, NLTK would work for you.