Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Text Mining South Park

kaylinwalker.com

Post reply on HN
Print view View original

21–30 of 53 posts

Re: Text Mining South Park

#21

Post by minimaxir »

nanis wrote: post #13
nanis wrote: post #12

I was in the process of reading this when I thought to check who this person is. Of course, by that time the site had failed, so I haven't read the whole thing yet. But, it seems to me that the author is falling in to a trap many an unwary data "scientist" falls by not understanding the discipline of Statistics. When one has the entire population data (i.e. a census), rather than a sample, there is no point in carryi…

Also, I am going to go out on a limb here and guess that R's `read.csv` doesn't do what one hopes it would when fed this CSV: 10,3,Brian,"You mean like the time you had tea with Mohammad, the prophet of the Muslim faith? Peter: Come on, Mohammad, let's get some tea. Mr. T: Try my ""Mr. T. ...tea."" " Well, it seems people are not understanding the problem with this line. Here is the screenshot of the original script:…

I took a look at the code in the author's GitHub repository.

The data sources are CSVs in this repository: https://github.com/BobAdamsEE/SouthParkData/

Looks like all the data is preprocessed, with everyone mostly having only 1 line. (Actually, it appears the line you note in 10-3 is broken!) You can make an argument that the script isn't processed correctly, but that's beyond the scope of the analysis, although a note might be helpful.

Re: Text Mining South Park

#22

Post by bobadams5 »

minimaxir wrote: post #21
nanis wrote: post #13

Earlier quoted context omitted.

Also, I am going to go out on a limb here and guess that R's `read.csv` doesn't do what one hopes it would when fed this CSV: 10,3,Brian,"You mean like the time you had tea with Mohammad, the prophet of the Muslim faith? Peter: Come on, Mohammad, let's get some tea. Mr. T: Try my ""Mr. T. ...tea."" " Well, it seems people are not understanding the problem with this line. Here is the screenshot of the original script:…

I took a look at the code in the author's GitHub repository. The data sources are CSVs in this repository: https://github.com/BobAdamsEE/SouthParkData/ Looks like all the data is preprocessed, with everyone mostly having only 1 line. (Actually, it appears the line you note in 10-3 is broken!) You can make an argument that the script isn't processed correctly, but that's beyond the scope of the analysis, although a no…

It's my repository. I'll look at how the python script handles flashback events later today. Thanks for the feedback!

Re: Text Mining South Park

#24

Post by minimaxir »

wodenokoto wrote: post #11

> Reducing the sparsity brought that down to about 3,100 unique words [from 30,600 unique words] What does that mean? Does he remove words that are only said once or twice? Can anyone point me to a text explaining the difference between Identifying Characteristic Words using Log Likelihood and using tfidf . ?

Relevant line in code:

   # remove sparse terms
   all.tdm.75
I believe it corresponds to the tfidf factor.

Re: Text Mining South Park

#25

Post by JoeAltmaier »

nanis wrote: post #12

I was in the process of reading this when I thought to check who this person is. Of course, by that time the site had failed, so I haven't read the whole thing yet. But, it seems to me that the author is falling in to a trap many an unwary data "scientist" falls by not understanding the discipline of Statistics. When one has the entire population data (i.e. a census), rather than a sample, there is no point in carryi…

Hm. The show is still running? Then the show can be considered a sample of what the characters (ok, the writers) will say/put in their mouths. The statistics then have predictive value.

Re: Text Mining South Park

#26

Post by minimaxir »

nanis wrote: post #12

I was in the process of reading this when I thought to check who this person is. Of course, by that time the site had failed, so I haven't read the whole thing yet. But, it seems to me that the author is falling in to a trap many an unwary data "scientist" falls by not understanding the discipline of Statistics. When one has the entire population data (i.e. a census), rather than a sample, there is no point in carryi…

> If I know ALL the words spoken by someone, then I know which words they say the most without resorting to any tests simply by counting.

From the text, the author is performing statistical testing (chi-sq) for which words are most unique to a character, not which words they say the most. (although the two metrics are somewhat correlated)

Re: Text Mining South Park

#27

Post by nanis »

JoeAltmaier wrote: post #25
nanis wrote: post #12

I was in the process of reading this when I thought to check who this person is. Of course, by that time the site had failed, so I haven't read the whole thing yet. But, it seems to me that the author is falling in to a trap many an unwary data "scientist" falls by not understanding the discipline of Statistics. When one has the entire population data (i.e. a census), rather than a sample, there is no point in carryi…

Hm. The show is still running? Then the show can be considered a sample of what the characters (ok, the writers) will say/put in their mouths. The statistics then have predictive value.

Nope. https://en.wikipedia.org/wiki/Sample_%28statistics%29

Re: Text Mining South Park

#28

Post by nanis »

minimaxir wrote: post #26
nanis wrote: post #12

I was in the process of reading this when I thought to check who this person is. Of course, by that time the site had failed, so I haven't read the whole thing yet. But, it seems to me that the author is falling in to a trap many an unwary data "scientist" falls by not understanding the discipline of Statistics. When one has the entire population data (i.e. a census), rather than a sample, there is no point in carryi…

> If I know ALL the words spoken by someone, then I know which words they say the most without resorting to any tests simply by counting. From the text, the author is performing statistical testing (chi-sq) for which words are most unique to a character, not which words they say the most. (although the two metrics are somewhat correlated)

As I said, I could not read the whole thing. As I was skimming, I noticed the tests, tried to load the main page, and I was disconnected.

Once again, "words that are most unique" to character is a parameter that can easily be counted from the set of ALL words with no sampling uncertainty because, yes, we have the population.

Re: Text Mining South Park

#29

Post by JoeAltmaier »

nanis wrote: post #27
JoeAltmaier wrote: post #25

Earlier quoted context omitted.

Hm. The show is still running? Then the show can be considered a sample of what the characters (ok, the writers) will say/put in their mouths. The statistics then have predictive value.

Nope. https://en.wikipedia.org/wiki/Sample_%28statistics%29

By that definition: a complete sample?

Re: Text Mining South Park

#30

Post by lcfcjs »

nanis wrote: post #12

I was in the process of reading this when I thought to check who this person is. Of course, by that time the site had failed, so I haven't read the whole thing yet. But, it seems to me that the author is falling in to a trap many an unwary data "scientist" falls by not understanding the discipline of Statistics. When one has the entire population data (i.e. a census), rather than a sample, there is no point in carryi…

Don't be so negative, its supposed to be a fun exploratory exercise. It's South Park for gods sake. Get over yourself and have some fun. I think this is an awesome project.
Post reply on HN