Live data from Hacker News

Text Mining South Park

kaylinwalker.com

51–53 of 53 posts

Re: Text Mining South Park

#51
post #12

I was in the process of reading this when I thought to check who this person is. Of course, by that time the site had failed, so I haven't read the whole thing yet. But, it seems to me that the author is falling in to a trap many an unwary data "scientist" falls by not understanding the discipline of Statistics. When one has the entire population data (i.e. a census), rather than a sample, there is no point in carryi…

Hi, I'm the author. I appreciate the time you've taken to read and provide constructive criticism of my work. Here's my full write up (on GitHub, so it should continue to work): https://github.com/walkerkq/textmining_southpark/blob/master... I was working under the assumption that we do not know ALL the words since the show's been renewed through 2019. This covers the first 18 seasons. Additionally, the counting up t…

A simple tf-idf would get you similar results without a t-test.

I think that is what parent is implying.

Re: Text Mining South Park

#52
post #28

Earlier quoted context omitted.

As I said, I could not read the whole thing. As I was skimming, I noticed the tests, tried to load the main page, and I was disconnected. Once again, "words that are most unique" to character is a parameter that can easily be counted from the set of ALL words with no sampling uncertainty because, yes, we have the population.

I wouldn't say easily. Keep in mind that checking if something is "unique," it needs to be checked against every other character as well. For example, the Top 5 Unique Words for Randy Marsh per the analysis are: stan, stanley, lorde, shelly, son I downloaded the dataset and quickly calculated the Top 5 Most Frequently Said Words for Randy from the entire population. Those are: what, stan, yeah, ok, huh All characters…

It's the likelihood part he is bitching about, not the inverse frequency.
Post reply on HN