Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Text Mining South Park

kaylinwalker.com

Post reply on HN
Print view View original

31–40 of 53 posts

Re: Text Mining South Park

#31

Post by bonoboTP »

nanis wrote: post #12

I was in the process of reading this when I thought to check who this person is. Of course, by that time the site had failed, so I haven't read the whole thing yet. But, it seems to me that the author is falling in to a trap many an unwary data "scientist" falls by not understanding the discipline of Statistics. When one has the entire population data (i.e. a census), rather than a sample, there is no point in carryi…

Why so bitter and angry? As far as I can see, his calculations make sense and lead to interesting results. Instead of philosophical nitpicking, why not help him improve his understanding by explaining how you would have calculated/formalized/modeled this thing, so the scare-quote data "scientists" can learn something?

By the way, we definitely don't hear all words that these characters speak in their lives. It's implied in the story that there are conversations that we don't get to see in the actual episodes, but nevertheless these imaginary characters speak a lot more. For example we don't see each and every breakfast, lunch and dinner discussion, we don't hear all their words in the classroom etc.

Now of course the sampling isn't random, because the creators obviously "select" the more interesting bits of the characters' lives, but in statistics we always make assumptions that simplify the procedure but are known to be technically wrong.

Re: Text Mining South Park

#32

Post by minimaxir »

nanis wrote: post #28
minimaxir wrote: post #26

Earlier quoted context omitted.

> If I know ALL the words spoken by someone, then I know which words they say the most without resorting to any tests simply by counting. From the text, the author is performing statistical testing (chi-sq) for which words are most unique to a character, not which words they say the most. (although the two metrics are somewhat correlated)

As I said, I could not read the whole thing. As I was skimming, I noticed the tests, tried to load the main page, and I was disconnected. Once again, "words that are most unique" to character is a parameter that can easily be counted from the set of ALL words with no sampling uncertainty because, yes, we have the population.

I wouldn't say easily. Keep in mind that checking if something is "unique," it needs to be checked against every other character as well.

For example, the Top 5 Unique Words for Randy Marsh per the analysis are:

stan, stanley, lorde, shelly, son

I downloaded the dataset and quickly calculated the Top 5 Most Frequently Said Words for Randy from the entire population. Those are:

what, stan, yeah, ok, huh

All characters on the show are saying those words (Except "stan"). That's why log-likelihood/tdfif is used on a per-character basis.

Re: Text Mining South Park

#34

Post by nanis »

JoeAltmaier wrote: post #29
nanis wrote: post #27

Earlier quoted context omitted.

Nope. https://en.wikipedia.org/wiki/Sample_%28statistics%29

By that definition: a complete sample ?

No.

> A complete sample is a set of objects from a parent population that includes ALL such objects that satisfy a set of well-defined selection criteria.[3] For example, a complete sample of Australian men taller than 2m would consist of a list of every Australian male taller than 2m. But it wouldn't include German males, or tall Australian females, or people shorter than 2m ...

So, the entire set of all words spoken by South Park characters, by definition, is the population of all words spoken by South Park characters.

For this to be a complete sample, it needs to be a sample out of a larger population. What is that population?

Re: Text Mining South Park

#35

Post by vsbuffalo »

nanis wrote: post #12

I was in the process of reading this when I thought to check who this person is. Of course, by that time the site had failed, so I haven't read the whole thing yet. But, it seems to me that the author is falling in to a trap many an unwary data "scientist" falls by not understanding the discipline of Statistics. When one has the entire population data (i.e. a census), rather than a sample, there is no point in carryi…

You're treating this sample-is-the-population issue as if it's resolved in the statistics literature. It is not. Gelman has written on this [1][2], as the issue comes up in political science data frequently. As Gelman points out, 50 states are not a sample of states—it's the entire population. Similarly, the Correlates of War [3] data is every militarized international dispute between 1816-2007 that fits certain criteria—it too is not a sample but the entire population.

Treating his population as a large sample of a process that's uncertain or noisy and then applying frequentist statistics is not inherently wrong in the way you say it is. It may be that there's a better way to model the uncertainty in the process than treating the population as a sample, but that's a different point than the one you make.

[1]: http://andrewgelman.com/2009/07/03/how_does_statis/

[2]: http://www.stat.columbia.edu/~gelman/research/published/econ... (see finite population section)

Re: Text Mining South Park

#36

Post by bonoboTP »

vsbuffalo wrote: post #35
nanis wrote: post #12

I was in the process of reading this when I thought to check who this person is. Of course, by that time the site had failed, so I haven't read the whole thing yet. But, it seems to me that the author is falling in to a trap many an unwary data "scientist" falls by not understanding the discipline of Statistics. When one has the entire population data (i.e. a census), rather than a sample, there is no point in carryi…

You're treating this sample-is-the-population issue as if it's resolved in the statistics literature. It is not. Gelman has written on this [1][2], as the issue comes up in political science data frequently. As Gelman points out, 50 states are not a sample of states—it's the entire population. Similarly, the Correlates of War [3] data is every militarized international dispute between 1816-2007 that fits certain crit…

Exactly. There is an interpretation where the "population" is interpreted as a mathematical ideal process (with potentially infinite information content) and any real, physical manifestation is considered a "sample".

The old-school interpretation is stricter and considers both the "population" and the "sample" to be physical real things. It's understandable because these methods were developed for statistics about human populations (note the origin of the terminology), medical studies etc. (The word "statistics" itself derives from "state").

Somehow, frequentist statisticians are usually very conservative and set in one way of thinking and do not even like to entertain an alternative interpretation or paradigm... I'm not sure why it is so.

Re: Text Mining South Park

#37

Post by ppod »

nanis wrote: post #28
minimaxir wrote: post #26

Earlier quoted context omitted.

> If I know ALL the words spoken by someone, then I know which words they say the most without resorting to any tests simply by counting. From the text, the author is performing statistical testing (chi-sq) for which words are most unique to a character, not which words they say the most. (although the two metrics are somewhat correlated)

As I said, I could not read the whole thing. As I was skimming, I noticed the tests, tried to load the main page, and I was disconnected. Once again, "words that are most unique" to character is a parameter that can easily be counted from the set of ALL words with no sampling uncertainty because, yes, we have the population.

I think the idea is that what we are really trying to measure is something unobservable like the underlying nature of the character or the writers' tendencies to give characters certain ways of speaking. We can say that Stan uses a word at a rate certain rate corrected for that words base rate in the corpus, and compare this with the rate for another character. If that difference in rate is very small, it's true that we still know for certain that the difference is absolutely true for this corpus, but it may not reflect any substantive difference between the characters.

If this is the view taken, then the population is all of the text that might have been generated by the data-generating process of the scripts -- things like the writers' mental models of the characters. In this view the actual scripts are just a sample from all of the scripts that could have been written while keeping the variable of interest (the characters' character) constant.

Re: Text Mining South Park

#38

Post by dragonwriter »

vsbuffalo wrote: post #35
nanis wrote: post #12

I was in the process of reading this when I thought to check who this person is. Of course, by that time the site had failed, so I haven't read the whole thing yet. But, it seems to me that the author is falling in to a trap many an unwary data "scientist" falls by not understanding the discipline of Statistics. When one has the entire population data (i.e. a census), rather than a sample, there is no point in carryi…

You're treating this sample-is-the-population issue as if it's resolved in the statistics literature. It is not. Gelman has written on this [1][2], as the issue comes up in political science data frequently. As Gelman points out, 50 states are not a sample of states—it's the entire population. Similarly, the Correlates of War [3] data is every militarized international dispute between 1816-2007 that fits certain crit…

> Similarly, the Correlates of War [3] data is every militarized international dispute between 1816-2007 that fits certain criteria—it too is not a sample but the entire population.

Its the entire population of wars meeting a certain criteria in that time frame. If that is the topic of interest, then it is also the whole population. OTOH, datasets like that are often used in analysis that is intended to apply to, for instance, "what-if" scenarios about hypothetical wars that could have happened in that time frame, in which case the studied population is clearly not the population of interest, but is taken to be -- while there may be specific reasons to criticize this in specific cases for reasons other than "its the whole population, not a sample" -- a representative sample of a broader population.

Re: Text Mining South Park

#39

Post by pg_is_a_butt »

nanis wrote: post #12

I was in the process of reading this when I thought to check who this person is. Of course, by that time the site had failed, so I haven't read the whole thing yet. But, it seems to me that the author is falling in to a trap many an unwary data "scientist" falls by not understanding the discipline of Statistics. When one has the entire population data (i.e. a census), rather than a sample, there is no point in carryi…

uh... you don't know ALL the words... you only know the words spoken by someone that made the final cut of a show included in the study.

FYI, you're an idiot.

Re: Text Mining South Park

#40

Post by sdenton4 »

bonoboTP wrote: post #31
nanis wrote: post #12

I was in the process of reading this when I thought to check who this person is. Of course, by that time the site had failed, so I haven't read the whole thing yet. But, it seems to me that the author is falling in to a trap many an unwary data "scientist" falls by not understanding the discipline of Statistics. When one has the entire population data (i.e. a census), rather than a sample, there is no point in carryi…

Why so bitter and angry? As far as I can see, his calculations make sense and lead to interesting results. Instead of philosophical nitpicking, why not help him improve his understanding by explaining how you would have calculated/formalized/modeled this thing, so the scare-quote data "scientists" can learn something? By the way, we definitely don't hear all words that these characters speak in their lives. It's impl…

But we must protect ourselves from the research parasites! Man the ramparts and ready the harsh words!
Post reply on HN