11–20 of 53 posts
What does that mean? Does he remove words that are only said once or twice?
Can anyone point me to a text explaining the difference between Identifying Characteristic Words using Log Likelihood and using tfidf. ?
But, it seems to me that the author is falling in to a trap many an unwary data "scientist" falls by not understanding the discipline of Statistics.
When one has the entire population data (i.e. a census), rather than a sample, there is no point in carrying out statistical tests.
If I know ALL the words spoken by someone, then I know which words they say the most without resorting to any tests simply by counting.
No concept of "statistical significance" is applicable because there is no sample. We can calculate the population value of any parameter we can think of, because, we have the entire population (in this specific instance, ALL the words spoken by all the characters).
FYI, all budding data "scientists" ...
I was in the process of reading this when I thought to check who this person is. Of course, by that time the site had failed, so I haven't read the whole thing yet. But, it seems to me that the author is falling in to a trap many an unwary data "scientist" falls by not understanding the discipline of Statistics. When one has the entire population data (i.e. a census), rather than a sample, there is no point in carryi…
10,3,Brian,"You mean like the time you had tea with
Mohammad, the prophet of the Muslim faith?
Peter:
Come on, Mohammad, let's get some tea.
Mr. T:
Try my ""Mr. T. ...tea.""
"
Brian: You mean like the time you had tea with Mohammad, the prophet of the Muslim faith? [flashback #3]
Peter: Come on, Mohammad, let's get some tea. [Mohammad is covered by a black box with the words "IMAGE CENSORED BY FOX" printed several times from top to bottom inside the box. They stop at a tea stand.]
Mr. T: Try my "Mr. T. ...tea." [squints]
However, R's read.csv will assign all three characters' speech to Brian: http://imgur.com/gLpPKdl
> x[596, ]
Season Episode Character
596 10 3 Brian
Line
596 You mean like the time you had tea with Mohammad, the prophet of the Muslim faith? \nPeter:\nCome on, Mohammad, let's get some tea. \n
> x[597,]
Season Episode Character
597 10 3 Brian
Line
597 You mean like the time you had tea with Mohammad, the prophet of the Muslim faith? \nPeter:\nCome on, Mohammad, let's get some tea. \nMr. T:\nTry my "Mr. T. ...tea." \n
PS: In addition, both Muhammad and Mohammad appear, presumably under-counting the references to the prophet.
I was in the process of reading this when I thought to check who this person is. Of course, by that time the site had failed, so I haven't read the whole thing yet. But, it seems to me that the author is falling in to a trap many an unwary data "scientist" falls by not understanding the discipline of Statistics. When one has the entire population data (i.e. a census), rather than a sample, there is no point in carryi…
This also strongly implies you think the author is a 'budding data scientist' out of his/her league.
This is very much a 'sample' given the context that South Park is still releasing new episodes.
FYI all elitist 'statisticians' ...
I was in the process of reading this when I thought to check who this person is. Of course, by that time the site had failed, so I haven't read the whole thing yet. But, it seems to me that the author is falling in to a trap many an unwary data "scientist" falls by not understanding the discipline of Statistics. When one has the entire population data (i.e. a census), rather than a sample, there is no point in carryi…
This implies there aren't future episodes upon which this type of statistical analysis could be applied. This also strongly implies you think the author is a 'budding data scientist' out of his/her league. This is very much a 'sample' given the context that South Park is still releasing new episodes. FYI all elitist 'statisticians' ...
As far as I can tell, there are a lot of people out of their leagues going around with the title "data scientist".
This is not a sample. This is a census at this point in time. The fact that there will be another population tomorrow does not change the fact that you have the entire population of all words spoken by all characters up to today.
I am not a statistician. I am an economist who knows enough about statistics and econometrics to know when a significance test is applicable.
Also, do note the issue that R's csv parsing is going to mis-attribute some characters' speech to others. GIGO speaks loud.
I was in the process of reading this when I thought to check who this person is. Of course, by that time the site had failed, so I haven't read the whole thing yet. But, it seems to me that the author is falling in to a trap many an unwary data "scientist" falls by not understanding the discipline of Statistics. When one has the entire population data (i.e. a census), rather than a sample, there is no point in carryi…
Also, I am going to go out on a limb here and guess that R's `read.csv` doesn't do what one hopes it would when fed this CSV: 10,3,Brian,"You mean like the time you had tea with Mohammad, the prophet of the Muslim faith? Peter: Come on, Mohammad, let's get some tea. Mr. T: Try my ""Mr. T. ...tea."" " Well, it seems people are not understanding the problem with this line. Here is the screenshot of the original script:…
I was in the process of reading this when I thought to check who this person is. Of course, by that time the site had failed, so I haven't read the whole thing yet. But, it seems to me that the author is falling in to a trap many an unwary data "scientist" falls by not understanding the discipline of Statistics. When one has the entire population data (i.e. a census), rather than a sample, there is no point in carryi…
Also, I am going to go out on a limb here and guess that R's `read.csv` doesn't do what one hopes it would when fed this CSV: 10,3,Brian,"You mean like the time you had tea with Mohammad, the prophet of the Muslim faith? Peter: Come on, Mohammad, let's get some tea. Mr. T: Try my ""Mr. T. ...tea."" " Well, it seems people are not understanding the problem with this line. Here is the screenshot of the original script:…
Earlier quoted context omitted.
Also, I am going to go out on a limb here and guess that R's `read.csv` doesn't do what one hopes it would when fed this CSV: 10,3,Brian,"You mean like the time you had tea with Mohammad, the prophet of the Muslim faith? Peter: Come on, Mohammad, let's get some tea. Mr. T: Try my ""Mr. T. ...tea."" " Well, it seems people are not understanding the problem with this line. Here is the screenshot of the original script:…
Just tested, it handles that fine. (R 3.1.3)
Earlier quoted context omitted.
Also, I am going to go out on a limb here and guess that R's `read.csv` doesn't do what one hopes it would when fed this CSV: 10,3,Brian,"You mean like the time you had tea with Mohammad, the prophet of the Muslim faith? Peter: Come on, Mohammad, let's get some tea. Mr. T: Try my ""Mr. T. ...tea."" " Well, it seems people are not understanding the problem with this line. Here is the screenshot of the original script:…
why not? that's a valid single csv record with 4 "columns". When surrounded by quotes it IS legal for a csv entity to span multiple lines.