Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Machine learning of neural representations of emotion identifies suicidal youth

methodsman.com

Post reply on HN
Print view View original

41–50 of 77 posts

Re: Machine learning of neural representations of emotion identifies suicidal youth

#41

Post by smhost »

perfmode wrote: post #38
asperous wrote: post #11

Earlier quoted context omitted.

"On each fold, the trained classifier was tested on the data of the left-out participant. This procedure was reiterated for all 34 possible ways of leaving out one participant, yielding 34 classifications whose averaged accuracies are reported." Sounds like they overfit their cross validation score and reported that. The data is actually available here though: http://www.ccbi.cmu.edu/Suicidal-ideation-NATHUMBEH2017/

LOOCV can have high variance, but overfitting?

What's the smallest sample size anyone has used to get > 90% on the MNIST set?

Re: Machine learning of neural representations of emotion identifies suicidal youth

#42

Post by hk__2 »

loeg wrote: post #34

Doesn't 91% seem far too low to be useful for the general population? Consider that only 7% of the background population experiences one or more depressive episode per year[0] (edit: okay maybe 8% in youth). Assuming independence and using the higher 8% background rate figure for youth, .91 * .08 = 7.3% of the population will receive a true positive result and (1-.91) * (1-.08) = 8.3% of the population will receive a…

> I don't have the stats grounding to come up with the proportion of true positives to false positives, but I suspect this would be "pretty bad" — vastly more false positives than true positives

IANAStatistician, but let’s consider the system is right 91% of the time and we try to detect those 7% you mentioned. Let’s take 1000 people. 70 people are depressive and 930 aren’t. Out of those, 700.91=63 will be correctly classified as depressive by the system and 9300.91=846 will be correctly classified as non-depressive.

That leaves us with 63 positives, 846 negatives, 7 false negatives and 84 false positives. False positives largely outnumber false negatives, but they also outnumber the true positives.

(if a statistician read this, please correct me if I’m wrong)

Re: Machine learning of neural representations of emotion identifies suicidal youth

#44

Post by matt4077 »

onetwotree wrote: post #12
nieve wrote: post #4

"This study used machine-learning algorithms (Gaussian Naive Bayes) to identify such individuals (17 suicidal ideators versus 17 controls) with high (91%) accuracy, based on their altered functional magnetic resonance imaging neural signatures of death-related and life-related concepts." Anyone with a Nature subscription want to check whether they simply trained their discriminator and then used it on the same data s…

17 subjects per group is extremely small. Looking at it either from a machine learning or statistical point of view, using such a small sample is problematic. This is the chronic issue with fMRI studies, since administering an fMRI is extremely expensive, and has led to some very difficult to reproduce results in the field.

People love the "n=XX is far too little data!" argument, yet it's more complicated than that. Sometimes 600,000 is too little, yet sometimes 17 is enough.

Example: you believe a newly found plant species is toxic. You give it to 17 "grad students volunteers", while giving a placebo to 17 others. All in the first group die aa gruesome death within 20 hours. None of the others do.

Result: yes significance. (also: tenure!)

I'm not saying that this study is significant (the statistics seem to be slightly beyond my event horizon), and your criticism also stops short of an outright dismissal of the research. But sample size alone makes for a bad measure of quality. Yes, even p-values are better.

Re: Machine learning of neural representations of emotion identifies suicidal youth

#45

Post by jjoonathan »

mabbo wrote: post #39
jjoonathan wrote: post #35

Earlier quoted context omitted.

How does LOOCV overfit the cross validation score?

The simplest answer is that every single data point was used in some part of the training, and then later used to give test validation. We have no idea how this result would work on a new data point that has not been used in training. It's bad statistics, bad data science.

Multi-fold cross validation is an established technique. At no point is any data that was used to train a model also used to validate it.

> One round of cross-validation involves partitioning a sample of data into complementary subsets, performing the analysis on one subset (called the training set), and validating the analysis on the other subset (called the validation set or testing set). To reduce variability, multiple rounds of cross-validation are performed using different partitions, and the validation results are combined (e.g. averaged) over the rounds to estimate a final predictive model.

https://en.wikipedia.org/wiki/Cross-validation_(statistics)

You might object that it's difficult to achieve the level of inter-fold isolation required to make the technique sound, and indeed if you search the comments there's some question as to whether or not there was an information leak in their feature selection process. In that sense calling it "bad statistics, bad data science" might be reasonable, but it's also a powerful technique, so I don't think it's reasonable to dismiss out of hand without being more specific.

Re: Machine learning of neural representations of emotion identifies suicidal youth

#46

Post by matt4077 »

killjoywashere wrote: post #29
onetwotree wrote: post #12

Earlier quoted context omitted.

17 subjects per group is extremely small. Looking at it either from a machine learning or statistical point of view, using such a small sample is problematic. This is the chronic issue with fMRI studies, since administering an fMRI is extremely expensive, and has led to some very difficult to reproduce results in the field.

I believe some pretty fundamental fMRI spatial autocorrelation functions have been called into question as well (1). Sounds a bit like PowerPoint's autocontent wizard. (1) http://www.pnas.org/content/113/28/7900.abstract

Oh, but the paradigm-shifting revolution of a paper, sorry: "poster" surely is "Neural correlates of interspecies perspective taking in the post-mortem Atlantic Salmon": http://prefrontal.org/files/posters/Bennett-Salmon-2009.pdf

Re: Machine learning of neural representations of emotion identifies suicidal youth

#47

Post by jjoonathan »

nonbel wrote: post #18
nonbel wrote: post #3

Will it be that 10% of people are suicidal and it always predicts non-suicidal? Will it be that accuracy actually means AUC? Will it be that they are reporting predictive skill on the training data?

"Machine learning entails training a classifier on a subset of the data and testing the classifier on an independent subset. The crossvalidation procedure iterates through all possible partitionings (folds) of the data, always keeping the training and test sets separate from each other. The main machine learning here uses a GNB classifier (using pooled variance). [...] The features used by the classifier to character…

If they included feature generation in the training process and ran it once per fold, it would be OK, but I still haven't found any evidence that they did this and their wording suggests that they did not. Good catch.

Re: Machine learning of neural representations of emotion identifies suicidal youth

#48

Post by matt4077 »

lottin wrote: post #14
nonbel wrote: post #3

Will it be that 10% of people are suicidal and it always predicts non-suicidal? Will it be that accuracy actually means AUC? Will it be that they are reporting predictive skill on the training data?

Not only that, the researchers admit that 80% of suicidal people deny being suicidal. Then, how can they be sure than the ones in the control group are not suicidal?

That wouldn't call the results into question. It would, in fact, strengthen them.

That's because the measured difference between the groups would be lower (because the real difference would be lower if the groups are more alike than you think).

Say you're testing a drug that's supposed to make people taller. You don't know it yet, but it really does make everyone grow 10cm overnight. You give it to half of your volunteers, and the other half gets placebo. The next day you find that the first group grew by 10cm compared to the control.

Now say your grad student messed up and half of the control group also got the real thing instead of placebo. Those also grew by 10cm, making the average in the control group 5cm, and your treatment group's effect is suddenly lower.

Re: Machine learning of neural representations of emotion identifies suicidal youth

#49

Post by nonbel »

jjoonathan wrote: post #47
nonbel wrote: post #18

Earlier quoted context omitted.

"Machine learning entails training a classifier on a subset of the data and testing the classifier on an independent subset. The crossvalidation procedure iterates through all possible partitionings (folds) of the data, always keeping the training and test sets separate from each other. The main machine learning here uses a GNB classifier (using pooled variance). [...] The features used by the classifier to character…

If they included feature generation in the training process and ran it once per fold, it would be OK, but I still haven't found any evidence that they did this and their wording suggests that they did not. Good catch.

>"If they included feature generation in the training process and ran it once per fold, it would be OK"

How so? The data used as validation in one fold would be used to determine features in the next...

Re: Machine learning of neural representations of emotion identifies suicidal youth

#50

Post by wildmusings »

aplummer wrote: post #19
wildmusings wrote: post #9

Earlier quoted context omitted.

It can't hurt to get rid of the gun if you're suicidal. But would it actually make a difference? Suicide rates across countries aren't related to gun availability.

Suicide is definitely linked to gun availability. "A study by the Harvard School of Public Health of all 50 U.S. states reveals a powerful link between rates of firearm ownership and suicides. Based on a survey of American households conducted in 2002, HSPH Assistant Professor of Health Policy and Management Matthew Miller, Research Associate Deborah Azrael, and colleagues at the School’s Injury Control Research Cent…

https://en.m.wikipedia.org/wiki/List_of_countries_by_suicide...

Lots of Europe has a higher suicide rate than the US. Within the US, as the other response said, gun ownership is confounded by all sorts of other factors.

Post reply on HN