Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Machine learning of neural representations of emotion identifies suicidal youth

methodsman.com

Post reply on HN
Print view View original

31–40 of 77 posts

Re: Machine learning of neural representations of emotion identifies suicidal youth

#31

Post by nonbel »

jaibot wrote: post #23
nonbel wrote: post #3

Will it be that 10% of people are suicidal and it always predicts non-suicidal? Will it be that accuracy actually means AUC? Will it be that they are reporting predictive skill on the training data?

It's training data. There's 17 suicidal and 17 non-suicidal scans, for a total of 34 scans. They trained 34 models, leaving one scan out each time. Of those 34 models, 31 correctly predicted the left-out scan. IANAStatistician, but this seems like a trash result.

Cross validation is ok if you do it once, but they repeatedly did it and chose the features based on the results. You can't keep adjusting your model/features based on cross validation performance without overfitting to the training data.

Re: Machine learning of neural representations of emotion identifies suicidal youth

#32

Post by eksabajt »

sigstoat wrote: post #10
nieve wrote: post #4

"This study used machine-learning algorithms (Gaussian Naive Bayes) to identify such individuals (17 suicidal ideators versus 17 controls) with high (91%) accuracy, based on their altered functional magnetic resonance imaging neural signatures of death-related and life-related concepts." Anyone with a Nature subscription want to check whether they simply trained their discriminator and then used it on the same data s…

scihub has it. looks like leave-one-out cross validation? "A Gaussian Naive Bayes (GNB) classifier trained on the data of 33 out of 34 participants predicted the group membership of the remaining participant with a high accuracy of 0.91 (P<0.000001), correctly identifying 15 of the 17 suicidal participants and 16 of the 17 controls"

Correct. They also tested leaving out 9 of each group for cross-validation and got 76% accuracy in that case.

Re: Machine learning of neural representations of emotion identifies suicidal youth

#33

Post by wahern »

aplummer wrote: post #19
wildmusings wrote: post #9

Earlier quoted context omitted.

It can't hurt to get rid of the gun if you're suicidal. But would it actually make a difference? Suicide rates across countries aren't related to gun availability.

Suicide is definitely linked to gun availability. "A study by the Harvard School of Public Health of all 50 U.S. states reveals a powerful link between rates of firearm ownership and suicides. Based on a survey of American households conducted in 2002, HSPH Assistant Professor of Health Policy and Management Matthew Miller, Research Associate Deborah Azrael, and colleagues at the School’s Injury Control Research Cent…

Gun ownership in the U.S. is strongly correlated with socio-economic status, locality, etc.

Everybody points to the Australian example, where suicides declined after the 1996 gun control legislation. But unemployment in Australia peaked in 1995 and declined precipitously afterward until 2009.

Given everything we know about suicide rates in other countries, and about changes in suicide rates domestically (e.g. recent increase as gun ownership goes _down_[1]), it would very odd if gun ownership was a root cause of suicide.

That said, in a country with a strong gun culture like the U.S., I would totally expect a generational dip in suicides if we substantially removed access to guns. But then I'd expect it to normalize when suicidal individuals became more comfortable with other methods. Just like with mass shootings, there's a strong imitation effect. Take away the model that people imitate and it might be awhile until there's a regression to the mean.

Even so, that's still reasonable justification for limiting access to guns--saving tens of thousands of individuals. I'm not sure I'd agree with such a policy prescription because of the insane gun politics, but it's quite defensible from a public health perspective.

[1] Number of guns have increased but they're concentrated in fewer households.

Re: Machine learning of neural representations of emotion identifies suicidal youth

#34

Post by loeg »

Doesn't 91% seem far too low to be useful for the general population? Consider that only 7% of the background population experiences one or more depressive episode per year[0] (edit: okay maybe 8% in youth). Assuming independence and using the higher 8% background rate figure for youth, .91 * .08 = 7.3% of the population will receive a true positive result and (1-.91) * (1-.08) = 8.3% of the population will receive a false positive result. This is "pretty bad" — false positives outweigh the true positives — making the value of a positive result useless.

(Consider what happens to people so-diagnosed as suicidal when in fact they are not (false positives). Involuntary psychiatric imprisonment is a terrible thing if it isn't absolutely necessary.)

[0]: https://www.healthline.com/health/depression/facts-statistic...

Re: Machine learning of neural representations of emotion identifies suicidal youth

#35

Post by jjoonathan »

asperous wrote: post #11
nieve wrote: post #4

"This study used machine-learning algorithms (Gaussian Naive Bayes) to identify such individuals (17 suicidal ideators versus 17 controls) with high (91%) accuracy, based on their altered functional magnetic resonance imaging neural signatures of death-related and life-related concepts." Anyone with a Nature subscription want to check whether they simply trained their discriminator and then used it on the same data s…

"On each fold, the trained classifier was tested on the data of the left-out participant. This procedure was reiterated for all 34 possible ways of leaving out one participant, yielding 34 classifications whose averaged accuracies are reported." Sounds like they overfit their cross validation score and reported that. The data is actually available here though: http://www.ccbi.cmu.edu/Suicidal-ideation-NATHUMBEH2017/

How does LOOCV overfit the cross validation score?

Re: Machine learning of neural representations of emotion identifies suicidal youth

#36

Post by jjoonathan »

nonbel wrote: post #31
jaibot wrote: post #23

Earlier quoted context omitted.

It's training data. There's 17 suicidal and 17 non-suicidal scans, for a total of 34 scans. They trained 34 models, leaving one scan out each time. Of those 34 models, 31 correctly predicted the left-out scan. IANAStatistician, but this seems like a trash result.

Cross validation is ok if you do it once, but they repeatedly did it and chose the features based on the results. You can't keep adjusting your model/features based on cross validation performance without overfitting to the training data.

How did they adjust the model/features based on CV performance? It looks to me like they did LOOCV.

Re: Machine learning of neural representations of emotion identifies suicidal youth

#37

Post by nonbel »

jjoonathan wrote: post #36
nonbel wrote: post #31

Earlier quoted context omitted.

Cross validation is ok if you do it once, but they repeatedly did it and chose the features based on the results. You can't keep adjusting your model/features based on cross validation performance without overfitting to the training data.

How did they adjust the model/features based on CV performance? It looks to me like they did LOOCV.

Read the second paragraph I quoted above:

"The features used by the classifier to characterize a participant consisted of a vector of activation levels for several (discriminating) concepts in a set of (discriminating) brain locations. To determine how many and which concepts were most discriminating between ideators and controls, a reiterative procedure analogous to stepwise regression was used, first finding the single most discriminating concept and then the second most discriminating concept, reiterating until the next step reduced the accuracy. A similar procedure was used to determine the most discriminating locations (clusters)."

The features were chosen using the same data as used to assess predictive skill.

Re: Machine learning of neural representations of emotion identifies suicidal youth

#38

Post by perfmode »

asperous wrote: post #11
nieve wrote: post #4

"This study used machine-learning algorithms (Gaussian Naive Bayes) to identify such individuals (17 suicidal ideators versus 17 controls) with high (91%) accuracy, based on their altered functional magnetic resonance imaging neural signatures of death-related and life-related concepts." Anyone with a Nature subscription want to check whether they simply trained their discriminator and then used it on the same data s…

"On each fold, the trained classifier was tested on the data of the left-out participant. This procedure was reiterated for all 34 possible ways of leaving out one participant, yielding 34 classifications whose averaged accuracies are reported." Sounds like they overfit their cross validation score and reported that. The data is actually available here though: http://www.ccbi.cmu.edu/Suicidal-ideation-NATHUMBEH2017/

LOOCV can have high variance, but overfitting?

Re: Machine learning of neural representations of emotion identifies suicidal youth

#39

Post by mabbo »

jjoonathan wrote: post #35
asperous wrote: post #11

Earlier quoted context omitted.

"On each fold, the trained classifier was tested on the data of the left-out participant. This procedure was reiterated for all 34 possible ways of leaving out one participant, yielding 34 classifications whose averaged accuracies are reported." Sounds like they overfit their cross validation score and reported that. The data is actually available here though: http://www.ccbi.cmu.edu/Suicidal-ideation-NATHUMBEH2017/

How does LOOCV overfit the cross validation score?

The simplest answer is that every single data point was used in some part of the training, and then later used to give test validation.

We have no idea how this result would work on a new data point that has not been used in training.

It's bad statistics, bad data science.

Re: Machine learning of neural representations of emotion identifies suicidal youth

#40

Post by A5A5A5A5 »

Gibbon1 wrote: post #8
gremlinsinc wrote: post #2

Maybe they should use this test before gun purchases... I don't think someone suicidal should purchase a gun...hell I don't care if they kill themselves, but lately a lot of suicides were mass suicides...we don't need more of that shit.

Problem is people buy guns when they aren't suicidal. Tip: If you own a gun and are feeling suicidal, give it to a trusted person for safekeeping.

Unless the 'trusted person' is an immediate family member this is illegal in california as of last year http://leginfo.legislature.ca.gov/faces/displayCodeAndBill.x...

I don't know about other states but it's likely any that require background checks for private transfers ("closing the gunshow loophole") would make this behavior illegal.

Post reply on HN