Machine learning of neural representations of emotion identifies suicidal youth • Hacker News Board

Re: Machine learning of neural representations of emotion identifies suicidal youth

#51

Post by nonbel » Wed, Nov 01, 2017, 12:57 AM UTC

jjoonathan wrote: post #47

nonbel wrote: post #18

Earlier quoted context omitted.

"Machine learning entails training a classifier on a subset of the data and testing the classifier on an independent subset. The crossvalidation procedure iterates through all possible partitionings (folds) of the data, always keeping the training and test sets separate from each other. The main machine learning here uses a GNB classifier (using pooled variance). [...] The features used by the classifier to character…

If they included feature generation in the training process and ran it once per fold, it would be OK, but I still haven't found any evidence that they did this and their wording suggests that they did not. Good catch.

Anyway that isn't what they did. From the supplements:

"To identify the most discriminating concepts, a reiterative procedure analogous to stepwise regression was performed. In the first iteration, the group classification was performed using only one concept at a time, determining which single concept of the 30 resulted in the highest classification accuracy. In the second iteration, the classification was performed using pairs of concepts, namely the single concept that produced the highest accuracy in the first iteration as well as each of the 29 other concepts. All pairs that produced at least as high an accuracy as achieved on the previous iteration, were explored in the third iteration, where triplets of concepts were used, namely the pairs that produced the highest accuracy in the previous iteration, plus each of the remaining 28 concepts. Such stepwise addition of discriminating concepts continued until adding any one of the remaining concepts resulted in a decrease in accuracy. An analogous procedure identified the most discriminating locations."

But I still think even in your case they are doing:

  train: abc; val: d -> score1/ features0 -> features1
  train: abd; val: c -> score2/ features1 -> features2
  ...etc

score2/features1 would all contain info from c, etc.

Re: Machine learning of neural representations of emotion identifies suicidal youth

#52

Post by loeg » Wed, Nov 01, 2017, 1:02 AM UTC

hk__2 wrote: post #42

loeg wrote: post #34

Doesn't 91% seem far too low to be useful for the general population? Consider that only 7% of the background population experiences one or more depressive episode per year[0] (edit: okay maybe 8% in youth). Assuming independence and using the higher 8% background rate figure for youth, .91 * .08 = 7.3% of the population will receive a true positive result and (1-.91) * (1-.08) = 8.3% of the population will receive a…

> I don't have the stats grounding to come up with the proportion of true positives to false positives, but I suspect this would be "pretty bad" — vastly more false positives than true positives IANAStatistician, but let’s consider the system is right 91% of the time and we try to detect those 7% you mentioned. Let’s take 1000 people. 70 people are depressive and 930 aren’t. Out of those, 70 0.91=63 will be correctly…

Yeah. I came to largely the same conclusion but used the 8% number for youth, given the current title.

Re: Machine learning of neural representations of emotion identifies suicidal youth

#53

Post by yorwba » Wed, Nov 01, 2017, 1:12 AM UTC

nonbel wrote: post #37

jjoonathan wrote: post #36

Earlier quoted context omitted.

How did they adjust the model/features based on CV performance? It looks to me like they did LOOCV.

Read the second paragraph I quoted above: "The features used by the classifier to characterize a participant consisted of a vector of activation levels for several (discriminating) concepts in a set of (discriminating) brain locations. To determine how many and which concepts were most discriminating between ideators and controls, a reiterative procedure analogous to stepwise regression was used, first finding the si…

That quote does not support your summary, unless you are basing it on the information not explicitly mentioned. (I.e. they didn't say that they were only using training data to select features, but if they are any competent, they did.)

Re: Machine learning of neural representations of emotion identifies suicidal youth

#54

Post by wakkaflokka » Wed, Nov 01, 2017, 1:14 AM UTC

nonbel wrote: post #31

jaibot wrote: post #23

Earlier quoted context omitted.

It's training data. There's 17 suicidal and 17 non-suicidal scans, for a total of 34 scans. They trained 34 models, leaving one scan out each time. Of those 34 models, 31 correctly predicted the left-out scan. IANAStatistician, but this seems like a trash result.

Cross validation is ok if you do it once, but they repeatedly did it and chose the features based on the results. You can't keep adjusting your model/features based on cross validation performance without overfitting to the training data.

In this case nested cross-validation would have been the proper way to do this. Run your entire model selection process (scaling - feature selection w/ CV - model selection - hyper paramter tuning w/ CV) on each of the folds in the outter CV loop. That will tell you how good your process is at building a model that generalizes.

Re: Machine learning of neural representations of emotion identifies suicidal youth

#55

Post by nonbel » Wed, Nov 01, 2017, 1:22 AM UTC

yorwba wrote: post #53

nonbel wrote: post #37

Earlier quoted context omitted.

Read the second paragraph I quoted above: "The features used by the classifier to characterize a participant consisted of a vector of activation levels for several (discriminating) concepts in a set of (discriminating) brain locations. To determine how many and which concepts were most discriminating between ideators and controls, a reiterative procedure analogous to stepwise regression was used, first finding the si…

That quote does not support your summary, unless you are basing it on the information not explicitly mentioned. (I.e. they didn't say that they were only using training data to select features, but if they are any competent, they did.)

See the last part of this post: https://news.ycombinator.com/item?id=15598117

Can you provide pseudocode consistent with what they described (in the post you responding to) that wouldn't lead to leakage? I can't see it.

Re: Machine learning of neural representations of emotion identifies suicidal youth

#56

Post by m3kw9 » Wed, Nov 01, 2017, 1:56 AM UTC

Basically you extract a matrix representation of the active or inactive regions that is classified and have DNN learn it like you would learn images, is that a correct assumption?

Re: Machine learning of neural representations of emotion identifies suicidal youth

#57

Post by yorwba » Wed, Nov 01, 2017, 2:17 AM UTC

nonbel wrote: post #55

yorwba wrote: post #53

Earlier quoted context omitted.

That quote does not support your summary, unless you are basing it on the information not explicitly mentioned. (I.e. they didn't say that they were only using training data to select features, but if they are any competent, they did.)

See the last part of this post: https://news.ycombinator.com/item?id=15598117 Can you provide pseudocode consistent with what they described (in the post you responding to) that wouldn't lead to leakage? I can't see it.

Select a training set, leaving out one sample for validation. For all features, train a classifier on the training set using that feature. Keep the one that gives the highest discrimination score on the training set. Repeat with more features. Then evaluate the final classifier on the validation sample, which has so far not been seen in any of the steps. The result provides an estimate of the risk on unseen data from the same distribution.

To get the estimation variance down, you can repeat this for all possible choices of validation sample. That means, you start the feature selection process on the new training set over from scratch and obtain another risk estimate. If they kept the features selected earlier, that estimate would be "contaminated" and not independent, but if they correctly start over, the procedure is valid.

Re: Machine learning of neural representations of emotion identifies suicidal youth

#58

Post by verall » Wed, Nov 01, 2017, 2:22 AM UTC

avip wrote: post #43

It's the kind of "studies" you call BS on first, then go on to figure out the details. Not a very scientific process for sure, but always produces the correct result. https://www.naturalblaze.com/2017/03/scandal-mri-brain-imagi...

At least post the original article[0] if not the paper[1], rather than some weird alt-health website.

[0] https://www.sciencealert.com/a-bug-in-fmri-software-could-in... [1] http://www.pnas.org/content/113/28/7900.abstract

Re: Machine learning of neural representations of emotion identifies suicidal youth

#59

Post by nonbel » Wed, Nov 01, 2017, 3:05 AM UTC

yorwba wrote: post #57

nonbel wrote: post #55

Earlier quoted context omitted.

See the last part of this post: https://news.ycombinator.com/item?id=15598117 Can you provide pseudocode consistent with what they described (in the post you responding to) that wouldn't lead to leakage? I can't see it.

Select a training set, leaving out one sample for validation. For all features, train a classifier on the training set using that feature. Keep the one that gives the highest discrimination score on the training set. Repeat with more features. Then evaluate the final classifier on the validation sample, which has so far not been seen in any of the steps. The result provides an estimate of the risk on unseen data from…

My understanding is you are saying create N (N=34 in this case) different parallel models that use different features/etc. Then take the average (or whatever summary stat) of the accuracies to get the predictive skill.

When we want to use these models, we run new/test data through all N=34 models in parallel and calculate a prediction from each. Then somehow these predictions need to be combined (one again an average, etc). This is the average of the predictions, not accuracies/whatever.

Where was the step combining these predictions present during the training? It seems your scheme necessarily calculates an accuracy based on a different process than needs to be applied to new data.

Re: Machine learning of neural representations of emotion identifies suicidal youth

#60

Post by yorwba » Wed, Nov 01, 2017, 3:45 AM UTC

nonbel wrote: post #59

yorwba wrote: post #57

Earlier quoted context omitted.

Select a training set, leaving out one sample for validation. For all features, train a classifier on the training set using that feature. Keep the one that gives the highest discrimination score on the training set. Repeat with more features. Then evaluate the final classifier on the validation sample, which has so far not been seen in any of the steps. The result provides an estimate of the risk on unseen data from…

My understanding is you are saying create N (N=34 in this case) different parallel models that use different features/etc. Then take the average (or whatever summary stat) of the accuracies to get the predictive skill. When we want to use these models, we run new/test data through all N=34 models in parallel and calculate a prediction from each. Then somehow these predictions need to be combined (one again an average…

No, when you want to classify a new sample, you take a model trained on the complete labeled data you have and use the prediction of that. The validation procedure using those 34 models trained on subsets of the data is just to tell you how accurate you should expect the result to be. Afterwards, you can throw those models away.

Of course you could build an ensemble model, but if you want to know the expected accuracy of doing that, you need to include the ensemble-building into your validation procedure. (Or use some theorem that lets you estimate the ensemble performance from that of individual models, if that is possible.)