Machine learning of neural representations of emotion identifies suicidal youth

Post by nonbel » Wed, Nov 01, 2017, 4:02 AM UTC

yorwba wrote: post #57

nonbel wrote: post #55

Earlier quoted context omitted.

See the last part of this post: https://news.ycombinator.com/item?id=15598117 Can you provide pseudocode consistent with what they described (in the post you responding to) that wouldn't lead to leakage? I can't see it.

Select a training set, leaving out one sample for validation. For all features, train a classifier on the training set using that feature. Keep the one that gives the highest discrimination score on the training set. Repeat with more features. Then evaluate the final classifier on the validation sample, which has so far not been seen in any of the steps. The result provides an estimate of the risk on unseen data from…

Post by jacalata » Wed, Nov 01, 2017, 5:21 AM UTC

matt4077 wrote: post #44

onetwotree wrote: post #12

Earlier quoted context omitted.

17 subjects per group is extremely small. Looking at it either from a machine learning or statistical point of view, using such a small sample is problematic. This is the chronic issue with fMRI studies, since administering an fMRI is extremely expensive, and has led to some very difficult to reproduce results in the field.

People love the "n=XX is far too little data!" argument, yet it's more complicated than that. Sometimes 600,000 is too little, yet sometimes 17 is enough. Example: you believe a newly found plant species is toxic. You give it to 17 "grad students volunteers", while giving a placebo to 17 others. All in the first group die aa gruesome death within 20 hours. None of the others do. Result: yes significance. (also: tenur…

Post by asperous » Wed, Nov 01, 2017, 6:37 AM UTC

perfmode wrote: post #38

asperous wrote: post #11

Earlier quoted context omitted.

"On each fold, the trained classifier was tested on the data of the left-out participant. This procedure was reiterated for all 34 possible ways of leaving out one participant, yielding 34 classifications whose averaged accuracies are reported." Sounds like they overfit their cross validation score and reported that. The data is actually available here though: http://www.ccbi.cmu.edu/Suicidal-ideation-NATHUMBEH2017/

LOOCV can have high variance, but overfitting?

Post by [deleted] » Wed, Nov 01, 2017, 6:43 AM UTC

jjoonathan wrote: post #45

mabbo wrote: post #39

Earlier quoted context omitted.

The simplest answer is that every single data point was used in some part of the training, and then later used to give test validation. We have no idea how this result would work on a new data point that has not been used in training. It's bad statistics, bad data science.

Multi-fold cross validation is an established technique. At no point is any data that was used to train a model also used to validate it. > One round of cross-validation involves partitioning a sample of data into complementary subsets, performing the analysis on one subset (called the training set), and validating the analysis on the other subset (called the validation set or testing set). To reduce variability, mul…

Post by yorwba » Wed, Nov 01, 2017, 6:53 AM UTC

nonbel wrote: post #61

yorwba wrote: post #57

Earlier quoted context omitted.

Select a training set, leaving out one sample for validation. For all features, train a classifier on the training set using that feature. Keep the one that gives the highest discrimination score on the training set. Repeat with more features. Then evaluate the final classifier on the validation sample, which has so far not been seen in any of the steps. The result provides an estimate of the risk on unseen data from…

>"when you want to classify a new sample, you take a model trained on the complete labeled data you have and use the prediction of that." Using which set of features? You have 34 different models with different features...