Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

fMRI software bugs could upend years of research

theregister.co.uk

Post reply on HN
Print view View original

171–180 of 192 posts

Re: fMRI software bugs could upend years of research

#171

Post by nonbel »

mattkrause wrote: post #164
nonbel wrote: post #141

Earlier quoted context omitted.

The key point is that I am saying the second quote does say the null is false. This is where we differ. The actual null hypothesis they are testing is different than the one they actually care and talk about. It includes additional auxilliary assumptions that they say are wrong. What gets plugged in as the null hypothesis is not a problem in the realm of statistics. That is a choice of the researcher. The stats algor…

That sentence is closer to a link than actual content. If you look at paper (14), it has the following graph: http://imgur.com/a/oxSqp It shows that the null model is perfectly appropriate for individual voxels under the event-related paradigms. The FWER there is indistinguishable from 5%, as it should be if the null were true. It also shows that FWER for the null model is inflated by ~6x for blocked paradigms. As th…

Maybe this will help. I am thinking of the same distinction discussed by Meehl in Figure 2 of this paper:

"The poor social scientist, confronted with the twofold problem of dangerous inferential passage (right-to-left) in Figure 2 is rescued as to the (H → O) problem by the statistician. Comforted by these “objective” inferential tools (formulas and tables), the social scientist easily forgets about the far more serious, and less tractable, (T → H) problem, which the statistics text does not address." http://rhowell.ba.ttu.edu/Meehl1.pdf

Here, we are not even yet talking about the substantive theory, although that issue will also exist. There is also a difference between the statistical hypothesis (the mathematical object) and what the researcher wants to test. I think you are conflating those two hypotheses.

Re: fMRI software bugs could upend years of research

#172

Post by nonbel »

e12e wrote: post #131
nonbel wrote: post #128

Earlier quoted context omitted.

>"Okay, why do you think the null is crappy?" Because their goal is to determine if some sort of treatment has an effect. If the null is false for other reasons, then statistical significance can't be used to support the existence of a treatment effect. So these would be pointless, pedantic calculations.

@nonbel: Are you saying that an fMRI, when taken of a subject at rest, twice - then what the data here show - is that this is likely to be interpreted as a subject in two different states? And the researchers are ignoring the fact, rather insisting that we should be able to tell that these two states are the same (and then, perhaps tell them from other states)? I see a few ways this could come about: perhaps the way…

>"Are you saying that an fMRI, when taken of a subject at rest, twice - then what the data here show - is that this is likely to be interpreted as a subject in two different states?"

Yes, that is what they seem to be saying. I didn't read the code, or even the paper very closely. However, from what I quoted, they seem to be saying there is some assumption about autocorrelation that introduces what they call "false positives".

I am saying they have mischaracterized the problem. These are true positives.

Re: fMRI software bugs could upend years of research

#173

Post by dewarrn1 »

UVDMAS wrote: post #158

The paper has been rebutted by other researchers who argue that the original results hold: "This technical report revisits the analysis of family-wise error rates in statistical parametric mapping - using random field theory - reported in (Eklund et al., 2015). Contrary to the understandable spin that these sorts of analyses attract, a review of their results suggests that they endorse the use of parametric assumptio…

Sort of. The rebuttal (by Flandin and Friston) suggests that properly-applied parametric statistics of the kind they favor are valid. Eklund et al. wouldn't disagree with that because their own findings support it, but they would point out that not all researchers necessarily adhered to the conservative statistical approach that F&F discuss. More specifically, both sets of authors describe the importance of using a conservative "cluster defining threshold" to identify spatially contiguous 3D blobs of brain activation. Eklund et al. use their findings to raise the question of whether the bulk of fMRI reports were conservative in this regard.

Re: fMRI software bugs could upend years of research

#174

Post by bpchaps »

ihnorton wrote: post #166
bpchaps wrote: post #103

Earlier quoted context omitted.

Yeah, it's pretty frustrating. I tried getting my own data from an fMRI study, but was told that the signed paperwork specifically disallowed this sort of thing. Not even my doctor could request it. The only option I have is to completely withdrawal myself from the study, but that would be a pretty dick move. The other option is to wait until next year when the study's wrapped and then request it. Though, I'm not eve…

FWIW, typical raw fMRI data is mostly useless unless you know the design parameters of the study (stimulus timing, imaging onsets, etc.). Though there are some interesting data-driven analyses techniques, especially for resting-state data. Many research studies in the U.S. are required to do "structural" scans (high-resolution T1 or T2) and send them to a safety read by a radiologist, for liability reasons. At very l…

Right, they did actually give me the surface scan. It was really, really cool, though with really low resolution.

What got me interested in getting the data was this: [1]. It might be difficult to work with, but there was a lot of motivation to learn it.

[1] http://nbviewer.jupyter.org/github/GaelVaroquaux/nilearn_cou...

Re: fMRI software bugs could upend years of research

#175

Post by [deleted] »

ihnorton wrote: post #166
bpchaps wrote: post #103

Earlier quoted context omitted.

Yeah, it's pretty frustrating. I tried getting my own data from an fMRI study, but was told that the signed paperwork specifically disallowed this sort of thing. Not even my doctor could request it. The only option I have is to completely withdrawal myself from the study, but that would be a pretty dick move. The other option is to wait until next year when the study's wrapped and then request it. Though, I'm not eve…

FWIW, typical raw fMRI data is mostly useless unless you know the design parameters of the study (stimulus timing, imaging onsets, etc.). Though there are some interesting data-driven analyses techniques, especially for resting-state data. Many research studies in the U.S. are required to do "structural" scans (high-resolution T1 or T2) and send them to a safety read by a radiologist, for liability reasons. At very l…

[deleted]

Re: fMRI software bugs could upend years of research

#176

Post by [deleted] »

nonbel wrote: post #168
mattkrause wrote: post #164

Earlier quoted context omitted.

That sentence is closer to a link than actual content. If you look at paper (14), it has the following graph: http://imgur.com/a/oxSqp It shows that the null model is perfectly appropriate for individual voxels under the event-related paradigms. The FWER there is indistinguishable from 5%, as it should be if the null were true. It also shows that FWER for the null model is inflated by ~6x for blocked paradigms. As th…

>"They're saying that the pipeline that runs between the raw DICOM output of the scanner and the t-test, which is an equally important part of the testing procedure, is flawed." Yes, in other words the null hypothesis is being rendered false, by some aspect of the analysis pipeline. I am saying that is the correct description of the problem. Describing the problem as excess false positives is confused, because these…

[deleted]

Re: fMRI software bugs could upend years of research

#177

Post by ihnorton »

nonbel wrote: post #168
mattkrause wrote: post #164

Earlier quoted context omitted.

That sentence is closer to a link than actual content. If you look at paper (14), it has the following graph: http://imgur.com/a/oxSqp It shows that the null model is perfectly appropriate for individual voxels under the event-related paradigms. The FWER there is indistinguishable from 5%, as it should be if the null were true. It also shows that FWER for the null model is inflated by ~6x for blocked paradigms. As th…

>"They're saying that the pipeline that runs between the raw DICOM output of the scanner and the t-test, which is an equally important part of the testing procedure, is flawed." Yes, in other words the null hypothesis is being rendered false, by some aspect of the analysis pipeline. I am saying that is the correct description of the problem. Describing the problem as excess false positives is confused, because these…

> Describing the problem as excess false positives is confused, because these are true positives.

This is deeply wrong. A "true positive" in this context would mean that the resting brain activity of multiple subjects is actually correlated with arbitrary length, randomized, moving test windows. Again, the data is untreated; they are imposing arbitrary test windows and looking for increases in brain activity (increased image intensity) correlated with the stimulus-ON sections of the pseudo-treatment.

From reading some of your prior comments on the subject, it seems prudent to point out that the causal link between stimulus and BOLD effect is quite literally observable: http://www.ncbi.nlm.nih.gov/pmc/articles/PMC3130346/

Re: fMRI software bugs could upend years of research

#178

Post by slowmovintarget »

50CNT wrote: post #147
slowmovintarget wrote: post #145

Earlier quoted context omitted.

Quick, don't think of an elephant. There are many things that, when asked, people are suddenly bad at doing. I don't think that's evidence.

I've always found that these "Quick, don't think of 'x'" things are a silly example. You end up thinking of X the moment you read X, just before it gets put in context of "not" and "think". It's like telling the parser not to tokenize a given word. Then there's automaticity, which you have to turn off, which is made harder by "Quick", further setting you up to think further of elephants till you get around to graspin…

So what you're saying is that this is a good example of context affecting the outcome.

Re: fMRI software bugs could upend years of research

#179

Post by nonbel »

ihnorton wrote: post #177
nonbel wrote: post #168

Earlier quoted context omitted.

>"They're saying that the pipeline that runs between the raw DICOM output of the scanner and the t-test, which is an equally important part of the testing procedure, is flawed." Yes, in other words the null hypothesis is being rendered false, by some aspect of the analysis pipeline. I am saying that is the correct description of the problem. Describing the problem as excess false positives is confused, because these…

> Describing the problem as excess false positives is confused, because these are true positives. This is deeply wrong. A "true positive" in this context would mean that the resting brain activity of multiple subjects is actually correlated with arbitrary length, randomized, moving test windows. Again, the data is untreated; they are imposing arbitrary test windows and looking for increases in brain activity (increas…

>"This is deeply wrong. A "true positive" in this context would mean that the resting brain activity of multiple subjects is actually correlated with arbitrary length, randomized, moving test windows."

This is the same misconception I have been trying to dispel. The null hypothesis is not the inverse of that "layman's" statement, it is a specific set of predicted results calculated in a very specific way. It is a mathematical statement, not an English prose statement.

In this case, apparently one part of this calculation (this assumption about autocorrelation, whatever it is doesn't matter to my point) has lead to such a large deviation from the observations that the null model has been rejected. The null model has been rejected correctly. This is not a false positive.

The problem here is not false-positives due to faulty stats. It is the poor mapping between the hypothesis the researchers want to test, and the null hypothesis they are actually testing.

The tools provided by statisticians look like they worked just fine in this case. If the researchers have decided to use them for GIGO, that is not a statistical problem.

Re: fMRI software bugs could upend years of research

#180

Post by kpil »

clord wrote: post #112
acqq wrote: post #56

Earlier quoted context omitted.

Free will as in "there's something that can't be explained by chemistry and electrical signals in the brain" is just an attempt of some believers to rationalize their belief in the kind of the supernatural "soul."

Are you suggesting there is no room for quantum effects in the brain? Honest question because I thought determinism is out of fashion now-a-days for that reason.

I don't quite get why quantum effects are required, and the determinism have to be removed to explain "free will".

The deterministic effects in the brain must be negligible, as you can't reasonably reproduce the internal and external state of 100 billion neurons, swimming in a incomprehensibly complex chemical soup that is altered by the brain itself, at the same time experiencing the changing reality.

Post reply on HN