Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

fMRI software bugs could upend years of research

theregister.co.uk

Post reply on HN
Print view View original

71–80 of 192 posts

Re: fMRI software bugs could upend years of research

#71

Post by nonbel »

mattkrause wrote: post #68
nonbel wrote: post #41

I would send it back and ask for a detailed description of the null hypothesis they are testing, because they are not clear on this point at all: >"All of the analyses to this point have been based on resting-state fMRI data, where the null hypothesis should be true." They are not careful to explicitly define this null hypothesis anywhere, but earlier in the paper they describe some issues with the model used: >"Rest…

Nononono. The null hypothesis here is definitely true. They take resting state data ("lie here while we scan your brain") and pretend that it was acquired during an experiment, using either block or event-related designs. Next, they look for differences between the fake blocks (or events). The null hypothesis is that neural activity is the same across blocks. It must be true because the block boundaries are totally a…

>"The null hypothesis is that neural activity is the same across blocks."

You may have written this before seeing my other response, but I will repeat the point here. You have mentioned only one part of the null hypothesis, there are other assumptions being made that can be violated. From their description, it sounds like they discovered one that is causing the null model to make bad predictions.

Btw, I am using "null hypothesis" and "null model "interchangeably here. Really, null model should be the preferred term but people are so used to "hypothesis".

Re: fMRI software bugs could upend years of research

#72

Post by threatofrain »

Strilanc wrote: post #61
randcraw wrote: post #44

As someone who works in the biomedical imaging business and is also a fan of philosophy, I think this news will matter more to folks in the latter camp. For a couple years now philosophers have insisted that fMRI images prove there is no such thing as free will. Today's revelation should put an end to that whole line of reasoning (and the absurd amount of fatalism that it engendered). (The back story: Apparently fMRI…

Why would this put an end to that line of reasoning? I'd expect it to flare up the debate, not end the debate. They didn't disprove behavior being computed, they demonstrated that a class of data supporting it was useless. The natural reaction to this isn't "okay we give up", it's "better go get some good data". (Also, I don't like mixing the question "Is our behavior computed?" with the question "Does computed behav…

The problem with free will debates has to do with its definition. I think once people start hammering out the definition, free will either becomes a wimpy variant that people didn't want to talk about in the first place, or it becomes so ambitious that it's outside the scope of science to discuss.

If you define free will as the ability to generate and choose options based on constraints, then people get bored of that discussion because it seems to lack the freedom they want. If you define free will as the ability to escape biophysics, then it becomes an unscientific discussion of metaphysics.

But what people really want is the freedom from biophysics, not a discussion of a bounded system to generate and choose options. People want a reality where this state does not need to relate to the one before it, a non-markovy world.

That's why people mix the discussion of behavioral computation and free will. People want the most ambitious form of free will, something too special for computers -- freedom from biophysics, freedom from the prior state's tyranny over its future state.

Re: fMRI software bugs could upend years of research

#75

Post by nonbel »

nickledave wrote: post #69
nonbel wrote: post #41

I would send it back and ask for a detailed description of the null hypothesis they are testing, because they are not clear on this point at all: >"All of the analyses to this point have been based on resting-state fMRI data, where the null hypothesis should be true." They are not careful to explicitly define this null hypothesis anywhere, but earlier in the paper they describe some issues with the model used: >"Rest…

See mattkrause's comment below. I think you might not be understanding what they're testing. They're taking only resting state data, randomly sorting some of it into "pretend active state data" and then asking whether they get any statistically significant difference between these two groups when they shouldn't . But they do . That means the tests they used, the same tests many authors use, are giving false positives…

I think that like mattkrause (and the authors of the current work), you have forgotten that the null hypothesis is something larger than condition 1 == condition 2. There are various other components, usually (somewhat misleadingly) referred to as assumptions, that can also cause the predictions derived from the null model to deviate from the data.

For stuff like a t-test, one parameter value (ie the mean of the distribution) gets all the attention. But this is wrong, it is only one part of the model being tested.

Re: fMRI software bugs could upend years of research

#76

Post by mattkrause »

nonbel wrote: post #71
mattkrause wrote: post #68

Earlier quoted context omitted.

Nononono. The null hypothesis here is definitely true. They take resting state data ("lie here while we scan your brain") and pretend that it was acquired during an experiment, using either block or event-related designs. Next, they look for differences between the fake blocks (or events). The null hypothesis is that neural activity is the same across blocks. It must be true because the block boundaries are totally a…

>"The null hypothesis is that neural activity is the same across blocks." You may have written this before seeing my other response, but I will repeat the point here. You have mentioned only one part of the null hypothesis, there are other assumptions being made that can be violated. From their description, it sounds like they discovered one that is causing the null model to make bad predictions. Btw, I am using "nul…

Can you explain what those assumptions are then, even roughly?

I read the PNAS paper as being an extension of their previous work, not a contradiction of it. They've scaled the data set up (the previous paper had less data), added more packages (the previous paper used only SPM), and ran it as a group analysis (the previous paper was within subjects). They pretty much say that right in the PNAS text:

"That work found a high degree of false positives, up to 70% compared with the expected 5%, likely due to a simplistic temporal autocorrelation model in SPM. It was, however, not clear whether these problems would propagate to group studies. Another unanswered question was the statistical validity of other fMRI software packages. We address these limitations in the current work with an evaluation of group inference with the three most common fMRI software packages [SPM (15, 16), FSL (17), and AFNI (18)]."

Furthermore, the NeuroImage paper says that the AR(1) temporal autocorrelation model in SPM is actually not too problematic for event-related designs:

"Overall, a simple AR(1) model for temporal correlations appears to be adequate for fast designs (E1 and E2) at all three TRs. However, there is a massive inflation of false positive rates at short TRs that is particularly pronounced for slower (block) designs. "

If that's what is bugging you, just ignore the red and yellow bars in the PNAS figures.

Re: fMRI software bugs could upend years of research

#77

Post by dewarrn1 »

_delirium wrote: post #47
jey wrote: post #5

Title should really read "fMRI" instead of "MRI". The referenced journal article is titled "Cluster failure: Why fMRI inferences for spatial extent have inflated false-positive rates".

Fwiw there's a response to an earlier (preprint) version of that paper from some of the developers of the packages in question. Preprint version of the "Cluster failure" paper, from last year: http://arxiv.org/abs/1511.01863 Response: http://arxiv.org/abs/1606.08199

Flaundin & Friston's response is interesting because it essentially endorses the findings of Eklund et al. (except for one element of the Eklund analysis that they suggest is a modest error). F&F believe that by setting one parameter correctly (i.e., using a conservative cluster forming threshold) the validity of their preferred parametric statistical approach is upheld. Eklund et al. might quibble because their take-home message is that non-parametric methods should be used instead, but their findings are not misrepresented by F&F.

Regardless, an open and important question is how often other authors used a sufficiently conservative cluster forming threshold for their fMRI analyses. If nothing else, Eklund et al. will cause future reports to be more cautious in this regard.

Re: fMRI software bugs could upend years of research

#78

Post by jballanc »

The real takeaway lesson from this research should be the vital importance of Open Data to the modern scientific enterprise:

> "lamentable archiving and data-sharing practices" that prevent most of the discipline's body of work being re-analysed.

Keeping data private before publication is (at this point in time) understandable. Once results are published, however, there is no excuse for not depositing the raw data in an open repository for later re-evaluation.

Re: fMRI software bugs could upend years of research

#79

Post by nonbel »

mattkrause wrote: post #76
nonbel wrote: post #71

Earlier quoted context omitted.

>"The null hypothesis is that neural activity is the same across blocks." You may have written this before seeing my other response, but I will repeat the point here. You have mentioned only one part of the null hypothesis, there are other assumptions being made that can be violated. From their description, it sounds like they discovered one that is causing the null model to make bad predictions. Btw, I am using "nul…

Can you explain what those assumptions are then, even roughly? I read the PNAS paper as being an extension of their previous work, not a contradiction of it. They've scaled the data set up (the previous paper had less data), added more packages (the previous paper used only SPM), and ran it as a group analysis (the previous paper was within subjects). They pretty much say that right in the PNAS text: "That work found…

>"Can you explain what those assumptions are then, even roughly?"

I have not looked at the code myself. I only read their description of it. As I quoted earlier:

"assumed activity paradigm can have a large impact on the degree of false positives"

If the assumed activity paradigm affects (what they are calling) the false positive rate, it is obviously part of the model. If they are arbitrarily varying it because they don't know which one is "best", it is clearly wrong.

The t-tests will pick up this wrong assumption if there is enough data to do so (even without knowing the details, remember they say it affects the "false positive" rate). The t-test is doing it's job correctly, however, they are plugging in a known to be incorrect null hypothesis, rendering this testing procedure pointless.

Re: fMRI software bugs could upend years of research

#80

Post by toyg »

jballanc wrote: post #78

The real takeaway lesson from this research should be the vital importance of Open Data to the modern scientific enterprise: > "lamentable archiving and data-sharing practices" that prevent most of the discipline's body of work being re-analysed. Keeping data private before publication is (at this point in time) understandable. Once results are published, however, there is no excuse for not depositing the raw data in…

Uh, where would you store it though? IIRC from the time a relative was going through a chemistry PHD, they produced several GBs of raw data every half hour. Storage is cheap but it's not that cheap...
Post reply on HN