Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

fMRI software bugs could upend years of research

theregister.co.uk

Post reply on HN
Print view View original

41–50 of 192 posts

Re: fMRI software bugs could upend years of research

#41

Post by nonbel »

I would send it back and ask for a detailed description of the null hypothesis they are testing, because they are not clear on this point at all:

>"All of the analyses to this point have been based on resting-state fMRI data, where the null hypothesis should be true."

They are not careful to explicitly define this null hypothesis anywhere, but earlier in the paper they describe some issues with the model used:

>"Resting-state data should not contain systematic changes in brain activity, but our previous work (14) showed that the assumed activity paradigm can have a large impact on the degree of false positives. Several different activity paradigms were therefore used, two block based (B1 and B2) and two event related (E1 and E2); see Table 1 for details."

This means that they actually know the null model to be false and have even written papers about some of the major contributors to this:

>"The main reason for the high familywise error rates seems to be that the global AR(1) auto correlation correction in SPM fails to model the spectra of the residuals" http://www.sciencedirect.com/science/article/pii/S1053811912...

If the null hypothesis is false, it is no wonder they detect this. In fact, if the sample size was larger (they used only n=20/40 here) they would get near 100% false positive rates. The test seems to be telling them the truth, it is a trivial truth, but according to their description it is correct nonetheless.

Edit: I was quoting from the actual paper.

http://www.pnas.org/content/early/2016/06/27/1602413113.full

Re: fMRI software bugs could upend years of research

#42

Post by jessriedel »

Vexs wrote: post #6
williamscales wrote: post #4

"Our results suggest that the principal cause of the invalid cluster inferences is spatial autocorrelation functions that do not follow the assumed Gaussian shape." In other words, researchers cut corners. You should never assume that something is a certain way without rigorously proving it. How did these papers make it past peer review?

They were probably peer-reviewed by biologists, not statisticians. Peer review is mostly just a good think and read for about a month, not a perfect analysis.

...as long as by "read for about a month" you mean "sit on it for a month, then read for a few hours".

Re: fMRI software bugs could upend years of research

#43

Post by sverige »

greenyoda wrote: post #15
honkhonkpants wrote: post #8

Doesn't sound like a straight up bug, but rather unsound statistical methods which can happen with or without software. You get the same problem with finite element analysis software: the operator has to be aware of all the assumptions baked in, and has to ensure that the input conforms to them.

I'd say it's a bug, since the unsound statistical methods are incorporated into the three most common software packages that are specifically intended to be used for fMRI analysis. If these methods don't work well for fMRI data, fMRI software shouldn't be using them. From the paper: "Using this null data with different experimental designs, we estimate the incidence of significant results. In theory, we should find 5…

And it's a little disheartening when it takes this long to find such a significant bug. Lots of work gone to waste. I wonder if the data can be rerun? Sounds like lots of the raw data wasn't archived correctly or at all.

Re: fMRI software bugs could upend years of research

#44

Post by randcraw »

As someone who works in the biomedical imaging business and is also a fan of philosophy, I think this news will matter more to folks in the latter camp. For a couple years now philosophers have insisted that fMRI images prove there is no such thing as free will. Today's revelation should put an end to that whole line of reasoning (and the absurd amount of fatalism that it engendered).

(The back story: Apparently fMRI showed motor signals arising before the cognitive / conscious signals that should have created them, assuming we humans have free will. This has led to the widely adopted belief among philosophers that we humans act before we think, thus we don't and can't act willfully and freely. To wit, science has proven there is no such thing as free will; we're all just automatons.)

Just this week there was an article in The Atlantic on how we all must accept that we're mere robots and we don't really choose our actions (nor can we choose to believe in a god).

Ah well. It seems philosophers STILL haven't learned the importance of applying the scientific method before leaping to a conclusion -- sometimes just to check that someone else didn't just abuse the scientific method.

Re: fMRI software bugs could upend years of research

#45

Post by leemailll »

jessriedel wrote: post #42
Vexs wrote: post #6

Earlier quoted context omitted.

They were probably peer-reviewed by biologists, not statisticians. Peer review is mostly just a good think and read for about a month, not a perfect analysis.

...as long as by "read for about a month" you mean "sit on it for a month, then read for a few hours".

haha, so true

Re: fMRI software bugs could upend years of research

#46

Post by mattkrause »

Alex3917 wrote: post #7

It's already been known for several years that almost all MRI brain scan research is wrong, what exactly is new here?

There's this weird snobbishness about fMRI: it's uniquely terrible, the people are hacks, etc. It seems particularly common amongst first and second-year grad students who are doing something they think is "harder" science. I hate it.

In the right hands, fMRI can be a really powerful technique for probing neural activity in healthy human subjects; in fact, it's one of the only ways to do so that has decent spatial resolution and thus lets you link brain structure and function.

It certainly does have problems. There are plenty of ways to subtly mess up the data analysis or over-interpret results. The experimental design is often lacking, etc.

However, I think these largely reflect the very low barrier of entry to fMRI research--all you really need is a laptop and somewhere willing to sell you scanner time (almost all major universities or hospitals)--rather than some intrinsic limitation of the field. The good work remains very good.

Re: fMRI software bugs could upend years of research

#47

Post by _delirium »

jey wrote: post #5

Title should really read "fMRI" instead of "MRI". The referenced journal article is titled "Cluster failure: Why fMRI inferences for spatial extent have inflated false-positive rates".

Fwiw there's a response to an earlier (preprint) version of that paper from some of the developers of the packages in question.

Preprint version of the "Cluster failure" paper, from last year: http://arxiv.org/abs/1511.01863

Response: http://arxiv.org/abs/1606.08199

Re: fMRI software bugs could upend years of research

#48

Post by tucosan »

randcraw wrote: post #44

As someone who works in the biomedical imaging business and is also a fan of philosophy, I think this news will matter more to folks in the latter camp. For a couple years now philosophers have insisted that fMRI images prove there is no such thing as free will. Today's revelation should put an end to that whole line of reasoning (and the absurd amount of fatalism that it engendered). (The back story: Apparently fMRI…

So have these studies been invalidated by the software bug as well? If so, do you have any pointers? I. e. which were the infamous studies, and did they indeed use the faulty software to derive their conclusions? I'm genuinely interested.

Re: fMRI software bugs could upend years of research

#49

Post by dewarrn1 »

honkhonkpants wrote: post #8

Doesn't sound like a straight up bug, but rather unsound statistical methods which can happen with or without software. You get the same problem with finite element analysis software: the operator has to be aware of all the assumptions baked in, and has to ensure that the input conforms to them.

You're correct that the main finding reflects flawed statistical assumptions made by the major neuroimaging packages (or so these authors contend), but they did uncover a specific bug in one of the three packages (AFNI):

"... a 15-year-old bug was found in 3dClustSim while testing the three software packages (the bug was fixed by the AFNI group as of May 2015, during preparation of this manuscript). The bug essentially reduced the size of the image searched for clusters, underestimating the severity of the multiplicity correction and overestimating significance ..."

Re: fMRI software bugs could upend years of research

#50

Post by Alex3917 »

mattkrause wrote: post #36
goalieca wrote: post #18

Earlier quoted context omitted.

I wasn't in fmri research but some fellow students in my lab were on it. I do know there was a study with dead salmon which were showing up as having "active" brain regions. I have to find it but am on my phone. It's easy to imagine the main difficulty with this is mapping signal to actual thoughts and regions. It's a really complex biology and physics to reduce.

The salmon thing was a statistical problem, not a technical one. fMRI data generate an incredible number of data points: imagine a movie, but in three dimensions, so you get a sequence of x/y/z-volumes. A typical scan has ~128 to 256 voxels in each spatial dimension, for ~1 million voxels per volume. This means that if your analysis contains voxel-by-voxel tests, you're going to be running a huge number of them. Even…

> The salmon thing was a statistical problem, not a technical one.

How is that relevant to the question of whether or not most fMRI research is wrong?

Post reply on HN