fMRI software bugs could upend years of research • Hacker News Board

Re: fMRI software bugs could upend years of research

#31

Post by leemailll » Mon, Jul 04, 2016, 7:48 PM UTC

williamscales wrote: post #4

"Our results suggest that the principal cause of the invalid cluster inferences is spatial autocorrelation functions that do not follow the assumed Gaussian shape." In other words, researchers cut corners. You should never assume that something is a certain way without rigorously proving it. How did these papers make it past peer review?

Peer-review in biology-related fields doesn't bother too much about the details of your code. Only recently a few journals begin to ask guys to submit the source code if the manuscript is mainly about some new analysis methods. So it is more than likely even if you are reading submitted manuscript of a computer program, you won't read the source code with the paper.

Another thing is that statistical analysis is kind of the missing piece in many researches training. It maybe not be "researchers cut corners", but they might not know some of the prerequirements for the analyses used. Even in this year, 2016 when I talked with a colleague in my institute I found out he didn't know a single assumption for t-test, who is a senior postdoc and used it everyday.

Re: fMRI software bugs could upend years of research

#32

Post by Alex3917 » Mon, Jul 04, 2016, 7:49 PM UTC

acveilleux wrote: post #28

Alex3917 wrote: post #21

Earlier quoted context omitted.

Here are a couple Ben Goldacre articles on neuroscience in general: https://www.theguardian.com/commentisfree/2011/sep/09/bad-sc... For fMRI research specifically, I seem to remember that almost all of the analysis software is closed source, and that comparing the readings between software versions isn't actually meaningful even though a lot of papers do so.

That's actually the same article on two sites. Itself just a layman's piece on an actual article in Nature Neuroscience.

Oops, deleted the second link. The reason I didn't link to the original journal article is that JumpCrisscross specifically asked for a source for a layman.

Re: fMRI software bugs could upend years of research

#33

Post by Alex3917 » Mon, Jul 04, 2016, 7:52 PM UTC

acveilleux wrote: post #27

Alex3917 wrote: post #7

It's already been known for several years that almost all MRI brain scan research is wrong, what exactly is new here?

There's a fair bit of junky data and quite a few analysis methods that cannot compare data accurately between magnets, position wrt isocentre, or even processing software revisions. But your claim is a lot bolder than that.

> There's a fair bit of junky data and quite a few analysis methods that cannot compare data accurately between magnets, position wrt isocentre, or even processing software revisions. But your claim is a lot bolder than that.

What is the prevalence of each of those types of errors? Since AFAIK many of those errors occur in 30%+ of papers, unless you're assuming close to 0% independence, my claim doesn't seem especially bold...

Re: fMRI software bugs could upend years of research

#34

Post by [deleted] » Mon, Jul 04, 2016, 7:53 PM UTC

goalieca wrote: post #18

JumpCrisscross wrote: post #12

Earlier quoted context omitted.

> almost all MRI brain scan research is wrong Source for a layman?

I wasn't in fmri research but some fellow students in my lab were on it. I do know there was a study with dead salmon which were showing up as having "active" brain regions. I have to find it but am on my phone. It's easy to imagine the main difficulty with this is mapping signal to actual thoughts and regions. It's a really complex biology and physics to reduce.

[deleted]

Re: fMRI software bugs could upend years of research

#35

Post by acveilleux » Mon, Jul 04, 2016, 8:02 PM UTC

Alex3917 wrote: post #33

acveilleux wrote: post #27

Earlier quoted context omitted.

There's a fair bit of junky data and quite a few analysis methods that cannot compare data accurately between magnets, position wrt isocentre, or even processing software revisions. But your claim is a lot bolder than that.

> There's a fair bit of junky data and quite a few analysis methods that cannot compare data accurately between magnets, position wrt isocentre, or even processing software revisions. But your claim is a lot bolder than that. What is the prevalence of each of those types of errors? Since AFAIK many of those errors occur in 30%+ of papers, unless you're assuming close to 0% independence, my claim doesn't seem especial…

The methods I am familiar with, mostly dealing with volumetric measurements, registration, and automated segmentation were either robust or had well characterized limitations.

I take some offense to blanketing the entire field of MRI based on a couple articles pointing out that some (admittedly a lot) fMRI experiments are statistically unsound.

Re: fMRI software bugs could upend years of research

#36

Post by mattkrause » Mon, Jul 04, 2016, 8:02 PM UTC

goalieca wrote: post #18

JumpCrisscross wrote: post #12

Earlier quoted context omitted.

> almost all MRI brain scan research is wrong Source for a layman?

I wasn't in fmri research but some fellow students in my lab were on it. I do know there was a study with dead salmon which were showing up as having "active" brain regions. I have to find it but am on my phone. It's easy to imagine the main difficulty with this is mapping signal to actual thoughts and regions. It's a really complex biology and physics to reduce.

The salmon thing was a statistical problem, not a technical one.

fMRI data generate an incredible number of data points: imagine a movie, but in three dimensions, so you get a sequence of x/y/z-volumes. A typical scan has ~128 to 256 voxels in each spatial dimension, for ~1 million voxels per volume.

This means that if your analysis contains voxel-by-voxel tests, you're going to be running a huge number of them. Even if each test has a fairly low false-positive rate (say 0.1%), there are still a huge number of tests, and thus, a huge number of false positives.

There are principled ways of correcting for this; there are also hacky "folk methods" like setting a more stringent false positive threshold. The fish poster argues that the latter doesn't work, using a deliberately silly example.

http://prefrontal.org/files/posters/Bennett-Salmon-2009.pdf

Re: fMRI software bugs could upend years of research

#37

Post by psycr » Mon, Jul 04, 2016, 8:08 PM UTC

trentmb wrote: post #9

williamscales wrote: post #4

"Our results suggest that the principal cause of the invalid cluster inferences is spatial autocorrelation functions that do not follow the assumed Gaussian shape." In other words, researchers cut corners. You should never assume that something is a certain way without rigorously proving it. How did these papers make it past peer review?

n=30 ought to be enough for anybody

Depends how big the effect size is. For putting a loaded revolver next to your head, and pulling the trigger, n=30 is plenty.

Re: fMRI software bugs could upend years of research

#38

Post by leemailll » Mon, Jul 04, 2016, 8:09 PM UTC

Alex3917 wrote: post #21

JumpCrisscross wrote: post #12

Earlier quoted context omitted.

> almost all MRI brain scan research is wrong Source for a layman?

Here are a couple Ben Goldacre articles on neuroscience in general: https://www.theguardian.com/commentisfree/2011/sep/09/bad-sc... For fMRI research specifically, I seem to remember that almost all of the analysis software is closed source, and that comparing the readings between software versions isn't actually meaningful even though a lot of papers do so.

Actually for fMRI studies most people use matlab and share it. It is not really closed-sourced. But I guess many guys in the field won't bother too much to read the codes from a widely used and well-cited matlab script

Re: fMRI software bugs could upend years of research

#39

Post by Alex3917 » Mon, Jul 04, 2016, 8:12 PM UTC

acveilleux wrote: post #35

Alex3917 wrote: post #33

Earlier quoted context omitted.

> There's a fair bit of junky data and quite a few analysis methods that cannot compare data accurately between magnets, position wrt isocentre, or even processing software revisions. But your claim is a lot bolder than that. What is the prevalence of each of those types of errors? Since AFAIK many of those errors occur in 30%+ of papers, unless you're assuming close to 0% independence, my claim doesn't seem especial…

The methods I am familiar with, mostly dealing with volumetric measurements, registration, and automated segmentation were either robust or had well characterized limitations. I take some offense to blanketing the entire field of MRI based on a couple articles pointing out that some (admittedly a lot) fMRI experiments are statistically unsound.

How would you estimate the prevalence of inaccurate research in the field, if not by multiplying each (known) source of error by its estimated prevalence, and also using some estimate of the independence between errors? (And of course including general sources of error that affect all scientific research and aren't specific to any particular field.)

And as far as independence goes, I haven't seen any research to suggest these errors have anything less than 100% independence, although if there is any research suggesting that these errors (or any other errors in science research) are correlated then I'd love to see that.

Re: fMRI software bugs could upend years of research

#40

Post by marmaduke » Mon, Jul 04, 2016, 8:12 PM UTC

williamscales wrote: post #4

"Our results suggest that the principal cause of the invalid cluster inferences is spatial autocorrelation functions that do not follow the assumed Gaussian shape." In other words, researchers cut corners. You should never assume that something is a certain way without rigorously proving it. How did these papers make it past peer review?

Developing a statistical threshold requires some null hypothesis, which very often takes Gaussian or linear form. The paper on which the method is based likely states their assumptions clearly, allowing their result to pass peer review.

Papers merely employing some method are rarely reviewed by the same peers as those of the methods papers, and often neither aware of or state the assumptions of the method.

This is how this happens, at least in neuroscience. You can look down on it and judge, but neuroscience has not really found its Kuhnian paradigm yet, so be nice.