Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

fMRI software bugs could upend years of research

theregister.co.uk

Post reply on HN
Print view View original

51–60 of 192 posts

Re: fMRI software bugs could upend years of research

#51

Post by [deleted] »

leemailll wrote: post #38
Alex3917 wrote: post #21

Earlier quoted context omitted.

Here are a couple Ben Goldacre articles on neuroscience in general: https://www.theguardian.com/commentisfree/2011/sep/09/bad-sc... For fMRI research specifically, I seem to remember that almost all of the analysis software is closed source, and that comparing the readings between software versions isn't actually meaningful even though a lot of papers do so.

Actually for fMRI studies most people use matlab and share it. It is not really closed-sourced. But I guess many guys in the field won't bother too much to read the codes from a widely used and well-cited matlab script

[deleted]

Re: fMRI software bugs could upend years of research

#52

Post by nitrogen »

randcraw wrote: post #44

As someone who works in the biomedical imaging business and is also a fan of philosophy, I think this news will matter more to folks in the latter camp. For a couple years now philosophers have insisted that fMRI images prove there is no such thing as free will. Today's revelation should put an end to that whole line of reasoning (and the absurd amount of fatalism that it engendered). (The back story: Apparently fMRI…

There are loads of other reasons not to accept dualistic free will.

Re: fMRI software bugs could upend years of research

#53

Post by RGamma »

randcraw wrote: post #44

As someone who works in the biomedical imaging business and is also a fan of philosophy, I think this news will matter more to folks in the latter camp. For a couple years now philosophers have insisted that fMRI images prove there is no such thing as free will. Today's revelation should put an end to that whole line of reasoning (and the absurd amount of fatalism that it engendered). (The back story: Apparently fMRI…

To be fair, there are more arguments to be made contra free will (thought experiments like: At what point from conception to adulthood would free will develop (if at all), could free will be a merely probabilistic byproduct, etc...) and even if the details are wrong in these fMRI images, the general notion that a motor signal is generated before it appears in consciousness might still be valid.

Re: fMRI software bugs could upend years of research

#54

Post by freshhawk »

randcraw wrote: post #44

As someone who works in the biomedical imaging business and is also a fan of philosophy, I think this news will matter more to folks in the latter camp. For a couple years now philosophers have insisted that fMRI images prove there is no such thing as free will. Today's revelation should put an end to that whole line of reasoning (and the absurd amount of fatalism that it engendered). (The back story: Apparently fMRI…

Well, the non-pop philosophers will understand that as a minor hit to the model of consciousness as solely concerned with post-facto rationalization and basically removed from the real-time decision making loop. Those studies were never strong evidence anyway, the protocols were pretty weak as they required asking the subjects to self-report at what time they "decided" to act.

The real free will debate is about both the definition of free will (for the compatibilists and libertarians) and a debate about whether the evidence for materialism outweighs the subjective experience of free will.

But yes, this will hopefully stop those confused pop-philosophy stories.

Re: fMRI software bugs could upend years of research

#55

Post by mattkrause »

Alex3917 wrote: post #21
JumpCrisscross wrote: post #12

Earlier quoted context omitted.

> almost all MRI brain scan research is wrong Source for a layman?

Here are a couple Ben Goldacre articles on neuroscience in general: https://www.theguardian.com/commentisfree/2011/sep/09/bad-sc... For fMRI research specifically, I seem to remember that almost all of the analysis software is closed source, and that comparing the readings between software versions isn't actually meaningful even though a lot of papers do so.

> almost all of the analysis software is closed source

This is almost certainly not true. BrainSurfer is closed-source, but almost everything else I can think of is open-source.

* SPM is available under the GPL: http://www.fil.ion.ucl.ac.uk/spm/software/

* FSL (including the source) is available under their own noncommercial "As-Is" license. FSLView seems to have its own separate GPL license: http://fsl.fmrib.ox.ac.uk/fsldownloads/

* AFNI used to require (free) registration, but is now GPL: https://afni.nimh.nih.gov/afni/about/legal

* The BIC-MNI tools + source appear to be freely available for free under an "as-is" + attribution license. I only checked a few; please let me know if any are not and I will nag people accordingly: https://github.com/BIC-MNI

* Caret is GPL: http://brainvis.wustl.edu/wiki/index.php/Caret:About

* FreeSurfer has its own license which is very open: http://www.freesurfer.net/fswiki/FreeSurferSoftwareLicense

I suspect you're thinking of this, which reported differences between FreeSurfer versions: http://journals.plos.org/plosone/article/comment?id=info%3Ad...

That turned out to be a combination of 1) algorithmic changes 2) Different random seeds (some of these functions use Monte Carlo-like methods), and 3) 32 vs 64 bit platforms.

Re: fMRI software bugs could upend years of research

#56

Post by acqq »

randcraw wrote: post #44

As someone who works in the biomedical imaging business and is also a fan of philosophy, I think this news will matter more to folks in the latter camp. For a couple years now philosophers have insisted that fMRI images prove there is no such thing as free will. Today's revelation should put an end to that whole line of reasoning (and the absurd amount of fatalism that it engendered). (The back story: Apparently fMRI…

Free will as in "there's something that can't be explained by chemistry and electrical signals in the brain" is just an attempt of some believers to rationalize their belief in the kind of the supernatural "soul."

Re: fMRI software bugs could upend years of research

#57

Post by mattkrause »

Alex3917 wrote: post #50
mattkrause wrote: post #36

Earlier quoted context omitted.

The salmon thing was a statistical problem, not a technical one. fMRI data generate an incredible number of data points: imagine a movie, but in three dimensions, so you get a sequence of x/y/z-volumes. A typical scan has ~128 to 256 voxels in each spatial dimension, for ~1 million voxels per volume. This means that if your analysis contains voxel-by-voxel tests, you're going to be running a huge number of them. Even…

> The salmon thing was a statistical problem, not a technical one. How is that relevant to the question of whether or not most fMRI research is wrong?

1. It's trivially checkable for any existing individual paper. You skim the methods or search for "multiple comparisons" or "false discovery rate" or something like that.

2. For papers where this wasn't done, the already-collected data can be reanalyzed. In fact, you can often get correct it without access to the raw data (at least approximately).

3. It means that future papers (where future is somewhere after 2008-9 here) can be done correctly from the get-go; it's not a limitation of the technique or the signal itself.

Re: fMRI software bugs could upend years of research

#58

Post by strictfp »

iLoch wrote: post #19

" How X looks like" -- what's with this gramatical mistake? I see it everywhere. Is it a regional thing?

It's a common error for native Swedes, It's a direct translation from Swedish. Could be true for other languages as well, I believe most Germanic languages use "how".

Re: fMRI software bugs could upend years of research

#59

Post by nonbel »

sverige wrote: post #43
greenyoda wrote: post #15

Earlier quoted context omitted.

I'd say it's a bug, since the unsound statistical methods are incorporated into the three most common software packages that are specifically intended to be used for fMRI analysis. If these methods don't work well for fMRI data, fMRI software shouldn't be using them. From the paper: "Using this null data with different experimental designs, we estimate the incidence of significant results. In theory, we should find 5…

And it's a little disheartening when it takes this long to find such a significant bug. Lots of work gone to waste. I wonder if the data can be rerun? Sounds like lots of the raw data wasn't archived correctly or at all.

The real "bug" here isn't even really statistical. It is the usual logical issue that has been well known for years (yet the problem has only grown and spread):

"In most psychological research, improved power of a statistical design leads to a prior probability approaching 1/2 of finding a significant difference in the theoretically predicted direction. Hence the corroboration yielded by “success” is very weak, and becomes weaker with increased precision. “Statistical significance” plays a logical role in psychology precisely the reverse of its role in physics." http://cerco.ups-tlse.fr/pdf0609/Meehl_1967.pdf

In other words, the fMRI researchers have been testing a null hypothesis known to be false beforehand, so statistical significance tells us nothing of value, and definitely nothing about the theory of interest.

This paper is really interesting, because the authors don't seem to realize that this is what they are discovering. Getting a "statistically significant" result is the correct answer for the tests to return, these are not false positives. The problem is a poorly chosen null model.

Re: fMRI software bugs could upend years of research

#60

Post by dewarrn1 »

acveilleux wrote: post #22
sxg wrote: post #10

Earlier quoted context omitted.

The same tools (AFNI, SPM, and FSL) are used in a lot of other MR research protocols as well - not just fMRI.

They are but 3dClustSim (the culprit) is essentially used only for fMRI.

This is right and wrong: AFNI's 3dClustStim did contain a specific bug (since corrected) and is used for fMRI analysis, but the problem that the authors identified is more general than that and encompasses all of the tested packages.
Post reply on HN