Software faults raise questions about the validity of brain studies
1–10 of 23 posts
Re: Software faults raise questions about the validity of brain studies
#2[deleted]
Re: Software faults raise questions about the validity of brain studies
#3A 15 year old bug in a basic package used by fMRI software was, according to the researchers, causing numerous false positives in previously published studies. Which means some of the accepted fMRI based brain studies results are dubious.
Re: Software faults raise questions about the validity of brain studies
#4A 15 year old bug in a basic package used by fMRI software was, according to the researchers, causing numerous false positives in previously published studies. Which means some of the accepted fMRI based brain studies results are dubious.
They were dubious anyway
Re: Software faults raise questions about the validity of brain studies
#5A 15 year old bug in a basic package used by fMRI software was, according to the researchers, causing numerous false positives in previously published studies. Which means some of the accepted fMRI based brain studies results are dubious.
As I read it, the bug was a minor issue. The bigger issue is the general false positive rate, which isn't a bug exactly, but more like a statistical modeling question - with millions of vocals, what's the appropriate way to characterize an activation as spurious?
Re: Software faults raise questions about the validity of brain studies
#6A 15 year old bug in a basic package used by fMRI software was, according to the researchers, causing numerous false positives in previously published studies. Which means some of the accepted fMRI based brain studies results are dubious.
No, that was a side note about a bug that caused a ~10% increase in false positives. The real primary issue is that the clustering methods assume Gaussian autocorrelation distributions, which don't exist in reality. They estimated a 70% false-positive rate because of this
Re: Software faults raise questions about the validity of brain studies
#7There is a real problem with incentives in science strongly encouraging bad science. It is a publish or perish world, so every researcher needs to be finding some (apparantly) statistically significant result on a regular basis. Quantity is rewarded more than quality. Being super careful with the stats makes it much harder to get publishable results but doesn't significantly increase the rewards (i.e. academic credit). So the culture tends to sloppy stats making garbage or weak results look strong. And there is no academic credit for being open with data, and more of a risk that someone will discredit your results if you are open, so that is rare as well.
Re: Software faults raise questions about the validity of brain studies
#8So how many neuroscience papers can now be used as a scratch pad?
Re: Software faults raise questions about the validity of brain studies
#9I've recently made the argument that science publications should include code and data. See comment history e.g. https://news.ycombinator.com/item?id=11606278
Glad to see that concrete good is coming from such efforts:
> The researchers took advantage of a recent trend toward making data open for anyone to use or analyze. They were able to download hundreds of fMRI scans used in other studies to perform their analysis.
The more data and code that's made openly available, the stronger science will be. I hope that we can work toward a future where most if not all code and data are expected socially and by funding policies to be included in the publication process.
Re: Software faults raise questions about the validity of brain studies
#10I spent a few years in the late 90's and early 00's working on fMRI and I've been stunned by many of the claims that appear to arise from it's use. What's being measured relates to blood flow changes that correlate with an input block paradigm, typically from a visual stimulus. This isn't brain activity in the electrical sense but rather which areas of the brain appear to regulate their blood flow in relation to the stimulus. The assumption being that these areas require more 'fuel' to support their increased neural activity in response to the stimulus. Or as I like to think of it, attempting to estimate your home electricity usage by monitoring your water bill. Doing so with MRI, a technology I've spent most of my adult life analysing the data from and still can best describe what it does as 'magical', adding another not insignificant level of complexity.
I'm know there is good science going on in this field, by people that understand the limitations of the techniques and the technologies. However, I worked with psychiatrists - a clinical discipline starved of quantitative measurements until fMRI - that would happily ignore statistically significant activation in the air around the subjects head whilst laying claim as to the importance of those in the frontal cortex. Seeing the visual cortex 'light up' in response to flashing chequerboards is one thing, isolating those areas of the brain responsible for 'forgiveness' is something quite different.
Of course the software has bugs, I personally know people who wrote the package in question and they are extremely smart and also very human. I doubt I've ever published a paper using software that wasn't bug ridden. That's why open source is such an important part of the process, laying bare every last detail of what was done and not just what felt worth mentioning in the paper. The medical imaging research community is particularly good at this with most of the industry software making source code available. The problem with fMRI is not the software.