Live data from Hacker News

Software faults raise questions about the validity of brain studies

arstechnica.com

11–20 of 23 posts

Re: Software faults raise questions about the validity of brain studies

#11
Related classic: Neural correlates of interspecies perspective taking in the post-mortem atlantic salmon: an argument for proper multiple comparisons correction http://pages.vassar.edu/abigailbaird/files/2014/06/bennett_s...

http://blogs.scientificamerican.com/scicurious-brain/ignobel...

http://scan.oxfordjournals.org/content/4/4/417.full

Re: Software faults raise questions about the validity of brain studies

#12
post #9

I've recently made the argument that science publications should include code and data. See comment history e.g. https://news.ycombinator.com/item?id=11606278 Glad to see that concrete good is coming from such efforts: > The researchers took advantage of a recent trend toward making data open for anyone to use or analyze. They were able to download hundreds of fMRI scans used in other studies to perform their analysi…

Agree 100 %. The data of publicly funded studies should be available online so that people can run their own analyses on it. There could be a non-disclosure period of course to give the scientists that have done the original work an advantage when publishing their results.

I'm confident that we will see cloud-based systems for scientific data analysis in the near future though, especially for data which is uniform and for which good standards exists. For some areas like genomics we already see this, and I'm sure others will follow suit.

I also predict that being able to try out analysis algorithms on peta- or exabyte datasets will be a huge game changer for many areas of science, and will invalidate many current findings that are based on smaller datasets, while hopefully producing many new ones as well.

Re: Software faults raise questions about the validity of brain studies

#13
post #10

I spent a few years in the late 90's and early 00's working on fMRI and I've been stunned by many of the claims that appear to arise from it's use. What's being measured relates to blood flow changes that correlate with an input block paradigm, typically from a visual stimulus. This isn't brain activity in the electrical sense but rather which areas of the brain appear to regulate their blood flow in relation to the…

I think you're missing a key thing when talking about the activation estimation.

The measure is not blood flow, but the change in oxygenation of the blood, is it not? This does have issues but I think it's different from the issues you raised.

Re: Software faults raise questions about the validity of brain studies

#14
post #10

I spent a few years in the late 90's and early 00's working on fMRI and I've been stunned by many of the claims that appear to arise from it's use. What's being measured relates to blood flow changes that correlate with an input block paradigm, typically from a visual stimulus. This isn't brain activity in the electrical sense but rather which areas of the brain appear to regulate their blood flow in relation to the…

I guess that if we're measuring bloodflow instead of neuron activity, then perhaps our knowledge of where in the brain something is happening might be off, but the results are not per se invalid. Things might only get tricky if more than one vessel feeds the same neuron.

Re: Software faults raise questions about the validity of brain studies

#15
post #13
post #10

I spent a few years in the late 90's and early 00's working on fMRI and I've been stunned by many of the claims that appear to arise from it's use. What's being measured relates to blood flow changes that correlate with an input block paradigm, typically from a visual stimulus. This isn't brain activity in the electrical sense but rather which areas of the brain appear to regulate their blood flow in relation to the…

I think you're missing a key thing when talking about the activation estimation. The measure is not blood flow, but the change in oxygenation of the blood, is it not? This does have issues but I think it's different from the issues you raised.

You are right when you say the signal is BOLD and not flow directly. Flow regulation is a key component though. However, I don't think this negates my point. Oxygenation is being equated with brain activity. The approach is sensitive to BOLD signal changes that correlate with the input. I'm not a neuroscientist but I know many that would argue the brain is more complex than this and response are more complicated (what about areas of the brain that are on through out the experiment?)

Re: Software faults raise questions about the validity of brain studies

#16
Despite the title, software was not at fault here. Rather, the paper found a higher than expected false positive rate due to a choice in statistical modeling. The conclusion was that nonparametric tests can avoid the issue.

So, none of the results are truly invalid in the way of a hypothetical software bug, since we know what modeling assumptions were used in the previous studies. Personally, I don't see a major problem, since most fMRI papers are exploratory, and we should be reproducing the major findings anyway. We should certainly start using nonparametric tests from here on out though!

Re: Software faults raise questions about the validity of brain studies

#17
post #9

I've recently made the argument that science publications should include code and data. See comment history e.g. https://news.ycombinator.com/item?id=11606278 Glad to see that concrete good is coming from such efforts: > The researchers took advantage of a recent trend toward making data open for anyone to use or analyze. They were able to download hundreds of fMRI scans used in other studies to perform their analysi…

I think scientists already support that for the most part, but there are lots of barriers, e.g. cost of maintaining the data, privacy concerns, intellectual property, competition, etc.

Re: Software faults raise questions about the validity of brain studies

#19
post #14
post #10

I spent a few years in the late 90's and early 00's working on fMRI and I've been stunned by many of the claims that appear to arise from it's use. What's being measured relates to blood flow changes that correlate with an input block paradigm, typically from a visual stimulus. This isn't brain activity in the electrical sense but rather which areas of the brain appear to regulate their blood flow in relation to the…

I guess that if we're measuring bloodflow instead of neuron activity, then perhaps our knowledge of where in the brain something is happening might be off, but the results are not per se invalid. Things might only get tricky if more than one vessel feeds the same neuron.

Each blood vessel feeds hundreds of thousands of neurons.

Re: Software faults raise questions about the validity of brain studies

#20
There really ought to be some sort of PE-equivalent certification for mission critical software, where they're required to sign off on the code before it can be legally distributed. It seems like the majority of today's catastrophic failures are due to faulty software and currently there's no accountability when it happens. If engineers were routinely crashing aircraft and breaking MRI's they would be going to prison because of the accountability procedures that are in place.
Post reply on HN