fMRI software bugs could upend years of research • Hacker News Board

Re: fMRI software bugs could upend years of research

#161

Post by gpderetta » Tue, Jul 05, 2016, 1:38 PM UTC

Retra wrote: post #119

clord wrote: post #112

Earlier quoted context omitted.

Are you suggesting there is no room for quantum effects in the brain? Honest question because I thought determinism is out of fashion now-a-days for that reason.

Determinism and non-determinism are both just models that can be converted into each other fairly easily. Either way, that has nothing to do with free will, because getting random outcomes is not 'will' at all, let alone free will -- as ill-defined as that concept is.

I subscribe 100% to the 'free-will as an emotion' description you gave else thread, but to play the devil advocate, a non deterministic acausal process could be the gateway for for a metaphysical soul to affect the physical mind. How this can be done without violating the expected probability density function of the non deterministic process I leave it for others to explain.

Re: fMRI software bugs could upend years of research

#162

Post by omginternets » Tue, Jul 05, 2016, 1:39 PM UTC

jamesrom wrote: post #105

I thought that fMRI hasn't been taken seriously since they scanned a dead fish and found it was thinking.

>fMRI hasn't been taken seriously

I don't know where you got that (very strange) impression.

The dead fish paper showed that multiple comparisons yield false positives, which is a problem that is much broader than fMRI methodology. With proper method, fMRI is a very reliable and insightful tool. It's just very difficult to do properly.

Re: fMRI software bugs could upend years of research

#163

Post by omginternets » Tue, Jul 05, 2016, 1:42 PM UTC

Alex3917 wrote: post #50

mattkrause wrote: post #36

Earlier quoted context omitted.

The salmon thing was a statistical problem, not a technical one. fMRI data generate an incredible number of data points: imagine a movie, but in three dimensions, so you get a sequence of x/y/z-volumes. A typical scan has ~128 to 256 voxels in each spatial dimension, for ~1 million voxels per volume. This means that if your analysis contains voxel-by-voxel tests, you're going to be running a huge number of them. Even…

> The salmon thing was a statistical problem, not a technical one. How is that relevant to the question of whether or not most fMRI research is wrong?

The significance of the dead salmon paper is waaaay overblown.

That paper was a warning to researchers that family-wise error corrections are, indeed, important. A small (but non-negligible) portion of researchers were not doing these corrections simply because "the results looked nicer".

The dead fish guys wanted to demonstrate the extent to which this could produce false positives. The (rightly) make no comment on the soundness of fMRI as a tool for cognitive science.

Re: fMRI software bugs could upend years of research

#164

Post by mattkrause » Tue, Jul 05, 2016, 1:56 PM UTC

nonbel wrote: post #141

mattkrause wrote: post #132

Earlier quoted context omitted.

> They say the null is known to be false. Can you quote where they say that, because the article says: "Because two groups of subjects are randomly drawn from a large group of healthy controls, the null hypothesis of no group difference in brain activation should be true. Moreover, because the resting-state fMRI data should contain no consistent shifts in blood oxygen level-dependent (BOLD) activity, for a single gro…

The key point is that I am saying the second quote does say the null is false. This is where we differ. The actual null hypothesis they are testing is different than the one they actually care and talk about. It includes additional auxilliary assumptions that they say are wrong. What gets plugged in as the null hypothesis is not a problem in the realm of statistics. That is a choice of the researcher. The stats algor…

That sentence is closer to a link than actual content.

If you look at paper (14), it has the following graph: http://imgur.com/a/oxSqp It shows that the null model is perfectly appropriate for individual voxels under the event-related paradigms. The FWER there is indistinguishable from 5%, as it should be if the null were true.

It also shows that FWER for the null model is inflated by ~6x for blocked paradigms. As they say in the new paper, it is unclear if that would re-occur in a multi-subject experiment (that was apparently a criticism of the first paper).

Based on this, the absolute worst you can say is "the data on blocked paradigms is not very interesting", but that leaves their event-related data completely intact.

As for the stats thing, the actual formal itself test is obvious fine--it's a t-test! They're saying that the pipeline that runs between the raw DICOM output of the scanner and the t-test, which is an equally important part of the testing procedure, is flawed.

I'm happy to keep discussing this with you, but we seem to be going in circles. If "additional auxiliary assumptions" are the problem here, please describe them. If you think the 2012 NeuroImage paper proves that their null model is wrong in all cases, I'd love to read a few sentences describing how that follows.

Re: fMRI software bugs could upend years of research

#165

Post by Retric » Tue, Jul 05, 2016, 2:04 PM UTC

bad_user wrote: post #156

Retric wrote: post #111

Earlier quoted context omitted.

The logic is odd but strait forward. Basically, free will requires people to make choices independent of the past state of the universe. Aka if you replayed to universe up to this point several times they would not always respond in the same way. If that's the case people should be able to make unpredictable choices by definition. However, when asked to be unpredictable they fail. So, at best people have a limited fo…

Well OK, but if you want nondeterministic behavior, you can find a source of entropy and make decisions based on a random number (e.g. flip a coin). You chose to do it and it invalidates the hypothesis, because if you replay the universe up to this point, you won't behave in the same way and the choices you just made by flipping a coin are independent of the past state of the universe. So what am I missing, as I feel…

If you decide to follow an entropy source and I see the same output as you I can predict your behavior. Free will must be unpredictable as it does not depend on the past state of the universe. (Unless you mean an entropy source outside of the 'universe' which is a rather strange a circular requirement for free will.)

Now, you can argue for a lesser form of free will which is influenced, but not dependent on the state of the universe. However, that's progress even if somewhat obvious.

Re: fMRI software bugs could upend years of research

#166

Post by ihnorton » Tue, Jul 05, 2016, 2:10 PM UTC

bpchaps wrote: post #103

jballanc wrote: post #78

The real takeaway lesson from this research should be the vital importance of Open Data to the modern scientific enterprise: > "lamentable archiving and data-sharing practices" that prevent most of the discipline's body of work being re-analysed. Keeping data private before publication is (at this point in time) understandable. Once results are published, however, there is no excuse for not depositing the raw data in…

Yeah, it's pretty frustrating. I tried getting my own data from an fMRI study, but was told that the signed paperwork specifically disallowed this sort of thing. Not even my doctor could request it. The only option I have is to completely withdrawal myself from the study, but that would be a pretty dick move. The other option is to wait until next year when the study's wrapped and then request it. Though, I'm not eve…

FWIW, typical raw fMRI data is mostly useless unless you know the design parameters of the study (stimulus timing, imaging onsets, etc.). Though there are some interesting data-driven analyses techniques, especially for resting-state data.

Many research studies in the U.S. are required to do "structural" scans (high-resolution T1 or T2) and send them to a safety read by a radiologist, for liability reasons. At very least, this scan should be available directly from the hospital imaging department. If you are lucky all the data will have been sent through the hospital PACS and the imaging department will indiscriminately dump everything. At a research-only center it might be more complicated because such images are likely sent off-site for the safety read.

Re: fMRI software bugs could upend years of research

#167

Post by mcguire » Tue, Jul 05, 2016, 2:11 PM UTC

randcraw wrote: post #44

As someone who works in the biomedical imaging business and is also a fan of philosophy, I think this news will matter more to folks in the latter camp. For a couple years now philosophers have insisted that fMRI images prove there is no such thing as free will. Today's revelation should put an end to that whole line of reasoning (and the absurd amount of fatalism that it engendered). (The back story: Apparently fMRI…

"Apparently fMRI showed motor signals arising before the cognitive / conscious signals that should have created them, assuming we humans have free will."

Has anyone presented any theory where consciousness precedes neural activity that doesn't invoke hard-core dualism and an immaterial soul?

Re: fMRI software bugs could upend years of research

#168

Post by nonbel » Tue, Jul 05, 2016, 3:10 PM UTC

mattkrause wrote: post #164

nonbel wrote: post #141

Earlier quoted context omitted.

The key point is that I am saying the second quote does say the null is false. This is where we differ. The actual null hypothesis they are testing is different than the one they actually care and talk about. It includes additional auxilliary assumptions that they say are wrong. What gets plugged in as the null hypothesis is not a problem in the realm of statistics. That is a choice of the researcher. The stats algor…

That sentence is closer to a link than actual content. If you look at paper (14), it has the following graph: http://imgur.com/a/oxSqp It shows that the null model is perfectly appropriate for individual voxels under the event-related paradigms. The FWER there is indistinguishable from 5%, as it should be if the null were true. It also shows that FWER for the null model is inflated by ~6x for blocked paradigms. As th…

>"They're saying that the pipeline that runs between the raw DICOM output of the scanner and the t-test, which is an equally important part of the testing procedure, is flawed."

Yes, in other words the null hypothesis is being rendered false, by some aspect of the analysis pipeline. I am saying that is the correct description of the problem.

Describing the problem as excess false positives is confused, because these are true positives.

They appear to have successfully identified a problem, but then described and analyzed it incorrectly.

>"The FWER there is indistinguishable from 5%, as it should be if the null were true."

If the null is true, the pvalues should be samples from a uniform distribution. Another thing is they should have shown histograms of these. The 5% below 0.05 is not the whole story. There could be other ways the deviation manifests and/or they just chose sample sizes to be powered to get that result.

"I'd love to read a few sentences describing how that follows."

We have already read the exact sentences that mean the null model is false, multiple times. You have agreed that something was wrong, but want to call it something other than the null model.

Re: fMRI software bugs could upend years of research

#169

Post by rqebmm » Tue, Jul 05, 2016, 3:13 PM UTC

randcraw wrote: post #44

As someone who works in the biomedical imaging business and is also a fan of philosophy, I think this news will matter more to folks in the latter camp. For a couple years now philosophers have insisted that fMRI images prove there is no such thing as free will. Today's revelation should put an end to that whole line of reasoning (and the absurd amount of fatalism that it engendered). (The back story: Apparently fMRI…

> It seems philosophers STILL haven't learned the importance of applying the scientific method before leaping to a conclusion

To be fair, if you can apply the scientific method it's not really philosophy anymore, it's science. Philosophy exists in order to attempt understanding of domains we cannot rigorously apply empirical reasoning to.

Re: fMRI software bugs could upend years of research

#170

Post by vidarh » Tue, Jul 05, 2016, 3:44 PM UTC

nine_k wrote: post #109

threatofrain wrote: post #72

Earlier quoted context omitted.

The problem with free will debates has to do with its definition. I think once people start hammering out the definition, free will either becomes a wimpy variant that people didn't want to talk about in the first place, or it becomes so ambitious that it's outside the scope of science to discuss. If you define free will as the ability to generate and choose options based on constraints, then people get bored of that…

When a future state does not depend on previous states, it's called "randomness". Not quite unattainable, but does not sound too desirable either.

Whenever people claim we have free will, the first thing I do is generally to ask them to define it, and that is usually where that conversation ends up in a quagmire.