Live data from Hacker News

fMRI software bugs could upend years of research

theregister.co.uk

101–110 of 192 posts

Re: fMRI software bugs could upend years of research

#101
post #56

As someone who works in the biomedical imaging business and is also a fan of philosophy, I think this news will matter more to folks in the latter camp. For a couple years now philosophers have insisted that fMRI images prove there is no such thing as free will. Today's revelation should put an end to that whole line of reasoning (and the absurd amount of fatalism that it engendered). (The back story: Apparently fMRI…

Free will as in "there's something that can't be explained by chemistry and electrical signals in the brain" is just an attempt of some believers to rationalize their belief in the kind of the supernatural "soul."

Exactly. The whole concept is flawed.

The brain computes an abstraction of the reality (or most likely multiple competing abstractions) and decides on actions. There's the free will.

There is some kind of idea that the consciousness "floats" on top of this abstract soup as some kind of a running program in an operative system, and is making the calls, but this is most likely not how it works.

All these abstractions of sensory input, probabilities, decisions, etc, is probably what creates the consciousnesses but I doubt that there is a meaningful boundary.

After-the-fact rationalization may be true or not - it's obvious that events can't be processed in no time at all, and that there must be some processing time for the experienced "now."

But trying to build some meaningful philosophical arguments from it is probably a red herring.

After all, when trying to explain the reasons behind a decision made without explicit reasoning and external lists of pros and cons, it's often clear that there are many "hunches" that are not easy to summarize or valuate.

Re: fMRI software bugs could upend years of research

#102
post #79

Earlier quoted context omitted.

>"Can you explain what those assumptions are then, even roughly?" I have not looked at the code myself. I only read their description of it. As I quoted earlier: "assumed activity paradigm can have a large impact on the degree of false positives" If the assumed activity paradigm affects (what they are calling) the false positive rate, it is obviously part of the model. If they are arbitrarily varying it because they…

Maybe we're agreeing? The ENTIRE POINT of the paper is that there are inappropriate assumptions baked into the defaults of FSL and SPM. (AFNI has an actual code-does-the-wrong-thing bug). These errors result in cluster-wise tests being inappropriately sized: when thresholds are set to p They tested this by dividing a single stream of "off" data into segments of "ON" and "OFF" data. They do this in four different ways…

I think we are largely in agreement on the surface. The problem is that they are attributing the "positive" results to statistical error, when it isn't due to that.

The stats are working just fine and correctly telling us the null hypothesis is false. The problem is with the crappy null hypothesis, not the stats.

Re: fMRI software bugs could upend years of research

#103

The real takeaway lesson from this research should be the vital importance of Open Data to the modern scientific enterprise: > "lamentable archiving and data-sharing practices" that prevent most of the discipline's body of work being re-analysed. Keeping data private before publication is (at this point in time) understandable. Once results are published, however, there is no excuse for not depositing the raw data in…

Yeah, it's pretty frustrating. I tried getting my own data from an fMRI study, but was told that the signed paperwork specifically disallowed this sort of thing. Not even my doctor could request it. The only option I have is to completely withdrawal myself from the study, but that would be a pretty dick move. The other option is to wait until next year when the study's wrapped and then request it. Though, I'm not even sure if that'll get me the actual data..

Re: fMRI software bugs could upend years of research

#104

Earlier quoted context omitted.

> The salmon thing was a statistical problem, not a technical one. How is that relevant to the question of whether or not most fMRI research is wrong?

1. It's trivially checkable for any existing individual paper. You skim the methods or search for "multiple comparisons" or "false discovery rate" or something like that. 2. For papers where this wasn't done, the already-collected data can be reanalyzed. In fact, you can often get correct it without access to the raw data (at least approximately). 3. It means that future papers (where future is somewhere after 2008-9…

If you believe that the majority of papers in the field are not incorrect, what are the assumptions behind your estimation of the numbers?

Re: fMRI software bugs could upend years of research

#108
post #99

Earlier quoted context omitted.

I'm aware that there are assumptions implicit in what the null hypothesis is. You are the one who keeps saying the authors don't even realize what those assumptions are, but you haven't pointed out anything besides what the authors said. What are the other faulty assumptions you've identified that the authors are missing? I guess you're referring to some sort of issue with power that you mentioned in your previous co…

>"You are the one who keeps saying the authors don't even realize what those assumptions are, but you haven't pointed out anything besides what the authors said." I am saying they are confused because they say "the null hypothesis should be true" under their conditions, when they know for certain that it is false! Therefore these "false positives" are not false at all. They are totally legit "true positives". These a…

Okay, why do you think the null is crappy?

In the absence of any information whatsoever, the idea that an analysis pipeline produces false positives at or below its nominal rate seems pretty reasonable.

But, they have some prior information.

Let's look at the E1 paradigm (2 sec on, 6 sec off). In the NeuroImage Paper (Figure 1A, 2A), the FWER on voxel tests is statistically indistinguishable from 5%. In other words, it's appropriately sized. They replicate this result in the rightmost panel of the PNAS paper, where it's also within the 95% CI around 5%.

Now, for the cluster inference, look at the left-most panel of PNAS Figure 1A. The E1 paradigm is the green bar. Using the defaults for FSL (left panel) and SPM (middle panel), the FWER is about 30% and 25%, respectively. That is not good.

I agree that the Block designs look awful in the NeuroImage paper, which makes it hard to say whether this phenomena makes it worse in the PNAS data. It's unfortunate that these numbers are going to be in the press release (70% is much sexier than 30%), but 6x the nominal rate is still bad.

Re: fMRI software bugs could upend years of research

#109

Earlier quoted context omitted.

Why would this put an end to that line of reasoning? I'd expect it to flare up the debate, not end the debate. They didn't disprove behavior being computed, they demonstrated that a class of data supporting it was useless. The natural reaction to this isn't "okay we give up", it's "better go get some good data". (Also, I don't like mixing the question "Is our behavior computed?" with the question "Does computed behav…

The problem with free will debates has to do with its definition. I think once people start hammering out the definition, free will either becomes a wimpy variant that people didn't want to talk about in the first place, or it becomes so ambitious that it's outside the scope of science to discuss. If you define free will as the ability to generate and choose options based on constraints, then people get bored of that…

When a future state does not depend on previous states, it's called "randomness". Not quite unattainable, but does not sound too desirable either.

Re: fMRI software bugs could upend years of research

#110

Earlier quoted context omitted.

Why would this put an end to that line of reasoning? I'd expect it to flare up the debate, not end the debate. They didn't disprove behavior being computed, they demonstrated that a class of data supporting it was useless. The natural reaction to this isn't "okay we give up", it's "better go get some good data". (Also, I don't like mixing the question "Is our behavior computed?" with the question "Does computed behav…

The problem with free will debates has to do with its definition. I think once people start hammering out the definition, free will either becomes a wimpy variant that people didn't want to talk about in the first place, or it becomes so ambitious that it's outside the scope of science to discuss. If you define free will as the ability to generate and choose options based on constraints, then people get bored of that…

Conway and Kitchen have defined a theory of free will.

https://arxiv.org/pdf/0807.3286

Post reply on HN