Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

fMRI software bugs could upend years of research

theregister.co.uk

Post reply on HN
Print view View original

131–140 of 192 posts

Re: fMRI software bugs could upend years of research

#131

Post by e12e »

nonbel wrote: post #128
mattkrause wrote: post #108

Earlier quoted context omitted.

Okay, why do you think the null is crappy? In the absence of any information whatsoever, the idea that an analysis pipeline produces false positives at or below its nominal rate seems pretty reasonable. But, they have some prior information. Let's look at the E1 paradigm (2 sec on, 6 sec off). In the NeuroImage Paper (Figure 1A, 2A), the FWER on voxel tests is statistically indistinguishable from 5%. In other words,…

>"Okay, why do you think the null is crappy?" Because their goal is to determine if some sort of treatment has an effect. If the null is false for other reasons, then statistical significance can't be used to support the existence of a treatment effect. So these would be pointless, pedantic calculations.

@nonbel: Are you saying that an fMRI, when taken of a subject at rest, twice - then what the data here show - is that this is likely to be interpreted as a subject in two different states? And the researchers are ignoring the fact, rather insisting that we should be able to tell that these two states are the same (and then, perhaps tell them from other states)?

I see a few ways this could come about: perhaps the way we record and model activity doesn't conform to the distribution we assume (I'm not sure if they assume a normal distribution here - or if that even makes sense given the nature of the data) -- or perhaps the issue is with taking 3d/4d data and "turning it into" an easy-to-model statistical model (like the normal distribution)?

At any rate, it does seem that they're saying we can't tell that one individual at rest, measured twice, is in the same (rest) state both times? Hence, they're null hypothesis is bunk?

Re: fMRI software bugs could upend years of research

#132

Post by mattkrause »

nonbel wrote: post #126
mattkrause wrote: post #115

Earlier quoted context omitted.

The experiment is set up so that there are no true positives. When the threshold is p Are you trying to say that it's not the t-test per se, but something before it in the pipeline? If so, sure, of course, but that's not the null. The null is that chunks randomly labeled as ON are the same as chunks randomly labeled as OFF, and since it's not physically possible for the labels to affect the signal, this null must be…

>"The experiment is set up so that there are no true positives." No, a true positive is when the null model is called false when it is false. They say the null is known to be false. The errors here are actually the false negatives, in this case that is every "non-significant" result. >"When the threshold is p Only if the null model is correct. In this case they describe problematic aspects of their null model. Ie the…

> They say the null is known to be false.

Can you quote where they say that, because the article says:

"Because two groups of subjects are randomly drawn from a large group of healthy controls, the null hypothesis of no group difference in brain activation should be true. Moreover, because the resting-state fMRI data should contain no consistent shifts in blood oxygen level-dependent (BOLD) activity, for a single group of subjects the null hypothesis of mean zero activation should also be true."

There is also this quote:

"Resting-state data should not contain systematic changes in brain activity, but our previous work (14) showed that the assumed activity paradigm can have a large impact on the degree of false positives. "

This does not mean the null is false. That papers shows that temporal autocorrelations can also introduce spurious false positives. These factors affect block, but not event-related designs. They can be avoided by filtering--and maybe by averaging over many subjects (since these oscillations aren't locked to anything). In any case, they don't appear, as you can verify on the right side of Figure 1A.

Finally, the discussion has a paragraph describing reasons why the experiment might be flawed, but most of those possible objections seem answered.

Re: fMRI software bugs could upend years of research

#133

Post by moptar »

bpchaps wrote: post #103
jballanc wrote: post #78

The real takeaway lesson from this research should be the vital importance of Open Data to the modern scientific enterprise: > "lamentable archiving and data-sharing practices" that prevent most of the discipline's body of work being re-analysed. Keeping data private before publication is (at this point in time) understandable. Once results are published, however, there is no excuse for not depositing the raw data in…

Yeah, it's pretty frustrating. I tried getting my own data from an fMRI study, but was told that the signed paperwork specifically disallowed this sort of thing. Not even my doctor could request it. The only option I have is to completely withdrawal myself from the study, but that would be a pretty dick move. The other option is to wait until next year when the study's wrapped and then request it. Though, I'm not eve…

There's a science reason for denying participants access to their own data. For example, if you suspect you might have some brain anomaly, you might enroll in the study as a healthy person hoping to use it as a free way to get a brain scan. People doing that would distort the sampling.

Re: fMRI software bugs could upend years of research

#134

Post by moptar »

greenyoda wrote: post #15
honkhonkpants wrote: post #8

Doesn't sound like a straight up bug, but rather unsound statistical methods which can happen with or without software. You get the same problem with finite element analysis software: the operator has to be aware of all the assumptions baked in, and has to ensure that the input conforms to them.

I'd say it's a bug, since the unsound statistical methods are incorporated into the three most common software packages that are specifically intended to be used for fMRI analysis. If these methods don't work well for fMRI data, fMRI software shouldn't be using them. From the paper: "Using this null data with different experimental designs, we estimate the incidence of significant results. In theory, we should find 5…

All finite element analysis software uses "bad algorithms" in the sense that they give wrong results if the user doesn't take their limitations into account. A common type of error causes them to underestimate stress at the highest stress locations - where failure is most likely! Other problems occur with structures that stiffen as they're loaded. That effect is ignored by linear analysis and engineers often don't do the more accurate non-linear analysis because it cost more time (computer resources) and money (more expensive licenses) as well as sometimes just being more difficult to set up. It's more that the technology isn't advanced enough to be foolproof than that there are bugs.

Re: fMRI software bugs could upend years of research

#135

Post by bpchaps »

dewarrn1 wrote: post #120
bpchaps wrote: post #103

Earlier quoted context omitted.

Yeah, it's pretty frustrating. I tried getting my own data from an fMRI study, but was told that the signed paperwork specifically disallowed this sort of thing. Not even my doctor could request it. The only option I have is to completely withdrawal myself from the study, but that would be a pretty dick move. The other option is to wait until next year when the study's wrapped and then request it. Though, I'm not eve…

I completely agree that this can be frustrating. For what it's worth, the data from a research study aren't necessarily going to be clinically informative to a GP or even to a radiologist. Structural MRI data could potentially be interpreted, but the sequences collected for research are still different than what you'd get if you went in for a clinical scan. And if you're interested in having the data for your own pur…

Agreed on leaking information. It make total sense why they wouldn't want the participants to get access to the data.

Though, when I found that out, I did some naive scans to see if they had any public webservers that would have the research information. I didn't see any, but there were a huge number of HIPAA violations where many, many computers with health information were exposed directly to the internet. Their security team told me that it wasn't anything to be concerned about. A HIPAA complaint didn't go anywhere, either. Already spoke with a journalist to try to get some traction on it..

Re: fMRI software bugs could upend years of research

#136

Post by mc32 »

randcraw wrote: post #44

As someone who works in the biomedical imaging business and is also a fan of philosophy, I think this news will matter more to folks in the latter camp. For a couple years now philosophers have insisted that fMRI images prove there is no such thing as free will. Today's revelation should put an end to that whole line of reasoning (and the absurd amount of fatalism that it engendered). (The back story: Apparently fMRI…

I thought someone had found that while "fMRI showed motor signals arising before the cognitive / conscious signals" that they also had found that we could choose to negate/override that signal thus allowing free will. That is free will was expressed by overriding the default.

Re: fMRI software bugs could upend years of research

#137

Post by Tomte »

hammock wrote: post #93
randcraw wrote: post #44

As someone who works in the biomedical imaging business and is also a fan of philosophy, I think this news will matter more to folks in the latter camp. For a couple years now philosophers have insisted that fMRI images prove there is no such thing as free will. Today's revelation should put an end to that whole line of reasoning (and the absurd amount of fatalism that it engendered). (The back story: Apparently fMRI…

What are the actionable insights that have come out of fMRI studies? Even when properly conducted (no false positives), the conclusions that are often drawn have always felt dubious to me. Basically you are looking for regions of the brain that light up with various stimuli. Except that's as far as it goes, we don't yet understand much beyond that. It's as if you figure out that your car is making a funny sound, and…

I was participant in a study in linguistics [1] that compared native Polish and native German speakers who were put into an fRMI and played speech sounds. Both Polish and German ones.

It clearly showed that speech sounds from your native language are processed in a different part of the brain than non-native speech sounds.

Yes, that does not explain much. But it leads towards all kinds of questions. And I found that fascinating.

[1] Silvia Lipski, Neurosci Lett. 2007 Mar 19, „A Magnetoencephalographic study on auditory processing of native and nonnative fricative contrasts in Polish and German listeners.“

Re: fMRI software bugs could upend years of research

#138

Post by maroonblazer »

threatofrain wrote: post #72
Strilanc wrote: post #61

Earlier quoted context omitted.

Why would this put an end to that line of reasoning? I'd expect it to flare up the debate, not end the debate. They didn't disprove behavior being computed, they demonstrated that a class of data supporting it was useless. The natural reaction to this isn't "okay we give up", it's "better go get some good data". (Also, I don't like mixing the question "Is our behavior computed?" with the question "Does computed behav…

The problem with free will debates has to do with its definition. I think once people start hammering out the definition, free will either becomes a wimpy variant that people didn't want to talk about in the first place, or it becomes so ambitious that it's outside the scope of science to discuss. If you define free will as the ability to generate and choose options based on constraints, then people get bored of that…

Well put. Serendipitously, Dan Dennett and Sam Harris just discussed this last week and released the recording of their conversation[0].

The conversation was interesting and disappointing.

Harris argues that the commonly-held notion of 'free will' is nonsensical. Dennett doesn't disagree but worries that people may construe this to mean that 'all bets are off' and the world will descend into chaos. Harris attempts to explain why this would not be the case; that we'd still have good reasons to imprison people who want to do harm. Dennett agrees but then restates the same worry differently. They never manage to get past this.

Still, a good listen.

[0]https://www.samharris.org/podcast/item/free-will-revisited

Re: fMRI software bugs could upend years of research

#139

Post by xenadu02 »

kpil wrote: post #101
acqq wrote: post #56

Earlier quoted context omitted.

Free will as in "there's something that can't be explained by chemistry and electrical signals in the brain" is just an attempt of some believers to rationalize their belief in the kind of the supernatural "soul."

Exactly. The whole concept is flawed. The brain computes an abstraction of the reality (or most likely multiple competing abstractions) and decides on actions. There's the free will. There is some kind of idea that the consciousness "floats" on top of this abstract soup as some kind of a running program in an operative system, and is making the calls, but this is most likely not how it works. All these abstractions o…

The idea that after the fact rationalization negates free will has never been convincing. Observing the consequences of a "choice" then actively attempting to rewire the brain to make a different one in the future is just as much free will as anything else. Free will doesn't require instantaneous control over decision making.

Re: fMRI software bugs could upend years of research

#140

Post by shiny »

maroonblazer wrote: post #138
threatofrain wrote: post #72

Earlier quoted context omitted.

The problem with free will debates has to do with its definition. I think once people start hammering out the definition, free will either becomes a wimpy variant that people didn't want to talk about in the first place, or it becomes so ambitious that it's outside the scope of science to discuss. If you define free will as the ability to generate and choose options based on constraints, then people get bored of that…

Well put. Serendipitously, Dan Dennett and Sam Harris just discussed this last week and released the recording of their conversation[0]. The conversation was interesting and disappointing. Harris argues that the commonly-held notion of 'free will' is nonsensical. Dennett doesn't disagree but worries that people may construe this to mean that 'all bets are off' and the world will descend into chaos. Harris attempts to…

Listened to that whole thing as well, and was just as frustrated. I prefer Brian Greene's 1 minute case for the absence of free will: https://www.youtube.com/watch?v=KBNzaXx6eKg
Post reply on HN