Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

fMRI software bugs could upend years of research

theregister.co.uk

Post reply on HN
Print view View original

61–70 of 192 posts

Re: fMRI software bugs could upend years of research

#61

Post by Strilanc »

randcraw wrote: post #44

As someone who works in the biomedical imaging business and is also a fan of philosophy, I think this news will matter more to folks in the latter camp. For a couple years now philosophers have insisted that fMRI images prove there is no such thing as free will. Today's revelation should put an end to that whole line of reasoning (and the absurd amount of fatalism that it engendered). (The back story: Apparently fMRI…

Why would this put an end to that line of reasoning? I'd expect it to flare up the debate, not end the debate. They didn't disprove behavior being computed, they demonstrated that a class of data supporting it was useless. The natural reaction to this isn't "okay we give up", it's "better go get some good data".

(Also, I don't like mixing the question "Is our behavior computed?" with the question "Does computed behavior imply no free will?".)

Re: fMRI software bugs could upend years of research

#63

Post by threatofrain »

randcraw wrote: post #44

As someone who works in the biomedical imaging business and is also a fan of philosophy, I think this news will matter more to folks in the latter camp. For a couple years now philosophers have insisted that fMRI images prove there is no such thing as free will. Today's revelation should put an end to that whole line of reasoning (and the absurd amount of fatalism that it engendered). (The back story: Apparently fMRI…

There's free will as in a decision-making system that generates different options and selects one based on constraints, which is something a computer can do, and then there's free will as in the ability to defy the mechanics of one's biophysics, which is just an indirect way of saying metaphysical soul.

I don't think empiricists can even discuss the kind of free will which defies causal explanation, the kind of mind which exists outside of the brain.

If you reject that kind of metaphysical soul from the discussion, what remains is whatever soul can be housed in a cage of biophysics. And whether you believe there is fundamental randomness in the universe, or whether given perfect information the universe becomes predictable, both perspectives are equally hostile to the kind of free will people dream about.

People who talk about free will want to escape biophysics, and the only way is to talk about the mind outside the brain, or the ghost outside the machine.

Re: fMRI software bugs could upend years of research

#64

Post by mattkrause »

nonbel wrote: post #59
sverige wrote: post #43

Earlier quoted context omitted.

And it's a little disheartening when it takes this long to find such a significant bug. Lots of work gone to waste. I wonder if the data can be rerun? Sounds like lots of the raw data wasn't archived correctly or at all.

The real "bug" here isn't even really statistical. It is the usual logical issue that has been well known for years (yet the problem has only grown and spread): "In most psychological research, improved power of a statistical design leads to a prior probability approaching 1/2 of finding a significant difference in the theoretically predicted direction. Hence the corroboration yielded by “success” is very weak, and b…

Are you sure about that? As I understand it, they are testing a really-really-null hypothesis.

Specifically, they started with some resting state data (a condition where the subjects just lie there, attempting to remain alive). They then took this data and imposed fake task structures over it, as if the subject were doing two different things (e.g., condition #1 starts at t=0, t=1 min, t=3 min, t=6 min; condition #2 starts at t=2, t=4, t=5 min). Once this is done, the task+data is submitted to their analysis pipeline.

The null hypothesis (fake condition #1 == fake condition #2) has to be true, by construction. This assumes that the original data doesn't have any structure, which may or may not be true (they discuss it a bit towards the end of the paper).

Re: fMRI software bugs could upend years of research

#65

Post by gmac »

nitrogen wrote: post #52
randcraw wrote: post #44

As someone who works in the biomedical imaging business and is also a fan of philosophy, I think this news will matter more to folks in the latter camp. For a couple years now philosophers have insisted that fMRI images prove there is no such thing as free will. Today's revelation should put an end to that whole line of reasoning (and the absurd amount of fatalism that it engendered). (The back story: Apparently fMRI…

There are loads of other reasons not to accept dualistic free will.

Yes: see e.g. https://www.rep.routledge.com/articles/free-will/v-1

Re: fMRI software bugs could upend years of research

#66

Post by dewarrn1 »

Trombone12 wrote: post #62

Well, I think the problems with interpreting fMRI scans have been at least vaguely known since that time a dead salmon activated its neurons when asked to judge the of a person from a photo, this was in 2009. http://www.wired.com/2009/09/fmrisalmon/

The dead salmon article is a bit of a red herring here. It's a clear demonstration that a shoddy statistical approach can undermine fMRI findings. Critically, the implications of the current paper extend to even research that has been rigorously analyzed using field-standard software. Statistical issues are at the heart of both papers, but the newer paper identifies problems that are subtle and ubiquitous.

Re: fMRI software bugs could upend years of research

#67

Post by nonbel »

mattkrause wrote: post #64
nonbel wrote: post #59

Earlier quoted context omitted.

The real "bug" here isn't even really statistical. It is the usual logical issue that has been well known for years (yet the problem has only grown and spread): "In most psychological research, improved power of a statistical design leads to a prior probability approaching 1/2 of finding a significant difference in the theoretically predicted direction. Hence the corroboration yielded by “success” is very weak, and b…

Are you sure about that? As I understand it, they are testing a really-really-null hypothesis. Specifically, they started with some resting state data (a condition where the subjects just lie there, attempting to remain alive). They then took this data and imposed fake task structures over it, as if the subject were doing two different things (e.g., condition #1 starts at t=0, t=1 min, t=3 min, t=6 min; condition #2…

>"The null hypothesis (fake condition #1 == fake condition #2) has to be true, by construction."

The null hypothesis is not only that condition 1 == condition 2, it involves other assumptions being made. From their own description, they knew it was false before even doing this analysis and used this knowledge to design the study. Apparently there is some incorrect assumption about autocorrelation being made.

It seems the authors did not really grasp what hypothesis they were testing, leading to their confused (but still productive) description of the problem. I go into more in this post: https://news.ycombinator.com/item?id=12032772

Re: fMRI software bugs could upend years of research

#68

Post by mattkrause »

nonbel wrote: post #41

I would send it back and ask for a detailed description of the null hypothesis they are testing, because they are not clear on this point at all: >"All of the analyses to this point have been based on resting-state fMRI data, where the null hypothesis should be true." They are not careful to explicitly define this null hypothesis anywhere, but earlier in the paper they describe some issues with the model used: >"Rest…

Nononono. The null hypothesis here is definitely true. They take resting state data ("lie here while we scan your brain") and pretend that it was acquired during an experiment, using either block or event-related designs. Next, they look for differences between the fake blocks (or events).

The null hypothesis is that neural activity is the same across blocks. It must be true because the block boundaries are totally arbitrary. In fact, they were defined after the scanning, so they could not possibly influence neural activity (assuming the resting state data is stationary, which may or may not be true; they discuss that).

The NeuroImage paper you linked to isn't disagreeing with this. They're saying that if you use the above procedure to make your null model, it doesn't work and it doesn't work because of the reason outlined in this paper.

Re: fMRI software bugs could upend years of research

#69

Post by nickledave »

nonbel wrote: post #41

I would send it back and ask for a detailed description of the null hypothesis they are testing, because they are not clear on this point at all: >"All of the analyses to this point have been based on resting-state fMRI data, where the null hypothesis should be true." They are not careful to explicitly define this null hypothesis anywhere, but earlier in the paper they describe some issues with the model used: >"Rest…

See mattkrause's comment below. I think you might not be understanding what they're testing. They're taking only resting state data, randomly sorting some of it into "pretend active state data" and then asking whether they get any statistically significant difference between these two groups when they shouldn't. But they do. That means the tests they used, the same tests many authors use, are giving false positives. The null hypothesis is "there will be no difference between testing state data and whatever data we get when we ask the subject to do some activity". They can "reject" that hypothesis using only randomly shuffled resting state data, so there's something wrong with the stats packages themselves

Re: fMRI software bugs could upend years of research

#70

Post by nickledave »

nickledave wrote: post #69
nonbel wrote: post #41

I would send it back and ask for a detailed description of the null hypothesis they are testing, because they are not clear on this point at all: >"All of the analyses to this point have been based on resting-state fMRI data, where the null hypothesis should be true." They are not careful to explicitly define this null hypothesis anywhere, but earlier in the paper they describe some issues with the model used: >"Rest…

See mattkrause's comment below. I think you might not be understanding what they're testing. They're taking only resting state data, randomly sorting some of it into "pretend active state data" and then asking whether they get any statistically significant difference between these two groups when they shouldn't . But they do . That means the tests they used, the same tests many authors use, are giving false positives…

@mattkrause: you beat me to it
Post reply on HN