Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

fMRI software bugs could upend years of research

theregister.co.uk

Post reply on HN
Print view View original

181–190 of 192 posts

Re: fMRI software bugs could upend years of research

#181

Post by PascLeRasc »

randcraw wrote: post #44

As someone who works in the biomedical imaging business and is also a fan of philosophy, I think this news will matter more to folks in the latter camp. For a couple years now philosophers have insisted that fMRI images prove there is no such thing as free will. Today's revelation should put an end to that whole line of reasoning (and the absurd amount of fatalism that it engendered). (The back story: Apparently fMRI…

This paper (http://www.ncbi.nlm.nih.gov/pubmed/19423830) tried to prove exactly what you talk about and called it free will, but they used the SPM software that was invalidated.

Re: fMRI software bugs could upend years of research

#182

Post by pfooti »

nkurz wrote: post #114
pfooti wrote: post #84

The dead salmon study seems relevant here in discussion of how fMRI is used, especially the theory -ladenness of observations. http://blogs.scientificamerican.com/scicurious-brain/ignobel...

Maybe I missed the link, but the full text of the readable, relevant, and enjoyable article that blog post discusses is here: The principled control of false positives in neuroimaging Bennett, Wolford, Miller 2009 http://scan.oxfordjournals.org/content/4/4/417.full

Thanks. I'm on a mobile in a foreign land, so had issues tracking down a useful link.

Re: fMRI software bugs could upend years of research

#183

Post by mattkrause »

clord wrote: post #112
acqq wrote: post #56

Earlier quoted context omitted.

Free will as in "there's something that can't be explained by chemistry and electrical signals in the brain" is just an attempt of some believers to rationalize their belief in the kind of the supernatural "soul."

Are you suggesting there is no room for quantum effects in the brain? Honest question because I thought determinism is out of fashion now-a-days for that reason.

It's hard to prove a negative but....

No one in "mainstream" neuroscience or cell biology takes these proposals seriously. Hameroff and Penrose have proposed some possible mechanisms (e.g., dendritic lamellar bodies), but none of these have panned out experimentally; for example, the DLBs are in the wrong place. They keep revising the theory around these data, but I think a lot of experimentalists have lost interest. I gather there are other more philosophical/first principles arguments against this too.

It would be neat if it were true though....

Re: fMRI software bugs could upend years of research

#184

Post by mattkrause »

mattkrause wrote: post #55
Alex3917 wrote: post #21

Earlier quoted context omitted.

Here are a couple Ben Goldacre articles on neuroscience in general: https://www.theguardian.com/commentisfree/2011/sep/09/bad-sc... For fMRI research specifically, I seem to remember that almost all of the analysis software is closed source, and that comparing the readings between software versions isn't actually meaningful even though a lot of papers do so.

> almost all of the analysis software is closed source This is almost certainly not true. BrainSurfer is closed-source, but almost everything else I can think of is open-source. * SPM is available under the GPL: http://www.fil.ion.ucl.ac.uk/spm/software/ * FSL (including the source) is available under their own noncommercial "As-Is" license. FSLView seems to have its own separate GPL license: http://fsl.fmrib.ox.ac.u…

Err...I was actually thinking of Brain Voyager, which is closed source and incredibly expensive. However, luckily for me, there is also a closed source package called Brain Surfer(!)

Re: fMRI software bugs could upend years of research

#185

Post by tdaltonc »

maroonblazer wrote: post #138
threatofrain wrote: post #72

Earlier quoted context omitted.

The problem with free will debates has to do with its definition. I think once people start hammering out the definition, free will either becomes a wimpy variant that people didn't want to talk about in the first place, or it becomes so ambitious that it's outside the scope of science to discuss. If you define free will as the ability to generate and choose options based on constraints, then people get bored of that…

Well put. Serendipitously, Dan Dennett and Sam Harris just discussed this last week and released the recording of their conversation[0]. The conversation was interesting and disappointing. Harris argues that the commonly-held notion of 'free will' is nonsensical. Dennett doesn't disagree but worries that people may construe this to mean that 'all bets are off' and the world will descend into chaos. Harris attempts to…

Every time I see Dennett deliver his, "free will exists, but it's not what you think it is" line. I imagine him in a Santa suit, with his grandchildren, when they realize that that is grandpa Dan in there. He must certainly say, "no no children, Santa exists, he just isn't who you thought he was."

Re: fMRI software bugs could upend years of research

#186

Post by ihnorton »

nonbel wrote: post #179
ihnorton wrote: post #177

Earlier quoted context omitted.

> Describing the problem as excess false positives is confused, because these are true positives. This is deeply wrong. A "true positive" in this context would mean that the resting brain activity of multiple subjects is actually correlated with arbitrary length, randomized, moving test windows. Again, the data is untreated; they are imposing arbitrary test windows and looking for increases in brain activity (increas…

>"This is deeply wrong. A "true positive" in this context would mean that the resting brain activity of multiple subjects is actually correlated with arbitrary length, randomized, moving test windows." This is the same misconception I have been trying to dispel. The null hypothesis is not the inverse of that "layman's" statement, it is a specific set of predicted results calculated in a very specific way. It is a mat…

> This is the same misconception I have been trying to dispel. The null hypothesis is not the inverse of that "layman's" statement, it is a specific set of predicted results calculated in a very specific way.

The entire point of this discussion is that the software is not calculating the null hypothesis that users expect. That the current model it does in fact calculate is internally-consistent is tautological and irrelevant (though actual bugs were found in at least one package).

As you yourself said: "I didn't read the code, or even the paper very closely." (https://news.ycombinator.com/item?id=12037207) Perhaps you should do?

Re: fMRI software bugs could upend years of research

#187

Post by nonbel »

ihnorton wrote: post #186
nonbel wrote: post #179

Earlier quoted context omitted.

>"This is deeply wrong. A "true positive" in this context would mean that the resting brain activity of multiple subjects is actually correlated with arbitrary length, randomized, moving test windows." This is the same misconception I have been trying to dispel. The null hypothesis is not the inverse of that "layman's" statement, it is a specific set of predicted results calculated in a very specific way. It is a mat…

> This is the same misconception I have been trying to dispel. The null hypothesis is not the inverse of that "layman's" statement, it is a specific set of predicted results calculated in a very specific way. The entire point of this discussion is that the software is not calculating the null hypothesis that users expect. That the current model it does in fact calculate is internally-consistent is tautological and ir…

There really is no reason for me to read the paper closely. They say that the null hypothesis is wrong, and they know exactly why. Then, like multiple people responding to me here, they also want to say somehow the null hypothesis is true.

Everyone who has questioned me also does admit there is something wrong with that hypothesis they tested. You do it too: "the software is not calculating the null hypothesis that users expect", but then you also want to say the null hypothesis they tested is true! Just bizarre, what underlying confusion is making people repeat something clearly incorrect? The null hypothesis cannot be both true and false at the same time.

There is a big difference between a "positive" result that is a "false positive" and a "positive" result due to a false null hypothesis. This is a clear cut case of the second.

Re: fMRI software bugs could upend years of research

#188

Post by mattkrause »

nonbel wrote: post #168
mattkrause wrote: post #164

Earlier quoted context omitted.

That sentence is closer to a link than actual content. If you look at paper (14), it has the following graph: http://imgur.com/a/oxSqp It shows that the null model is perfectly appropriate for individual voxels under the event-related paradigms. The FWER there is indistinguishable from 5%, as it should be if the null were true. It also shows that FWER for the null model is inflated by ~6x for blocked paradigms. As th…

>"They're saying that the pipeline that runs between the raw DICOM output of the scanner and the t-test, which is an equally important part of the testing procedure, is flawed." Yes, in other words the null hypothesis is being rendered false, by some aspect of the analysis pipeline. I am saying that is the correct description of the problem. Describing the problem as excess false positives is confused, because these…

> You have agreed that something was wrong, but want to call it something other than the null model.

That's not really what I've said.

In the previous NeuroImage paper, they generated null data by "imposing" block and event-related designs over resting state data. When they did a single subject analysis of that data with SPM, they found that the block designs had excess "positive[1]" results. However, analyzing data with an event-related design had the expected proportion of positive events.

Based on this, the event-related null model looks fine. The block designs may also be fine when data from enough data from multiple subjects is analyzed together. This makes sense because the problem was related to low-frequency oscillations that aren't synchronized across subjects.

However, you don't even have to assume this. The right-most panel of Figure 1A (and S5, S6, and S11) of the PNAS paper repeats this analysis and the voxel-by-voxel tests are appropriately sized, p> If the null is true, the pvalues should be samples from a uniform distribution. Another thing is they should have shown histograms of these. The 5% below 0.05 is not the whole story. There could be other ways the deviation manifests and/or they just chose sample sizes to be powered to get that result.

I agree that a p-value histogram would be nice, but I don't think it's essential. However, there's no way the sample sizes were cherry-picked, as you suggested: they get essentially the same result with 20, 40, and 499 subjects

------------

[1] As in the paper, I think of these as false positives. We know that there's no way the designs actually influenced neural activity. They are totally arbitrary and were assigned after the data was collected.

You seem to want to call these true positives instead. There are two reasons you might want to do this, but they both strike me as a little off. I suppose these are "true positives" from the perspective of the final t-test, but only because an earlier part of the analysis failed to remove something it should have. It seems like weird to draw a line in the middle of the analysis like that.

Alternately, you might call them true positives because you're not convinced that the slice-and-dice procedure generates two sets of indistinguishable data. If so, you should say that and say why you think that. The one sentence you quoted does not count, for the reasons I outlined above.

Re: fMRI software bugs could upend years of research

#189

Post by mattkrause »

lostlogin wrote: post #149
brandonl222 wrote: post #146

Earlier quoted context omitted.

It's a rather sad reflection of society, no telling how may breakthroughs we're missing out on due to the lack of cooperation in science on this front.

There is also the motivations of the scanner owner to consider. They are paid to do scans. A cheap scan might be low hundreds of dollars per hour. A more commercial rate would be well into the thousands. We don't want to dry up now do we?

Not really.

The scanner owners are almost all universities. One typically does have to pay to use the scanner(and quite a bit--$500+ per hour is typical), but this is typically meant to cover the scanner costs. The equipment costs at least four million dollars, plus some ongoing costs (cryogens, typically a full-time tech or two, and maintenance; the power bill is probably not trivial either).

Re: fMRI software bugs could upend years of research

#190

Post by nonbel »

mattkrause wrote: post #188
nonbel wrote: post #168

Earlier quoted context omitted.

>"They're saying that the pipeline that runs between the raw DICOM output of the scanner and the t-test, which is an equally important part of the testing procedure, is flawed." Yes, in other words the null hypothesis is being rendered false, by some aspect of the analysis pipeline. I am saying that is the correct description of the problem. Describing the problem as excess false positives is confused, because these…

> You have agreed that something was wrong, but want to call it something other than the null model. That's not really what I've said. In the previous NeuroImage paper, they generated null data by "imposing" block and event-related designs over resting state data. When they did a single subject analysis of that data with SPM, they found that the block designs had excess "positive[1]" results. However, analyzing data…

> "I agree that a p-value histogram would be nice, but I don't think it's essential."

Sure it is, here is an example using R where only about 2% of the tests report p

  set.seed(1234)
  Nsim=1000;   n=100
  p=matrix(nrow=Nsim,ncol=1)
  m=matrix(nrow=Nsim,ncol=2)
  for(i in 1:Nsim){
    a=rnorm(n,0,1); b=rcauchy(n,0,1)
    p[i,] = t.test(a,b, var.equal=T)$p.value
    m[i,] = cbind(mean(a),mean(b))
    sig = sum(ifelse(p
You can see from the histogram that there is a very clear deviation from the null hypothesis, and the %p values under 0.05 doesn't come close to telling the story of what is going on: https://s31.postimg.org/n9w3ydr63/Capture.jpg

> "I suppose these are "true positives" from the perspective of the final t-test, but only because an earlier part of the analysis failed to remove something it should have. It seems like weird to draw a line in the middle of the analysis like that."

Yes, precisely. That is the only perspective that matters because the p-value is being used as the actionable statistic in the end. This p-value has no necessary connection to what you think the null hypothesis was, it has to do with the actual values and calculations that were used.

Any other perspective is the perspective of someone who is confused about what hypothesis they are testing. This is, once again, not any fault of the statistics (maybe stats teachers... but that is another issue). Choosing an appropriate null hypothesis (that actually reflects what you believe is going on) is an investigation-specific logical problem outside the realm of statistics.

Post reply on HN