Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

fMRI software bugs could upend years of research

theregister.co.uk

Post reply on HN
Print view View original

91–100 of 192 posts

Re: fMRI software bugs could upend years of research

#91

Post by tansey »

toyg wrote: post #80
jballanc wrote: post #78

The real takeaway lesson from this research should be the vital importance of Open Data to the modern scientific enterprise: > "lamentable archiving and data-sharing practices" that prevent most of the discipline's body of work being re-analysed. Keeping data private before publication is (at this point in time) understandable. Once results are published, however, there is no excuse for not depositing the raw data in…

Uh, where would you store it though? IIRC from the time a relative was going through a chemistry PHD, they produced several GBs of raw data every half hour . Storage is cheap but it's not that cheap...

It's not that much data. You're storing about 2M voxel image every 2s or so. There are open repositories out there already: https://openfmri.org/

Re: fMRI software bugs could upend years of research

#92

Post by zachrose »

toyg wrote: post #80
jballanc wrote: post #78

The real takeaway lesson from this research should be the vital importance of Open Data to the modern scientific enterprise: > "lamentable archiving and data-sharing practices" that prevent most of the discipline's body of work being re-analysed. Keeping data private before publication is (at this point in time) understandable. Once results are published, however, there is no excuse for not depositing the raw data in…

Uh, where would you store it though? IIRC from the time a relative was going through a chemistry PHD, they produced several GBs of raw data every half hour . Storage is cheap but it's not that cheap...

But how expensive was the lab and the instruments that produce that amount of data?

Re: fMRI software bugs could upend years of research

#93

Post by hammock »

randcraw wrote: post #44

As someone who works in the biomedical imaging business and is also a fan of philosophy, I think this news will matter more to folks in the latter camp. For a couple years now philosophers have insisted that fMRI images prove there is no such thing as free will. Today's revelation should put an end to that whole line of reasoning (and the absurd amount of fatalism that it engendered). (The back story: Apparently fMRI…

What are the actionable insights that have come out of fMRI studies? Even when properly conducted (no false positives), the conclusions that are often drawn have always felt dubious to me. Basically you are looking for regions of the brain that light up with various stimuli. Except that's as far as it goes, we don't yet understand much beyond that.

It's as if you figure out that your car is making a funny sound, and you can pinpoint where it is coming from, you can even reproduce the sound on demand - but you have no idea WHY it sounds the way it does.

Re: fMRI software bugs could upend years of research

#94

Post by bad_user »

Retric wrote: post #86
freshhawk wrote: post #54

Earlier quoted context omitted.

Well, the non-pop philosophers will understand that as a minor hit to the model of consciousness as solely concerned with post-facto rationalization and basically removed from the real-time decision making loop. Those studies were never strong evidence anyway, the protocols were pretty weak as they required asking the subjects to self-report at what time they "decided" to act. The real free will debate is about both…

Interestingly one of the best arguments apposed to free will is people being terrible random number generators. Which tends to be hand waved away. Which suggests people have already decided which side they believe in and only try and rationalize it after the fact. ;)

I've never heard of that argument, though I admit I'm new to this discussion.

But, given that people can and have built pretty good random generators, by finding sources of entropy and even with pure math, thus sidestepping our limitations, the point is kind of moot IMHO.

Re: fMRI software bugs could upend years of research

#95

Post by jballanc »

zachrose wrote: post #92
toyg wrote: post #80

Earlier quoted context omitted.

Uh, where would you store it though? IIRC from the time a relative was going through a chemistry PHD, they produced several GBs of raw data every half hour . Storage is cheap but it's not that cheap...

But how expensive was the lab and the instruments that produce that amount of data?

Not to mention the researchers' and post-docs' and students' time. By not paying to store it, you're risking the possibility that all of this money is essentially flushed down the toilet by a software bug because it cannot now be reanalyzed.

Re: fMRI software bugs could upend years of research

#96

Post by nickledave »

nonbel wrote: post #75
nickledave wrote: post #69

Earlier quoted context omitted.

See mattkrause's comment below. I think you might not be understanding what they're testing. They're taking only resting state data, randomly sorting some of it into "pretend active state data" and then asking whether they get any statistically significant difference between these two groups when they shouldn't . But they do . That means the tests they used, the same tests many authors use, are giving false positives…

I think that like mattkrause (and the authors of the current work), you have forgotten that the null hypothesis is something larger than condition 1 == condition 2. There are various other components, usually (somewhat misleadingly) referred to as assumptions, that can also cause the predictions derived from the null model to deviate from the data. For stuff like a t-test, one parameter value (ie the mean of the dist…

I'm aware that there are assumptions implicit in what the null hypothesis is. You are the one who keeps saying the authors don't even realize what those assumptions are, but you haven't pointed out anything besides what the authors said. What are the other faulty assumptions you've identified that the authors are missing? I guess you're referring to some sort of issue with power that you mentioned in your previous comment?

Re: fMRI software bugs could upend years of research

#97

Post by mattkrause »

nonbel wrote: post #79
mattkrause wrote: post #76

Earlier quoted context omitted.

Can you explain what those assumptions are then, even roughly? I read the PNAS paper as being an extension of their previous work, not a contradiction of it. They've scaled the data set up (the previous paper had less data), added more packages (the previous paper used only SPM), and ran it as a group analysis (the previous paper was within subjects). They pretty much say that right in the PNAS text: "That work found…

>"Can you explain what those assumptions are then, even roughly?" I have not looked at the code myself. I only read their description of it. As I quoted earlier: "assumed activity paradigm can have a large impact on the degree of false positives" If the assumed activity paradigm affects (what they are calling) the false positive rate, it is obviously part of the model. If they are arbitrarily varying it because they…

Maybe we're agreeing?

The ENTIRE POINT of the paper is that there are inappropriate assumptions baked into the defaults of FSL and SPM. (AFNI has an actual code-does-the-wrong-thing bug). These errors result in cluster-wise tests being inappropriately sized: when thresholds are set to pThey tested this by dividing a single stream of "off" data into segments of "ON" and "OFF" data. They do this in four different ways (two block paradigms and two event-related paradigms). Most fMRI experiments use a paradigm that is close to one of these, so this isn't a fishing attempt--it's meant to make the results broadly applicable .

They argue that, by construction, no ON data is present in this pseudo-data. Therefore, any ON calls must be spurious/false alarms/type I errors/etc, which lets you calculate the actual FWER and compare it with its nominal size.

You seem to be arguing that this step is not true--some other assumption, which they knew about(?), causes it. The NeuroImage paper says that different assumed activity paradigms have different false positive rates because the assumed model is dumb. The event-related paradigms alternate between on and off rapidly (and at random), which washes out the effects of unmodelled low frequencies. The much slower block designs, on the other hand, are especially susceptible to this problem because they spend a long time in each condition.

The current work extends that to look at other ways in which the default assumptions might be dumb in SPM, while adding in more data, different tests, and other packages.

I am open to the argument that the Block Design data (e.g., in Figure 2 in PNAS) does not add much because the NeuroImage paper already suggests those designs are screwed. However, it also says that the event-related designs are actually in pretty good shape: the voxel-based familywise error rates in the NeuroImage paper are about right (Figure 2), but the cluster FWER for the same design is through the roof in the PNAS paper.

Otherwise though, I don't see many unwarranted/undiscussed assumptions in the paper. If you want to keep claiming that the authors (and I) are making wild assumptions, it would be nice to have a vague sense of what they are and why they/I am wrong.

Re: fMRI software bugs could upend years of research

#98

Post by mattkrause »

nickledave wrote: post #70
nickledave wrote: post #69

Earlier quoted context omitted.

See mattkrause's comment below. I think you might not be understanding what they're testing. They're taking only resting state data, randomly sorting some of it into "pretend active state data" and then asking whether they get any statistically significant difference between these two groups when they shouldn't . But they do . That means the tests they used, the same tests many authors use, are giving false positives…

@mattkrause: you beat me to it

Eh, you'll scoop me on something when it matters.

PS: If you are the song bird guy, I saw a talk of yours on youtube. Cool stuff.

Re: fMRI software bugs could upend years of research

#99

Post by nonbel »

nickledave wrote: post #96
nonbel wrote: post #75

Earlier quoted context omitted.

I think that like mattkrause (and the authors of the current work), you have forgotten that the null hypothesis is something larger than condition 1 == condition 2. There are various other components, usually (somewhat misleadingly) referred to as assumptions, that can also cause the predictions derived from the null model to deviate from the data. For stuff like a t-test, one parameter value (ie the mean of the dist…

I'm aware that there are assumptions implicit in what the null hypothesis is. You are the one who keeps saying the authors don't even realize what those assumptions are, but you haven't pointed out anything besides what the authors said. What are the other faulty assumptions you've identified that the authors are missing? I guess you're referring to some sort of issue with power that you mentioned in your previous co…

>"You are the one who keeps saying the authors don't even realize what those assumptions are, but you haven't pointed out anything besides what the authors said."

I am saying they are confused because they say "the null hypothesis should be true" under their conditions, when they know for certain that it is false! Therefore these "false positives" are not false at all. They are totally legit "true positives".

These authors are blaming the statistical test when the problem lies with their crappy choice of null hypothesis.

There may very well be other issues, but I have not inspected the code or done anything other than read the description in the paper.

Post reply on HN