Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

I don't use Bayes factors in my research (2019)

datacolada.org

Post reply on HN
Print view View original

11–20 of 78 posts

Re: I don't use Bayes factors in my research (2019)

#11

Post by concinds »

p-e-w wrote: post #3

The statistical interpretation of observations is so subtle and complex that it's a good idea to assume that any publication from the empirical sciences is complete garbage, until you know for sure that a qualified statistician has supervised the process. A semester of "introduction to statistical methods" (which is all the background that most scientists have) is NOT enough. Imagine a mathematician writing a paper o…

I read a survey once, that found that a huge number of PhDs/researchers in the studied sample gave an incorrect definition for what a "95% confidence interval" (/p-value, etc) actually means, and that several popular introductory textbooks defined it incorrectly as well. Wish I bookmarked it.

At bare minimum, journals need to require that researchers publish all their data alongside every paper, so statistical analyses can be redone and flaws can be spotted.

Re: I don't use Bayes factors in my research (2019)

#12

Post by [deleted] »

concinds wrote: post #11
p-e-w wrote: post #3

The statistical interpretation of observations is so subtle and complex that it's a good idea to assume that any publication from the empirical sciences is complete garbage, until you know for sure that a qualified statistician has supervised the process. A semester of "introduction to statistical methods" (which is all the background that most scientists have) is NOT enough. Imagine a mathematician writing a paper o…

I read a survey once, that found that a huge number of PhDs/researchers in the studied sample gave an incorrect definition for what a "95% confidence interval" (/p-value, etc) actually means, and that several popular introductory textbooks defined it incorrectly as well. Wish I bookmarked it. At bare minimum, journals need to require that researchers publish all their data alongside every paper, so statistical analys…

[deleted]

Re: I don't use Bayes factors in my research (2019)

#13

Post by neonate »

jldugger wrote: post #8

> Note: By theory I merely mean the rationale for investigating the effect of x on y. A theory can be as simple as “I think people value a mug more once they own it”. Hoo boy, the [2019] is well deserved on this one -- that's a dan arielly reference from before The 2021 Accusation and before the recent NPR story refuting his excuse[1]. [1]: https://www.npr.org/2023/07/27/1190568472/dan-ariely-frances...

What is the reference? I don't get it.

Re: I don't use Bayes factors in my research (2019)

#14

Post by constantcrying »

Konohamaru wrote: post #2

Milton Friedman was correct: because the true minimum wage is $0.00 (unemployment), he was correct to compare wage increase to the null hypothesis. The potshot in the opening paragraph ("Milton feels bad about the unemployed but good about his theory.") is simultaneously an appeal to emotion and a presumptuous ad hominem.

The point isn't that Friedman is right or wrong, but that the statistic model tells him to reject his hypothesis, even though he observed a result consistent with his hypothesis.

>is simultaneously an appeal to emotion and a presumptuous ad hominem.

I don't see how that is the case.

Re: I don't use Bayes factors in my research (2019)

#16

Post by travisjungroth »

Konohamaru wrote: post #2

Milton Friedman was correct: because the true minimum wage is $0.00 (unemployment), he was correct to compare wage increase to the null hypothesis. The potshot in the opening paragraph ("Milton feels bad about the unemployed but good about his theory.") is simultaneously an appeal to emotion and a presumptuous ad hominem.

Potshot? It just seems like a joke, but one that puts this character (a nod to Milton Friedman but not like a serious insert) in a positive light. He’s pleased by being correct but sympathetic since he was right about something bad happening to people (unemployment).

Re: I don't use Bayes factors in my research (2019)

#17

Post by constantcrying »

concinds wrote: post #11
p-e-w wrote: post #3

The statistical interpretation of observations is so subtle and complex that it's a good idea to assume that any publication from the empirical sciences is complete garbage, until you know for sure that a qualified statistician has supervised the process. A semester of "introduction to statistical methods" (which is all the background that most scientists have) is NOT enough. Imagine a mathematician writing a paper o…

I read a survey once, that found that a huge number of PhDs/researchers in the studied sample gave an incorrect definition for what a "95% confidence interval" (/p-value, etc) actually means, and that several popular introductory textbooks defined it incorrectly as well. Wish I bookmarked it. At bare minimum, journals need to require that researchers publish all their data alongside every paper, so statistical analys…

I think you may be talking about "Mindless statistics" by Gigerenzer. He has some surveys about p-values and how radically wrong they are usually interpreted.

>At bare minimum, journals need to require that researchers publish all their data alongside every paper, so statistical analyses can be redone and flaws can be spotted.

Absolutely.

Re: I don't use Bayes factors in my research (2019)

#18

Post by samch93 »

concinds wrote: post #11
p-e-w wrote: post #3

The statistical interpretation of observations is so subtle and complex that it's a good idea to assume that any publication from the empirical sciences is complete garbage, until you know for sure that a qualified statistician has supervised the process. A semester of "introduction to statistical methods" (which is all the background that most scientists have) is NOT enough. Imagine a mathematician writing a paper o…

I read a survey once, that found that a huge number of PhDs/researchers in the studied sample gave an incorrect definition for what a "95% confidence interval" (/p-value, etc) actually means, and that several popular introductory textbooks defined it incorrectly as well. Wish I bookmarked it. At bare minimum, journals need to require that researchers publish all their data alongside every paper, so statistical analys…

probably this article: Hoekstra, R., Morey, R.D., Rouder, J.N. et al. Robust misinterpretation of confidence intervals. Psychon Bull Rev 21, 1157–1164 (2014). https://doi.org/10.3758/s13423-013-0572-3

another good article on misinterpretation of p-values and confidence intervals is: Greenland, S., Senn, S.J., Rothman, K.J. et al. Statistical tests, P values, confidence intervals, and power: a guide to misinterpretations. Eur J Epidemiol 31, 337–350 (2016). https://doi.org/10.1007/s10654-016-0149-3

Re: I don't use Bayes factors in my research (2019)

#19

Post by ggm »

constantcrying wrote: post #14
Konohamaru wrote: post #2

Milton Friedman was correct: because the true minimum wage is $0.00 (unemployment), he was correct to compare wage increase to the null hypothesis. The potshot in the opening paragraph ("Milton feels bad about the unemployed but good about his theory.") is simultaneously an appeal to emotion and a presumptuous ad hominem.

The point isn't that Friedman is right or wrong, but that the statistic model tells him to reject his hypothesis, even though he observed a result consistent with his hypothesis. >is simultaneously an appeal to emotion and a presumptuous ad hominem. I don't see how that is the case.

Because it makes Friedman heartless. He feels bad but he still promulgated theories which wreaked the badness he felt bad about. So it goes to character.

If he _really_ felt bad, he'd have done what Norbert Weiner did and move out of the field. He stayed an economist. Not so bad feeling, eh?

Re: I don't use Bayes factors in my research (2019)

#20

Post by travisjungroth »

I’ve been working a lot with Bayes factors lately. I don’t want to sound cultish, but I think part of the issue is this stuff doesn’t work “half way”. As soon as you’re talking about the null hypothesis and Bayes factors, you’re mixing up two schools of thought that don’t play nice.

Bayes factors work with comparing models. There is no null model. What, 0% effect? Ok, there was a non-zero effect. That model loses since it put the probability of 0% at 1 and everything else at 0. And if you do anything else, you’re encoding some amount of belief into the model, some judgment you’ve made.

So, you need to pick two models and compare them. I’m not saying this is right for science. It’s working well for my purposes. One model meaning “as planned”, one model meaning “not as planned”, use the Bayes factor to decide if things are going as planned. But you do need to be explicit about what models you’re comparing. You have to be able to just put some data in and get a probability back, or it’s not going to work.

This is what makes this criticism of Bayes factors so unpersuasive. They’re very easy to calculate, but they’re never calculated here! It’s just the ratio of marginal likelihoods, the probability of the data under the model.

Post reply on HN