Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

I don't use Bayes factors in my research (2019)

datacolada.org

Post reply on HN
Print view View original

31–40 of 78 posts

Re: I don't use Bayes factors in my research (2019)

#31

Post by travisjungroth »

LudwigNagasena wrote: post #27
travisjungroth wrote: post #20

I’ve been working a lot with Bayes factors lately. I don’t want to sound cultish, but I think part of the issue is this stuff doesn’t work “half way”. As soon as you’re talking about the null hypothesis and Bayes factors, you’re mixing up two schools of thought that don’t play nice. Bayes factors work with comparing models. There is no null model. What, 0% effect? Ok, there was a non-zero effect. That model loses sin…

> Bayes factors work with comparing models. So does the traditional Neyman–Pearson hypothesis testing. > There is no null model. Why can’t there be? > What, 0% effect? Ok, there was a non-zero effect. That model loses since it put the probability of 0% at 1 and everything else at 0%. Well, if your null hypothesis is deterministic and says 0% effect, getting anything other than 0% absolutely will make you reject the n…

You've sliced up what I've said to the point it doesn't really make sense. This is exactly what I said was confusing. I'm talking about Bayes factors and you're talking about null hypothesis testing.

I'll just answer your question, why there can't be a null model. You can have a hypothesis that represents all differences between groups are due to chance. To make this a statistical model, something that can calculate the probability of an event, you have to make assumptions. Maybe it's just about the distribution. Maybe it's independence. But, it's always something. You said it yourself "You don’t assume any belief on the probability of a specific model to be true." To be a statistical model, to calculate the probability of an event, to calculate marginal likelihoods, to calculate Bayes factors, you have to do that.

This is largely a philosophical point. You can have a null model. Something you pick to represent "no effect". But there's not the null, this belief free model that's categorically different from a model with priors.

If there's a belief-free model that can give a marginal likelihood, then I'm wrong. I'd also very much like to know about it.

Re: I don't use Bayes factors in my research (2019)

#32

Post by bakuninsbart »

concinds wrote: post #11
p-e-w wrote: post #3

The statistical interpretation of observations is so subtle and complex that it's a good idea to assume that any publication from the empirical sciences is complete garbage, until you know for sure that a qualified statistician has supervised the process. A semester of "introduction to statistical methods" (which is all the background that most scientists have) is NOT enough. Imagine a mathematician writing a paper o…

I read a survey once, that found that a huge number of PhDs/researchers in the studied sample gave an incorrect definition for what a "95% confidence interval" (/p-value, etc) actually means, and that several popular introductory textbooks defined it incorrectly as well. Wish I bookmarked it. At bare minimum, journals need to require that researchers publish all their data alongside every paper, so statistical analys…

While I agree on the data point, it would kill so much research. It is bad that a lot of research validation basically comes down to "trust me guys", but with data being both very valuable and often times highly sensitive, it can be really difficult to just publish the data along with the research.

A decent compromise would be to at least require meta-data to sufficiently exclude some flaws. A different approach could be to have researchers document and publish the process of th research, similar to a git-repo with the main branch being completely off limits to history-rewriting.

Re: I don't use Bayes factors in my research (2019)

#33

Post by ggm »

constantcrying wrote: post #30
ggm wrote: post #26

Earlier quoted context omitted.

> None of this has to do with article, which makes a very simple point about statistics. I am addressing a 'how is this ad hom' question not Friedman's character or the article. I seek to explain what view of him would be a critique of character not substance.

It isn't an adhom because it is irrelevant.

This isn't a very good line of reasoning because it's a non sequiteur. At this point I think you're trolling more than discussing rationally.

Re: I don't use Bayes factors in my research (2019)

#34

Post by civilized »

Bayesian methods are not easy to use, I agree. But that's because they're trying to answer much more meaningful (but harder) questions than frequentist ones, which researchers should be trying to do. You can't ignore Bayesian epistemology just by not using Bayesian methods. The underlying considerations in the Bayesian framework will inevitably become relevant to how you interpret your data, whether or not you use a formal Bayesian method.

The thing is, formally, frequentist methods like Null Hypothesis Significance Testing don't tell you what you really want to know. If you get a significant p-value, that means the data you observed wouldn't often happen by chance (within your model of the null). This doesn't actually tell you if your particular hypothesis should be favored. That requires other considerations, including ones that Simonsohn is negative about in this article.

For example, Simonsohn's conclusion says:

> To use Bayes factors to test hypotheses: you need to be OK with the following two things:

> 1. Accepting the null when “the alternative” you consider, and reject, does not represent the theory of interest.

> 2. Rejecting a theory after observing an outcome that the theory predicts.

He implies that these should be points against Bayes factors. But #2 is something you actually should do sometimes. Demonstrably. If the data suggests a wildly implausible effect size that doesn't show up consistently in other analyses, that should be a point against your theory and in favor of some more mundane explanation, like noisy data from an underpowered study [1].

Not using Bayesian methods is understandable if you don't feel comfortable with the very heavy demands they can make on your statistics acumen. But if you're, say, a social scientist incentivized to get "sexy" results and you refuse to engage with Bayesian epistemology at all, your career will almost certainly just be a contribution of more noise publications to the replication crisis.

[1] http://www.stat.columbia.edu/~gelman/presentations/ziff.pdf

Re: I don't use Bayes factors in my research (2019)

#36

Post by LudwigNagasena »

travisjungroth wrote: post #31
LudwigNagasena wrote: post #27

Earlier quoted context omitted.

> Bayes factors work with comparing models. So does the traditional Neyman–Pearson hypothesis testing. > There is no null model. Why can’t there be? > What, 0% effect? Ok, there was a non-zero effect. That model loses since it put the probability of 0% at 1 and everything else at 0%. Well, if your null hypothesis is deterministic and says 0% effect, getting anything other than 0% absolutely will make you reject the n…

You've sliced up what I've said to the point it doesn't really make sense. This is exactly what I said was confusing. I'm talking about Bayes factors and you're talking about null hypothesis testing. I'll just answer your question, why there can't be a null model. You can have a hypothesis that represents all differences between groups are due to chance. To make this a statistical model, something that can calculate…

I think there is some confusion going on. Nobody claims that there is the null hypothesis. I think you are fighting windmills.

Let’s say you study P. You know that P belongs to the family 𝒫. For example, 𝒫 = {N(μ,σ²): μ∈ℝ, σ²>0}. To be aware of 𝒫 is a prerequisite to do any sort of testing. For example, to test H0: μ=0 vs H1: μ≠0. After all, a typical hypothesis test is just a likehood ratio test.

What you don’t have to know or even to assume existence of—if you don’t do Bayesian stuff—is a probability measure Π on 𝒫 (and its appropriate sigma-field). That’s the philosophical difference. But you have to have well-defined 𝒫 either way.

The priors is Π. Existence of 𝒫 means that you have a family of models, but it doesn’t force you to assume any priors about those models. I don’t see how not having Π makes P∈𝒫 less of a model. You are still allowed to do conditional reasoning, eg the aforementioned type I and type II errors.

Re: I don't use Bayes factors in my research (2019)

#37

Post by kkoncevicius »

civilized wrote: post #34

Bayesian methods are not easy to use, I agree. But that's because they're trying to answer much more meaningful (but harder) questions than frequentist ones, which researchers should be trying to do. You can't ignore Bayesian epistemology just by not using Bayesian methods. The underlying considerations in the Bayesian framework will inevitably become relevant to how you interpret your data, whether or not you use a…

> Bayesian methods are not easy to use, I agree. But that's because they're trying to answer much more meaningful (but harder) questions than frequentist ones

I think your view about the difference between Frequentist and Bayesian methods is wrong. There is this rant I like from Larry Wasserman [1] on the subject:

  My opinions have shifted a bit. [...] Bayes-Frequentist debate still matters. And people — including many statisticians — are still confused about the distinction. I thought the basic Bayes-Frequentist debate was behind us. A year and a half of blogging (as well as reading other blogs) convinced me I was wrong here too. And this still does matter.
  My emphasis on high-dimensional models is germane, however. In our world of high-dimensional, complex models I can’t see how anyone can interpret the output of a Bayesian analysis in any meaningful way.
  I wish people were clearer about what Bayes is/is not and what frequentist inference is/is not. Bayes is the analysis of subjective beliefs but provides no frequency guarantees. Frequentist inference is about making procedures that have frequency guarantees but makes no pretense of representing anyone’s beliefs. In the high dimensional world, you have to choose: objective frequency guarantees or subjective beliefs. Choose whichever you prefer, but you can’t have both. I don’t care which one people pick; I just wish they would be clear about what they are giving up when they make their choice.
  [...]
  Of course, one can embrace objective Bayesian inference. If this means “Bayesian procedures with good frequentist properties” then I am all for it. But this is just frequentist inference in Bayesian clothing."
[1]: https://errorstatistics.com/2013/12/27/deconstructing-larry-...

Re: I don't use Bayes factors in my research (2019)

#38

Post by kgwgk »

kkoncevicius wrote: post #37
civilized wrote: post #34

Bayesian methods are not easy to use, I agree. But that's because they're trying to answer much more meaningful (but harder) questions than frequentist ones, which researchers should be trying to do. You can't ignore Bayesian epistemology just by not using Bayesian methods. The underlying considerations in the Bayesian framework will inevitably become relevant to how you interpret your data, whether or not you use a…

> Bayesian methods are not easy to use, I agree. But that's because they're trying to answer much more meaningful (but harder) questions than frequentist ones I think your view about the difference between Frequentist and Bayesian methods is wrong. There is this rant I like from Larry Wasserman [1] on the subject: My opinions have shifted a bit. [...] Bayes-Frequentist debate still matters. And people — including man…

> But this is just frequentist inference in Bayesian clothing.

Or maybe this is "just" frequentist inference done right - on a Bayesian foundation.

Re: I don't use Bayes factors in my research (2019)

#39

Post by civilized »

kkoncevicius wrote: post #37
civilized wrote: post #34

Bayesian methods are not easy to use, I agree. But that's because they're trying to answer much more meaningful (but harder) questions than frequentist ones, which researchers should be trying to do. You can't ignore Bayesian epistemology just by not using Bayesian methods. The underlying considerations in the Bayesian framework will inevitably become relevant to how you interpret your data, whether or not you use a…

> Bayesian methods are not easy to use, I agree. But that's because they're trying to answer much more meaningful (but harder) questions than frequentist ones I think your view about the difference between Frequentist and Bayesian methods is wrong. There is this rant I like from Larry Wasserman [1] on the subject: My opinions have shifted a bit. [...] Bayes-Frequentist debate still matters. And people — including man…

Having a specific opinion isn't inherently a bias, and it's rather uninspiring to put effort into a substantial comment and then get a reply which is essentially nothing but a baseless accusation of bias followed by a long block quote of unclear relevance (it's extremely far from true that all statistical analyses worth consideration are in high dimensions).

Review the Hacker News guidelines:

> Edit out swipes.

https://news.ycombinator.com/newsguidelines.html

EDIT: much appreciated.

Re: I don't use Bayes factors in my research (2019)

#40

Post by travisjungroth »

LudwigNagasena wrote: post #36
travisjungroth wrote: post #31

Earlier quoted context omitted.

You've sliced up what I've said to the point it doesn't really make sense. This is exactly what I said was confusing. I'm talking about Bayes factors and you're talking about null hypothesis testing. I'll just answer your question, why there can't be a null model. You can have a hypothesis that represents all differences between groups are due to chance. To make this a statistical model, something that can calculate…

I think there is some confusion going on. Nobody claims that there is the null hypothesis. I think you are fighting windmills. Let’s say you study P. You know that P belongs to the family 𝒫. For example, 𝒫 = {N(μ,σ²): μ∈ℝ, σ²>0}. To be aware of 𝒫 is a prerequisite to do any sort of testing. For example, to test H0: μ=0 vs H1: μ≠0. After all, a typical hypothesis test is just a likehood ratio test. What you don’t h…

> Nobody claims that there is the null hypothesis.

Google “the null hypothesis”.

If you mean null model, then I’m not fighting against anyone. We all agree which null model to use is a choice to be made.

Otherwise, I’m not even sure what you’re trying to convince me of at this point. I’ll restate the essence of my first comment more concisely.

Bayes factors are a method of model comparison. You take the ratio of marginal likelihoods for two models given the data. Choosing a null model for this purpose requires more assumptions than doing null hypothesis testing with frequentist statistics. Mixing the schools of thought of Bayesian and frequentist makes things more confusing than operating within them individually. Bayes factors have other uses than null hypothesis testing.

Maybe you straight up disagree with one of those sentences to the point you could quote it and say “this is wrong because…”.

I have this feeling you got to my second paragraph on my first comment and started quoting stuff before reading it all the way through. Or maybe you just had some stuff you really wanted to talk about. Because my whole point was how calculating Bayes factors and the normal mentality of null hypothesis testing don’t play nice together, but can have other benefits. So a line by line comparison doesn’t really make sense.

Post reply on HN