Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

I don't use Bayes factors in my research (2019)

datacolada.org

Post reply on HN
Print view View original

71–78 of 78 posts

Re: I don't use Bayes factors in my research (2019)

#71

Post by kgwgk »

travisjungroth wrote: post #70
kgwgk wrote: post #68

Earlier quoted context omitted.

A> Pick a measure. That’s what I mean by “effect is 0%”. It’s a straw man here. I don't understand what you mean by "pick a measure" but maybe the "it’s a straw man here" (that I don't really understand either) indicates that looking at the other two options is enough. B> Pick a fully specified model. This is a model that, up front, you could ask what is the probability of event E? For a normal distribution, this wou…

“Pick a measure” just meant that you’re predicting the difference will be exactly 0%. P(0) = 1. The difference between a Bayes factor and a likelihood ratio is Bayes factor uses the marginal likelihood. So you need to pick your parameters ahead of time, weighted by priors. With a likelihood ratio, you can use the best parameters given the data. You can do the likelihood ratio in an objective way, because you’re choos…

> “Pick a measure” just meant that you’re predicting the difference will be exactly 0%.

The difference of what? If you mean for example the difference between the population means of two groups that doesn’t mean that the observed difference between two sample means is zero. A non-zero observation doesn’t mean that “the model loses”. A non-zero observed difference is not just something that can happen, it’s what is expected.

If you mean that the null hypothesis is really “the difference between the observed means is exactly zero” that doesn’t seem very useful and I’ve never seen anyone do that. You don’t need statistics of any kind to reject the model “the observation is zero” when the observation is not zero.

Apart from that I agree that different hypothesis testing procedures do different things and their respective merits are debatable. My point was just that if you have a well-defined “null” model you can do different things with it. Using the same exact model.

Re: I don't use Bayes factors in my research (2019)

#72

Post by travisjungroth »

kgwgk wrote: post #71
travisjungroth wrote: post #70

Earlier quoted context omitted.

“Pick a measure” just meant that you’re predicting the difference will be exactly 0%. P(0) = 1. The difference between a Bayes factor and a likelihood ratio is Bayes factor uses the marginal likelihood. So you need to pick your parameters ahead of time, weighted by priors. With a likelihood ratio, you can use the best parameters given the data. You can do the likelihood ratio in an objective way, because you’re choos…

> “Pick a measure” just meant that you’re predicting the difference will be exactly 0%. The difference of what? If you mean for example the difference between the population means of two groups that doesn’t mean that the observed difference between two sample means is zero. A non-zero observation doesn’t mean that “the model loses”. A non-zero observed difference is not just something that can happen, it’s what is ex…

> that doesn’t seem very useful and I’ve never seen anyone do that.

Yes. That’s why it’s a straw man. I’m being sort of uncharitable in that description. My point is that’s a starting point. To go from there, you need to choose a model which will have parameters or priors.

Re: I don't use Bayes factors in my research (2019)

#73

Post by kgwgk »

travisjungroth wrote: post #72
kgwgk wrote: post #71

Earlier quoted context omitted.

> “Pick a measure” just meant that you’re predicting the difference will be exactly 0%. The difference of what? If you mean for example the difference between the population means of two groups that doesn’t mean that the observed difference between two sample means is zero. A non-zero observation doesn’t mean that “the model loses”. A non-zero observed difference is not just something that can happen, it’s what is ex…

> that doesn’t seem very useful and I’ve never seen anyone do that. Yes. That’s why it’s a straw man. I’m being sort of uncharitable in that description. My point is that’s a starting point. To go from there, you need to choose a model which will have parameters or priors.

That’s where the « null hypothesis » comes in. If it fixes the parameters in the model you get a well-defined « null » model with a well-defined probability distribution for the observation and - just like you can take this null hypothesis model and do frequentist calculations with it - you can take this model and calculate a Bayes factor relative to some other model.

(To be clear, if the null hypothesis doesn’t fully specify the parameters the preceding paragraph doesnt apply and the situation is more complex.)

Re: I don't use Bayes factors in my research (2019)

#74

Post by LudwigNagasena »

travisjungroth wrote: post #67
LudwigNagasena wrote: post #64

Earlier quoted context omitted.

Then I don't get the meaning of this: > Bayes factors work with comparing models. There is no null model. What, 0% effect? Ok, there was a non-zero effect. That model loses since it put the probability of 0% at 1 and everything else at 0. And if you do anything else, you’re encoding some amount of belief into the model, some judgment you’ve made. > So, you need to pick two models and compare them. I’m not saying this…

> It is the same way with traditional hypothesis testing. You take two models and compare their likehood. With a Bayes factor you compare the marginal likelihood. You have to account for the weight of the parameters according to the priors. With a likelihood ratio, you pick the best parameters and take the ratio of those likelihoods. This means a model used in a Bayes factor must be able to make predictions that foll…

> With a Bayes factor you compare the marginal likelihood. You have to account for the weight of the parameters according to the priors. With a likelihood ratio, you pick the best parameters and take the ratio of those likelihoods.

Yeah, that's the difference that I mentioned. And seems very different from whatever "it put the probability of 0% at 1 and everything else at 0" is supposed to refer to.

> This means a model used in a Bayes factor must be able to make predictions that follow probability axioms. Models in likelihood ratios don’t have this restriction.

Models in likehood ratios absolutely have to follow probability axioms, otherwise it would make no sense to apply probability axioms to study them.

Re: I don't use Bayes factors in my research (2019)

#75

Post by LudwigNagasena »

randomcarbloke wrote: post #61
LudwigNagasena wrote: post #36

Earlier quoted context omitted.

I think there is some confusion going on. Nobody claims that there is the null hypothesis. I think you are fighting windmills. Let’s say you study P. You know that P belongs to the family 𝒫. For example, 𝒫 = {N(μ,σ²): μ∈ℝ, σ²>0}. To be aware of 𝒫 is a prerequisite to do any sort of testing. For example, to test H0: μ=0 vs H1: μ≠0. After all, a typical hypothesis test is just a likehood ratio test. What you don’t h…

>Nobody claims that there is the null hypothesis. I think you are fighting windmills. it says it right there in the blog post...

It provides a null hypothesis. You can compare two hypotheses without any of them designating absence of effect. You don't have to have "the null hypothesis" in some philosophical sense to do a t-test and whatnot.

Re: I don't use Bayes factors in my research (2019)

#76

Post by LudwigNagasena »

amluto wrote: post #58
LudwigNagasena wrote: post #36

Earlier quoted context omitted.

I think there is some confusion going on. Nobody claims that there is the null hypothesis. I think you are fighting windmills. Let’s say you study P. You know that P belongs to the family 𝒫. For example, 𝒫 = {N(μ,σ²): μ∈ℝ, σ²>0}. To be aware of 𝒫 is a prerequisite to do any sort of testing. For example, to test H0: μ=0 vs H1: μ≠0. After all, a typical hypothesis test is just a likehood ratio test. What you don’t h…

I think this approach to non-Bayesian hypothesis testing is dangerous or misleading. > Let’s say you study P. You know that P belongs to the family 𝒫. For example, 𝒫 = {N(μ,σ²): μ∈ℝ, σ²>0}. To be aware of 𝒫 is a prerequisite to do any sort of testing. Okay, sure: you’ve decided on a family of probability distributions. It’s surely an approximation (very few things you would test are actually Gaussian — for one thi…

What is dangerous or misleading? It is a formulation I learned in my graduate statistics course. It is the one that makes the most sense and requires the least amount of handwaving.

> But H0 occupies a lower-dimensional space than 𝒫 — it’s a measure-zero subset.

In non-Bayesian framework your hypotheses don't have to be a part of a measurable structure at all. Nevertheless, if you have a measure, it doesn't have to be a zero at every point. I think it is quite intuitive to see. Let's look at two questions: (1) Does X have an effect? (2) How large is that effect? If your prior puts a non-zero probability that the answer to the first question is "No", then priors for the second question will have non-zero at point 0, even though the probability of any other point may be zero.

> And this results in what I consider to be useless headline results

These headlines don't have anything to do with Bayesian vs non-Bayesian as far as I see. People not doing power analysis is a people problem, not an issue with a statistical framework.

Re: I don't use Bayes factors in my research (2019)

#77

Post by amluto »

LudwigNagasena wrote: post #76
amluto wrote: post #58

Earlier quoted context omitted.

I think this approach to non-Bayesian hypothesis testing is dangerous or misleading. > Let’s say you study P. You know that P belongs to the family 𝒫. For example, 𝒫 = {N(μ,σ²): μ∈ℝ, σ²>0}. To be aware of 𝒫 is a prerequisite to do any sort of testing. Okay, sure: you’ve decided on a family of probability distributions. It’s surely an approximation (very few things you would test are actually Gaussian — for one thi…

What is dangerous or misleading? It is a formulation I learned in my graduate statistics course. It is the one that makes the most sense and requires the least amount of handwaving. > But H0 occupies a lower-dimensional space than 𝒫 — it’s a measure-zero subset. In non-Bayesian framework your hypotheses don't have to be a part of a measurable structure at all. Nevertheless, if you have a measure, it doesn't have to…

I’ve never taken graduate statistics, but I’m generally not impressed with the state of statistics or statistics education. Just because almost every university teaches almost every student this (and most published papers work this way) doesn’t mean it’s good. Intuitive, sure. Wise, not so much.

> Let's look at two questions: (1) Does X have an effect? (2) How large is that effect? If your prior puts a non-zero probability that the answer to the first question is "No", then priors for the second question will have non-zero at point 0, even though the probability of any other point may be zero.

Even ignoring whether it makes sense to have a nonzero prior for X having no effect, it’s generally not useful to learn the answer. An arbitrarily small effect is, for practical purposes, indistinguishable from no effect — an experiment that is intended to be useful should say something about the size of an effect. If a pill helps depression enough to be worth taking the pill, that’s one thing. If it helps in the sense that you could dose literally everyone in the world and one person would feel very slightly better for a day, that’s not helpful. Similarly, the existence of an effect says nothing about the sign of the effect.

So I think hypotheses being tested should be useful. If you want to determine whether something is useful, at least set a threshold for usefulness and test that. Or come up with a quantitative measure. The Bayesian-vs-frequentist debate is IMO somewhat secondary to this except insofar as it seems less common to make worthless but mathematically correct Bayesian tests because thinking about priors at all requires some acknowledgment of whether a prior is remotely plausible.

Also, how exactly can you make a well defined experiment that can confirm the null hypothesis without putting something resembling a prior on the non-null hypothesis if the non-null hypothesis contains distributions that are arbitrarily close to null? I’m sure it’s doable, but it seems likely to be pretty messy if you dig in.

(There are exceptions. For example, the existence of a neutrino mass is very interesting irrespective of what that mass is. But even then, physics results like this generally put bounds on a value that is hypothesized to be zero instead of merely testing for zeroness, because every experiment has finite power to detect small effects, and the readers of an outcome of an experiment should care more about the detection limits than, say, the number of dollars the experiment cost.)

Re: I don't use Bayes factors in my research (2019)

#78

Post by LudwigNagasena »

amluto wrote: post #77
LudwigNagasena wrote: post #76

Earlier quoted context omitted.

What is dangerous or misleading? It is a formulation I learned in my graduate statistics course. It is the one that makes the most sense and requires the least amount of handwaving. > But H0 occupies a lower-dimensional space than 𝒫 — it’s a measure-zero subset. In non-Bayesian framework your hypotheses don't have to be a part of a measurable structure at all. Nevertheless, if you have a measure, it doesn't have to…

I’ve never taken graduate statistics, but I’m generally not impressed with the state of statistics or statistics education. Just because almost every university teaches almost every student this (and most published papers work this way) doesn’t mean it’s good. Intuitive, sure. Wise, not so much. > Let's look at two questions: (1) Does X have an effect? (2) How large is that effect? If your prior puts a non-zero proba…

> it’s generally not useful to learn the answer

The point of priors is to represent beliefs about reality. It would certainly be surprising if you had a zero prior for neutrino to have no charge. Why cannot people have similar beliefs about drugs?

> an experiment that is intended to be useful should say something about the size of an effect

I agree. Nothing stops people who study depression drugs from doing power analysis and reporting those numbers.

> how exactly can you make a well defined experiment that can confirm the null hypothesis

I think you refer to the common of idea of rejecting or not rejecting H0. Well, from the point of view of statistical decision theory, the distinction between non-rejecting and accepting just doesn’t exist. You either choose H0 or H1. If you had n=1, of course there is a high chance of Type II error. Do power analysis and design an appropriate experiment given your desired significance level, statistical power and effect size.

Post reply on HN