Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

I don't use Bayes factors in my research (2019)

datacolada.org

Post reply on HN
Print view View original

61–70 of 78 posts

Re: I don't use Bayes factors in my research (2019)

#61

Post by randomcarbloke »

LudwigNagasena wrote: post #36
travisjungroth wrote: post #31

Earlier quoted context omitted.

You've sliced up what I've said to the point it doesn't really make sense. This is exactly what I said was confusing. I'm talking about Bayes factors and you're talking about null hypothesis testing. I'll just answer your question, why there can't be a null model. You can have a hypothesis that represents all differences between groups are due to chance. To make this a statistical model, something that can calculate…

I think there is some confusion going on. Nobody claims that there is the null hypothesis. I think you are fighting windmills. Let’s say you study P. You know that P belongs to the family 𝒫. For example, 𝒫 = {N(μ,σ²): μ∈ℝ, σ²>0}. To be aware of 𝒫 is a prerequisite to do any sort of testing. For example, to test H0: μ=0 vs H1: μ≠0. After all, a typical hypothesis test is just a likehood ratio test. What you don’t h…

>Nobody claims that there is the null hypothesis. I think you are fighting windmills.

it says it right there in the blog post...

Re: I don't use Bayes factors in my research (2019)

#62

Post by travisjungroth »

kgwgk wrote: post #53
travisjungroth wrote: post #50

Earlier quoted context omitted.

Mean and standard deviation up front, not from the sample. At least, that would go against my understanding of Bayes factors and how I’ve calculated them. You can do other stats, t-test for example, without declaring that up front.

I don’t understand what you say, sorry. If my choice of null model is p(x)=exp(-x^2/2) and I get some observation xobs I can do frequentist things with it - like calculating the p-value p(|x|>|xobs|) for example - and I can compare it with some alternative model p’(x) using the Bayes factor p(xobs)/p’(xobs). What does “Mean and standard deviation up front, not from the sample.” mean in this context?

Mean and standard deviation up front would mean that you set your null as a normal distribution with a mean of 0 and a standard deviation of 2%, for example. This is different from saying that you’ll just say it’s normal, and take the mean and standard deviation from the sample.

I mean, I could be wrong on this. You could do that if you want. I just think of Bayes factors as a competition and it needs to be “fair”. So it doesn’t make sense to let the null update as data comes in but not the alternative.

Re: I don't use Bayes factors in my research (2019)

#63

Post by kgwgk »

travisjungroth wrote: post #62
kgwgk wrote: post #53

Earlier quoted context omitted.

I don’t understand what you say, sorry. If my choice of null model is p(x)=exp(-x^2/2) and I get some observation xobs I can do frequentist things with it - like calculating the p-value p(|x|>|xobs|) for example - and I can compare it with some alternative model p’(x) using the Bayes factor p(xobs)/p’(xobs). What does “Mean and standard deviation up front, not from the sample.” mean in this context?

Mean and standard deviation up front would mean that you set your null as a normal distribution with a mean of 0 and a standard deviation of 2%, for example. This is different from saying that you’ll just say it’s normal, and take the mean and standard deviation from the sample. I mean, I could be wrong on this. You could do that if you want. I just think of Bayes factors as a competition and it needs to be “fair”. S…

Ok.

As you said "Bayes factors work with comparing models."

I don't think that there is a concept of "fairness" that prevents us from including in the comparison a very simple model.

"There is no null model. What, 0% effect?"

For example. But even if the true effect is zero you may get a non-zero observation because measurements are not perfect.

If what you measure is precisely what you want to know there is no need for statistical analysis!

Let's say for the sake of this example that you know that your measurement error is distributed normally with unit variance.

"Ok, there was a non-zero effect."

There was a non-zero _observation_.

"That model loses since it put the probability of 0% at 1 and everything else at 0."

It didn't. The prediction of the null model is that observations will be normally distributed around 0.

That's precisely what the author of the blogpost complains about: that if your observation is 1 the prediction of the null model (close to 0) was more accurate than the prediction of the second model (somewhere between 1 and 10) and the observation favours the former over the latter.

Re: I don't use Bayes factors in my research (2019)

#64

Post by LudwigNagasena »

travisjungroth wrote: post #55
LudwigNagasena wrote: post #52

Earlier quoted context omitted.

> Google “the null hypothesis”. > If you mean null model, then I’m not fighting against anyone. We all agree which null model to use is a choice to be made. I don’t understand what difference you are trying to imply by drawing a distinction between a null model and a null hypothesis. > Otherwise, I’m not even sure what you’re trying to convince me of at this point. I’ll restate the essence of my first comment more co…

> > Bayes factors are a method of model comparison. > Are you implying that hypothesis testing isn’t? No.

Then I don't get the meaning of this:

> Bayes factors work with comparing models. There is no null model. What, 0% effect? Ok, there was a non-zero effect. That model loses since it put the probability of 0% at 1 and everything else at 0. And if you do anything else, you’re encoding some amount of belief into the model, some judgment you’ve made.

> So, you need to pick two models and compare them. I’m not saying this is right for science. It’s working well for my purposes. One model meaning “as planned”, one model meaning “not as planned”, use the Bayes factor to decide if things are going as planned. But you do need to be explicit about what models you’re comparing. You have to be able to just put some data in and get a probability back, or it’s not going to work.

It is the same way with traditional hypothesis testing. You take two models and compare their likehood.

Re: I don't use Bayes factors in my research (2019)

#65

Post by randysalami »

p-e-w wrote: post #9
blablabla123 wrote: post #4

Earlier quoted context omitted.

On the other hand it seems there's also a lack of testing subjects. It's already frequently pointed out how medical results might not represent everyone. I would also assume that e.g. pharmaceutical certification processes do apply more sophisticated statistics.

The ugly truth is that lots of "science" being done today isn't actually science – it's a performance art that superficially imitates certain behaviors that are associated with real science. And how could it be otherwise? There are nearly 10 million scientists in the world right now. And all of them are pushing out papers as fast as humanly possible. There isn't anywhere near enough statistical brainpower available t…

Bayes theorem for laymen? I read an AI book and it talked about how you can predict the cause from the effects given certain probabilities.

Re: I don't use Bayes factors in my research (2019)

#66

Post by travisjungroth »

kgwgk wrote: post #63
travisjungroth wrote: post #62

Earlier quoted context omitted.

Mean and standard deviation up front would mean that you set your null as a normal distribution with a mean of 0 and a standard deviation of 2%, for example. This is different from saying that you’ll just say it’s normal, and take the mean and standard deviation from the sample. I mean, I could be wrong on this. You could do that if you want. I just think of Bayes factors as a competition and it needs to be “fair”. S…

Ok. As you said "Bayes factors work with comparing models." I don't think that there is a concept of "fairness" that prevents us from including in the comparison a very simple model. "There is no null model. What, 0% effect?" For example. But even if the true effect is zero you may get a non-zero observation because measurements are not perfect. If what you measure is precisely what you want to know there is no need…

As far as I see it, you have three options:

Pick a measure. That’s what I mean by “effect is 0%”. It’s a straw man here.

Pick a fully specified model. This is a model that, up front, you could ask what is the probability of event E? For a normal distribution, this would require choosing concrete mean and standard deviation.

Pick an under-specified model. This would be that it’s normal, but you don’t pick the mean and standard deviation. You pull them from the sample. As I’ve described it here, you can’t get P(E) from that.

The expectation from our alternative hypothesis and model is that it’s fully formed before we look at the data. It’s a choice whether you want that to be the case or not with the null model. “Fair” as I’m describing it is that you would pick something.

Re: I don't use Bayes factors in my research (2019)

#67

Post by travisjungroth »

LudwigNagasena wrote: post #64
travisjungroth wrote: post #55

Earlier quoted context omitted.

> > Bayes factors are a method of model comparison. > Are you implying that hypothesis testing isn’t? No.

Then I don't get the meaning of this: > Bayes factors work with comparing models. There is no null model. What, 0% effect? Ok, there was a non-zero effect. That model loses since it put the probability of 0% at 1 and everything else at 0. And if you do anything else, you’re encoding some amount of belief into the model, some judgment you’ve made. > So, you need to pick two models and compare them. I’m not saying this…

> It is the same way with traditional hypothesis testing. You take two models and compare their likehood.

With a Bayes factor you compare the marginal likelihood. You have to account for the weight of the parameters according to the priors. With a likelihood ratio, you pick the best parameters and take the ratio of those likelihoods.

This means a model used in a Bayes factor must be able to make predictions that follow probability axioms. Models in likelihood ratios don’t have this restriction.

I agree likelihood ratios and Bayes factors are similar. They’re also different.

Re: I don't use Bayes factors in my research (2019)

#68

Post by kgwgk »

travisjungroth wrote: post #66
kgwgk wrote: post #63

Earlier quoted context omitted.

Ok. As you said "Bayes factors work with comparing models." I don't think that there is a concept of "fairness" that prevents us from including in the comparison a very simple model. "There is no null model. What, 0% effect?" For example. But even if the true effect is zero you may get a non-zero observation because measurements are not perfect. If what you measure is precisely what you want to know there is no need…

As far as I see it, you have three options: Pick a measure. That’s what I mean by “effect is 0%”. It’s a straw man here. Pick a fully specified model. This is a model that, up front, you could ask what is the probability of event E? For a normal distribution, this would require choosing concrete mean and standard deviation. Pick an under-specified model. This would be that it’s normal, but you don’t pick the mean and…

A> Pick a measure. That’s what I mean by “effect is 0%”. It’s a straw man here.

I don't understand what you mean by "pick a measure" but maybe the "it’s a straw man here" (that I don't really understand either) indicates that looking at the other two options is enough.

B> Pick a fully specified model. This is a model that, up front, you could ask what is the probability of event E? For a normal distribution, this would require choosing concrete mean and standard deviation.

Ok. That seems to describe a simple classical null hypothesis like the example I gave in my previous comment. The underlying thing of interest is zero and the sampling distribution for the data is normally distributed around zero.

C> Pick an under-specified model. This would be that it’s normal, but you don’t pick the mean and standard deviation. You pull them from the sample. As I’ve described it here, you can’t get P(E) from that.

That is not the kind of null hypothesis I gave in my example, I think we can agree on that.

> The expectation from our alternative hypothesis and model is that it’s fully formed before we look at the data.

I don't understand that sentence. What is "it" that is fully formed before we look at the data? The alternative hypothesis and model?

> It’s a choice whether you want that to be the case or not with the null model.

What is "that"? Being formed before we look at the data? (In that case I hope that the null model I described would satisfy that.)

> “Fair” as I’m describing it is that you would pick something.

Pick something of what? I'm completely lost, I'm afraid.

I'm just saying that I can have a null model of the form B like in the example "the underlying thing is zero and the data generated by this model has a probability distribution p(x)=exp(-x^2/2)".

And I can compare that model it with any other model described by a distribution probability for the underlying thing which, taking into account the measurement error, results in a probability distribution p'(x) for the data generated.

Re: I don't use Bayes factors in my research (2019)

#69

Post by tpoacher »

p-values aren't problematic. How people use them is.

Same with bayes factors. I've seen people claim "anything above 3 is significant".

Incidentally, the theory behind p-values is actually beautiful, and p-values can generalise really well in theory, but in practice most people don't know this.

E.g., did you know that you can have "bayesian" p-values? (in the sense that the p-value can be designed to take priors and other models into account, without violating its definition in any way)

Re: I don't use Bayes factors in my research (2019)

#70

Post by travisjungroth »

kgwgk wrote: post #68
travisjungroth wrote: post #66

Earlier quoted context omitted.

As far as I see it, you have three options: Pick a measure. That’s what I mean by “effect is 0%”. It’s a straw man here. Pick a fully specified model. This is a model that, up front, you could ask what is the probability of event E? For a normal distribution, this would require choosing concrete mean and standard deviation. Pick an under-specified model. This would be that it’s normal, but you don’t pick the mean and…

A> Pick a measure. That’s what I mean by “effect is 0%”. It’s a straw man here. I don't understand what you mean by "pick a measure" but maybe the "it’s a straw man here" (that I don't really understand either) indicates that looking at the other two options is enough. B> Pick a fully specified model. This is a model that, up front, you could ask what is the probability of event E? For a normal distribution, this wou…

“Pick a measure” just meant that you’re predicting the difference will be exactly 0%. P(0) = 1.

The difference between a Bayes factor and a likelihood ratio is Bayes factor uses the marginal likelihood. So you need to pick your parameters ahead of time, weighted by priors. With a likelihood ratio, you can use the best parameters given the data.

You can do the likelihood ratio in an objective way, because you’re choosing whatever has the least error given the data. Bayes factor you can’t be totally objective. You need to choose ahead. Upside is it reduces overfitting.

Post reply on HN