Earlier quoted context omitted.
What is the reference? I don't get it.
I could swear one of his research projects involved asking people to make a mug and pricing it out after they finished. But I guess it was Kahneman that researched mugs? Whoops
I don't use Bayes factors in my research (2019)
51–60 of 78 posts
Re: I don't use Bayes factors in my research (2019)
#52Earlier quoted context omitted.
I think there is some confusion going on. Nobody claims that there is the null hypothesis. I think you are fighting windmills. Let’s say you study P. You know that P belongs to the family 𝒫. For example, 𝒫 = {N(μ,σ²): μ∈ℝ, σ²>0}. To be aware of 𝒫 is a prerequisite to do any sort of testing. For example, to test H0: μ=0 vs H1: μ≠0. After all, a typical hypothesis test is just a likehood ratio test. What you don’t h…
> Nobody claims that there is the null hypothesis. Google “the null hypothesis”. If you mean null model, then I’m not fighting against anyone. We all agree which null model to use is a choice to be made. Otherwise, I’m not even sure what you’re trying to convince me of at this point. I’ll restate the essence of my first comment more concisely. Bayes factors are a method of model comparison. You take the ratio of marg…
> Google “the null hypothesis”.
> If you mean null model, then I’m not fighting against anyone. We all agree which null model to use is a choice to be made.
I don’t understand what difference you are trying to imply by drawing a distinction between a null model and a null hypothesis.
> Otherwise, I’m not even sure what you’re trying to convince me of at this point. I’ll restate the essence of my first comment more concisely.
I will try to make it as clear as possible.
> Bayes factors are a method of model comparison.
Are you implying that hypothesis testing isn’t? That’s just false. And I’ve explained why.
> You take the ratio of marginal likelihoods for two models given the data. Choosing a null model for this purpose requires more assumptions than doing null hypothesis testing with frequentist statistics.
And in frequentist statistics you just calculate likehood because you can’t integrate over your model probabilities to get marginal likehood because you don’t assume your models to have a probability of being true. That’s the only extra assumption you have in Bayesian statistics. Everything else is the same. If you are saying that there are some other extra assumptions, that’s just false as I’ve explained in my previous comments. There are no extra assumptions for a “null model” beyond putting a prior on it.
> Mixing the schools of thought of Bayesian and frequentist makes things more confusing than operating within them individually. Bayes factors have other uses than null hypothesis testing.
There is no any confusing “mixing”. It’s just statistical decision theory. In the frequentists approach you calculate the risk of your decision rule for each model and call it a day. In the Bayesian approach you go one step further and average your risks using your priors to get the “total” Bayes risk.
Both approaches have uses other than null hypothesis testing. Null hypothesis testing is just a particular case of a decision problem with a 0-1 loss function. The loss is 0 if you have chosen the correct hypothesis and it is 1 if you have encountered type I or type II error.
Re: I don't use Bayes factors in my research (2019)
#53Earlier quoted context omitted.
> Choosing a null model for this purpose requires more assumptions than doing null hypothesis testing with frequentist statistics. How so? I could choose the same null model that predicts that the observation is distributed as, say, a standard Gaussian. What additional assumptions are required?
Mean and standard deviation up front, not from the sample. At least, that would go against my understanding of Bayes factors and how I’ve calculated them. You can do other stats, t-test for example, without declaring that up front.
I don’t understand what you say, sorry.
If my choice of null model is p(x)=exp(-x^2/2) and I get some observation xobs I can do frequentist things with it - like calculating the p-value p(|x|>|xobs|) for example - and I can compare it with some alternative model p’(x) using the Bayes factor p(xobs)/p’(xobs).
What does “Mean and standard deviation up front, not from the sample.” mean in this context?
Re: I don't use Bayes factors in my research (2019)
#54Earlier quoted context omitted.
P-values work great when they’re super low, experiments run at a human-scale frequency, and hypotheses are extremely precise in their predictions, e.g. some physics. If you run an experiment a day and get p interpretation of p-values approximately don’t matter. Running social sciences experiments with p < 0.05 threshold is where things get weird.
Did you read the article? >even your interpretation of p-values approximately don’t matter "Small number means good" is not a sufficient working understanding of p-values for doing science.
But what if it’s really small?
Re: I don't use Bayes factors in my research (2019)
#55Earlier quoted context omitted.
> Nobody claims that there is the null hypothesis. Google “the null hypothesis”. If you mean null model, then I’m not fighting against anyone. We all agree which null model to use is a choice to be made. Otherwise, I’m not even sure what you’re trying to convince me of at this point. I’ll restate the essence of my first comment more concisely. Bayes factors are a method of model comparison. You take the ratio of marg…
> Google “the null hypothesis”. > If you mean null model, then I’m not fighting against anyone. We all agree which null model to use is a choice to be made. I don’t understand what difference you are trying to imply by drawing a distinction between a null model and a null hypothesis. > Otherwise, I’m not even sure what you’re trying to convince me of at this point. I’ll restate the essence of my first comment more co…
> > Bayes factors are a method of model comparison.
> Are you implying that hypothesis testing isn’t?
No.
Re: I don't use Bayes factors in my research (2019)
#56Earlier quoted context omitted.
Did you read the article? >even your interpretation of p-values approximately don’t matter "Small number means good" is not a sufficient working understanding of p-values for doing science.
But what if it’s really small?
It's completely irrelevant if you don't understand how to interpret it. It is not a number which tells you how correct your hypothesis is.
That is literally mindless statistics. Which coincidentally is the name of the article I talked about. Did you read it?
(In a social science, if your p-value is 1E-5 or something, the most likely interpretation is that you are doing something very wrong)
Re: I don't use Bayes factors in my research (2019)
#57Earlier quoted context omitted.
But what if it’s really small?
It's completely irrelevant if you don't understand how to interpret it. It is not a number which tells you how correct your hypothesis is. That is literally mindless statistics. Which coincidentally is the name of the article I talked about. Did you read it? (In a social science, if your p-value is 1E-5 or something, the most likely interpretation is that you are doing something very wrong)
I did. My statement was hyperbolic. More directly: P-values are more resilient to misuse at their extremes.
Re: I don't use Bayes factors in my research (2019)
#58Earlier quoted context omitted.
You've sliced up what I've said to the point it doesn't really make sense. This is exactly what I said was confusing. I'm talking about Bayes factors and you're talking about null hypothesis testing. I'll just answer your question, why there can't be a null model. You can have a hypothesis that represents all differences between groups are due to chance. To make this a statistical model, something that can calculate…
I think there is some confusion going on. Nobody claims that there is the null hypothesis. I think you are fighting windmills. Let’s say you study P. You know that P belongs to the family 𝒫. For example, 𝒫 = {N(μ,σ²): μ∈ℝ, σ²>0}. To be aware of 𝒫 is a prerequisite to do any sort of testing. For example, to test H0: μ=0 vs H1: μ≠0. After all, a typical hypothesis test is just a likehood ratio test. What you don’t h…
I think this approach to non-Bayesian hypothesis testing is dangerous or misleading.
> Let’s say you study P. You know that P belongs to the family 𝒫. For example, 𝒫 = {N(μ,σ²): μ∈ℝ, σ²>0}. To be aware of 𝒫 is a prerequisite to do any sort of testing.
Okay, sure: you’ve decided on a family of probability distributions. It’s surely an approximation (very few things you would test are actually Gaussian — for one thing, the negative tail often makes no sense).
> For example, to test H0: μ=0 vs H1: μ≠0. After all, a typical hypothesis test is just a likehood ratio test.
This is indeed the usual formulation.
> What you don’t have to know or even to assume existence of—if you don’t do Bayesian stuff—is a probability measure Π on 𝒫 (and its appropriate sigma-field). … But you have to have well-defined 𝒫 either way.
If 𝒫 is well defined, then there is some probability that H0 is true. But H0 occupies a lower-dimensional space than 𝒫 — it’s a measure-zero subset. Most probability measures 𝒫 (and all measures that are continuous on their parameters) give zero probability to H0. So (in Bayesian terms) H0 is a priori wrong w.p. 1. And in non-Bayesian terms, you’re calculating the likelihood of your measurements under two competing hypotheses, one of which is correct w.p. 0 even conditioned on one of the two hypotheses being correct.
And this results in what I consider to be useless headline results:
“This intervention has an effect with significance 0.02” — great, of course it has an effect. What is the effect? Can you say anything intelligent about effect size? Did you even try?
“We did not find significant evidence that some intervention causes some undesirable effect” — great, but that’s actually a statement about your trial and conveys essentially no information about whether the effect is there. I can do a study with n=1 and fail to find significant evidence of anything! But I also learn nothing! Why didn’t you either (a) come up with an actual reasonable hypothesis and test that or (b) put some confidence bounds on the size of the undesirable effect.
And you can do (b) without a Bayesian prior as long as it choose your hypothesis well. “Our data is inconsistent with the intervention causing the undesired effect in more than 0.001% of cases” with some clarification as to what “inconsistent” means.
Re: I don't use Bayes factors in my research (2019)
#59Earlier quoted context omitted.
It's completely irrelevant if you don't understand how to interpret it. It is not a number which tells you how correct your hypothesis is. That is literally mindless statistics. Which coincidentally is the name of the article I talked about. Did you read it? (In a social science, if your p-value is 1E-5 or something, the most likely interpretation is that you are doing something very wrong)
I did. My statement was hyperbolic. More directly: P-values are more resilient to misuse at their extremes.
ok
Re: I don't use Bayes factors in my research (2019)
#60Earlier quoted context omitted.
I think there is some confusion going on. Nobody claims that there is the null hypothesis. I think you are fighting windmills. Let’s say you study P. You know that P belongs to the family 𝒫. For example, 𝒫 = {N(μ,σ²): μ∈ℝ, σ²>0}. To be aware of 𝒫 is a prerequisite to do any sort of testing. For example, to test H0: μ=0 vs H1: μ≠0. After all, a typical hypothesis test is just a likehood ratio test. What you don’t h…
I think this approach to non-Bayesian hypothesis testing is dangerous or misleading. > Let’s say you study P. You know that P belongs to the family 𝒫. For example, 𝒫 = {N(μ,σ²): μ∈ℝ, σ²>0}. To be aware of 𝒫 is a prerequisite to do any sort of testing. Okay, sure: you’ve decided on a family of probability distributions. It’s surely an approximation (very few things you would test are actually Gaussian — for one thi…
> Most probability measures 𝒫 (and all measures that are continuous on their parameters) give zero probability to H0.
And some don't.
> So (in Bayesian terms) H0 is a priori wrong w.p. 1.
Or not.
> And in non-Bayesian terms, you’re calculating the likelihood of your measurements under two competing hypotheses, one of which is correct w.p. 0 even conditioned on one of the two hypotheses being correct.
If one of the two (H0 or H1) is correct and you don't know which one then it could be either... Of course is you knew a priori which one is correct you wouldn't be considering the other at all.