111–120 of 130 posts
Earlier quoted context omitted.
There are different approaches a Bayesian might take. The one that I described is certainly among them, though it is not the only one.
I think the word "naive" is problematic here. Have you seen instances where Bayesians choose a prior that isn't at least somewhat informed by exploratory analysis?
The article submitted here leads to the American Statistical Association statement on the meaning of p values,[1] the first such methodological statement ever formally issued by the association. It's free to read and download. The statement summarizes into these main points, with further explanation in the text of the statement. "What is a p-value? "Informally, a p-value is the probability under a specified statistic…
> "3. Scientific conclusions and business or policy decisions should not be based only on whether a p-value passes a specific threshold. There is no lower threshold at which the data becomes non-predictive?
Earlier quoted context omitted.
> That feels like a bigger problem to me than people actually getting good data but then using it too confidently P value alone does not tell you whether you have good data. It just tells you how well a particular model fits the data that you have. It won't (and can't) tell you if your data set is missing data that would alter the P value were it to be included. P value alone is not enough to say "good" or "bad." Tha…
I'm not sure I agree. If P-value is really low then it implies either the existing data set is bad or your new data set is bad. Either way you haven't used data to get to a confident decision. Put another way: tons of stuff in prod dev gets done without p-values entirely because it's the first time you have any data at all about something. I'm questioning whether this is hugely valuable and obviously better than usin…
Earlier quoted context omitted.
1 and 3 are really the same thing. If the complaint is that the frequentist test can't tell you anything if the assumed distribution wasn't the right one (which is what's happening if you would have done something different), consider the bayesian case. There one might argue you at least still have the probability of each hypothesis given the data. But that forgets that it is only the probability of each hypothesis g…
We are clearly not on the same page. Because I think that 1 and 3 are rather different things, and you don't. In particular 1 consists of exact statements about the likelihood of 7 births in a row being mmmmmmf. By contrast 3 consists of statements about what Bill and Lorena's childbearing plans would have been if something different had happened. Those are very different types of statement. There is no connection be…
For Bayes' theorem, we need a theory of how the data is produced given the parameter of interest. Bill and Lorena's plans certainly influence what data I observe: in scenario two, I can never observe the data BBBBGGG, but in scenario one, I can. My point is that your first category is not "exact statements about the likelihood of 7 births in a row being mmmmmmmf", it is "exact statements about the likelihood of observing mmmmmmf", which is, in fact, quite different, if you admit the possibility of Bill and Lorena having particular childbearing plans.
I agree as well! Here is what probability theory teaches us. The proper role of data is to adjust our prior beliefs about probabilities to posterior beliefs through Bayes' theorem. The challenge is how to best communicate this result to people who may have had a wide range of prior beliefs. p-values capture a degree of surprise in the result. Naively, a surprising result should catch our attention and cause us to ret…
Simple Bayesian approaches take the opposite approach. You generally start with some relatively naive prior, and then treat the posterior as being the conclusion. Which is not very realistic if the real prior was something quite different. I don't think this is a completely accurate portrayal of Bayesian stats. In Bayesian stats, there is no "real prior". Probability distributions are all subjective representations o…
Earlier quoted context omitted.
You are also normalising the difference by the variance, so the t-statistic has no units.
The claim the significance has nothing to do with the magnitude of a difference is just wrong though. It clearly does, this information is just merged with other information about the variance and sample size to get the p-value, which is compared to a threshold to get significance.
Depending on the study, finding a delta whose p-value is significant does not necessarily mean that the size of the effect (i.e., delta) might be significant enough to be useful.
Earlier quoted context omitted.
> That feels like a bigger problem to me than people actually getting good data but then using it too confidently P value alone does not tell you whether you have good data. It just tells you how well a particular model fits the data that you have. It won't (and can't) tell you if your data set is missing data that would alter the P value were it to be included. P value alone is not enough to say "good" or "bad." Tha…
I'm not sure I agree. If P-value is really low then it implies either the existing data set is bad or your new data set is bad. Either way you haven't used data to get to a confident decision. Put another way: tons of stuff in prod dev gets done without p-values entirely because it's the first time you have any data at all about something. I'm questioning whether this is hugely valuable and obviously better than usin…
The article submitted here leads to the American Statistical Association statement on the meaning of p values,[1] the first such methodological statement ever formally issued by the association. It's free to read and download. The statement summarizes into these main points, with further explanation in the text of the statement. "What is a p-value? "Informally, a p-value is the probability under a specified statistic…
Sure, but is there not a significant correlation between the two in practice? Or would you trust something that gives a 1% p-value equally as one that gives a 99% p-value?
(Yes, I realize it's easy to construct counterexamples, hence why I asked "in practice".)
It's not just p-values. Some people just don't understand even very basic statistics. I remember talking to one person in marketing who ran surveys of the company's users. They would send out a survey to all registered users, get back responses from 1% of them or something, and then proceed to report findings based on the responses. They were really happy, since a 1% response rate is great for surveys like this. I tr…
So, how does one exploit this apparent bad practice? I.e., how does one make a profit from others making this mistake? The answer is, of course, one doesn't - otherwise others would've done so. So what does this tell us about the state of affairs? Is it that bad sampling doesn't matter for practical purposes, or that marketing research is useless? I don't know, and I'm not trying to be belligerent here. A situation similar to this shows up in dozens of places every day - more often than not, it doesn't matter if things are done 'right', there are large margins within which 'good enough' is indistinguishable from 'right'.
This bothers me greatly, but it's hard to argue against this conclusion, empirically. How to deal with this cognitive dissonance? I mean this is the exact topic at least half of the blog posts that make it to the HN front page deal are fundamentally about.