31–40 of 106 posts
ELI5... what’s a p-hack?
“the misuse of data analysis to find patterns in data that can be presented as statistically significant when in fact there is no real underlying effect. This is done by performing many statistical tests on the data and only paying attention to those that come back with significant results, instead of stating a single hypothesis about an underlying effect before the analysis and then conducting a single test for it.“…
I was the only one there who knew he's a particle physicist.
The OP is horrifyingly right.
Earlier quoted context omitted.
This. I can p-hack in my field, if I wanted to, up to a p-value of arbitrary strictness, given enough time.
> This is wrong. P hacking has nothing to do with the p being too lenient. > This. I can p-hack in my field, if I wanted to, up to a p-value of arbitrary strictness, given enough time. I'm not a practicing scientist/academic, so I want to be careful here. But, I think both of you are being a little uncharitable/pedantic. P-hacking is one contributor to the broader reproducibility crisis. Lowering the p-value to addre…
Earlier quoted context omitted.
I'm not being nitpicky - they are components of a problem of reproducibility, but orthogonal to each other. Bad UI design and a poor backend are both reasons "X website sucks!" but that doesn't mean they're the same. A perfectly designed, un-p-hacked study should still perhaps be held to a stricter p-value criteria than 0.05. And I am correct - because I've done it. Presently working on a paper where, because I prima…
First off, I don’t have any experience with publishing based off the results of simulations. My (short) time in writing papers centered around economics research with observational datasets. I can tell you that given a fixed size dataset, it is not possible to p-hack below a certain threshold in any meaningful way. My colleagues would’ve asked why the gigantic dataset we purchased had 1/4 of its observations thrown o…
>I can tell you that given a fixed size dataset, it is not possible to p-hack below a certain threshold in any meaningful way
Correct, but the most common methods of p-hacking involve changing the dataset size, either by repeating the experiment until the desired result is achieved (a la xkcd [0]), or by removing a large part of the dataset due to a seemingly-legitimate excuse (like the fivethirtyeight demo that has been linked already).
Pre-announcing your dataset size is pre-announcing your sample size. If you pre-announce your dataset, p-hacking is not possible. This is true. But most research doesn't use a public dataset that is pre-decided.
>Would it take you more time to p-hack a lower threshold
Yes.
>In simulation based research like yours, it seems to me that even other p-hacking “fixes” like forcing there to be a pre-announcement or sample size, sample structure, etc.
This doesn't follow.
Earlier quoted context omitted.
First off, I don’t have any experience with publishing based off the results of simulations. My (short) time in writing papers centered around economics research with observational datasets. I can tell you that given a fixed size dataset, it is not possible to p-hack below a certain threshold in any meaningful way. My colleagues would’ve asked why the gigantic dataset we purchased had 1/4 of its observations thrown o…
Not the above poster, but... >I can tell you that given a fixed size dataset, it is not possible to p-hack below a certain threshold in any meaningful way Correct, but the most common methods of p-hacking involve changing the dataset size, either by repeating the experiment until the desired result is achieved (a la xkcd [0]), or by removing a large part of the dataset due to a seemingly-legitimate excuse (like the f…
E.g. if I say “I will do 10000 runs of my simulation”, what’s to prevent me from doing those runs multiple times, and selecting the one that gives me the desired p-value? For observational research, there’s obviously a physical limit to how many subjects you can observe etc. Would still love an answer from the grandparent comment.
Hi, founder of VWO here. We revamped our testing engine to Bayesian in 2015 to prevent the ‘peeking problem’ with frequentists approaches. You can read about our approach https://vwo.com/blog/smartstats-testing-for-truth/
Optimizely being for large enterprises, curious how people do A/B tests at their respective startups. Do most roll their own? How do you make sure your science is sound?
Earlier quoted context omitted.
The parent asked very straightforwardly what is p hacking and you replied with a red herring. If I'm being pedantic, you're being unhelpful.
> clever experimental design and cherry picking can generate many results that are statistically significant at that level. I’m not sure what’s incorrect about this statement? If you disagree with the “fix” to the problem that is most familiar to me, that’s fine. It’s one of many approaches. But, at what point did I mislead the parent as to what p-hacking is? What’s your definition?
>p-hacking is a set of related techniques, whereby clever experimental design and cherry picking of data can generate results that falsely appear statistically significant.
There are a few important differences here:
1. The effect is not statistically significant. In fact often, there is no effect at all.
2. There is no mention of a specific significance level.
Those are both important.
Once at a programming conference, I was talking with a very senior developer at a well known company. He was going on and on about their A/B testing efforts. I asked how they decided how long they would run an experiment for. The answer was "until we get a significant result." I was shocked then, but now I am used to getting these kinds of responses from developers ... That and a belief that false positives are not a…