p-Hacking and False Discovery in A/B Testing
papers.ssrn.com
p-Hacking and False Discovery in A/B Testing
1–10 of 106 posts
Re: p-Hacking and False Discovery in A/B Testing
#2ELI5... what’s a p-hack?
Re: p-Hacking and False Discovery in A/B Testing
#3ELI5... what’s a p-hack?
Doing the same experiment again and again until you get the answer you want.
Re: p-Hacking and False Discovery in A/B Testing
#4ELI5... what’s a p-hack?
Not an easy ELI5, but consider these resources.
Re: p-Hacking and False Discovery in A/B Testing
#5ELI5... what’s a p-hack?
“the misuse of data analysis to find patterns in data that can be presented as statistically significant when in fact there is no real underlying effect. This is done by performing many statistical tests on the data and only paying attention to those that come back with significant results, instead of stating a single hypothesis about an underlying effect before the analysis and then conducting a single test for it.“
https://en.m.wikipedia.org/wiki/Data_dredging
And here’s a great example of real life P-Hacking to get a catchy article about the health benefits of chocolate:
https://io9.gizmodo.com/i-fooled-millions-into-thinking-choc...
Re: p-Hacking and False Discovery in A/B Testing
#6ELI5... what’s a p-hack?
In statistics, a p-value [1] is a the odds of getting a result by chance if there's not actually an effect from the thing you're testing. A common threshold to use is .05, which means that you'd expect the given result less than 5% of the time by chance. Typically this indicates a relationship, but if you simultaneously test enough things that aren't causally linked you'd expect some to have p[1] https://en.m.wikipedia.org/wiki/P-value
[2] https://xkcd.com/882/
Re: p-Hacking and False Discovery in A/B Testing
#7ELI5... what’s a p-hack?
Re: p-Hacking and False Discovery in A/B Testing
#8Optimizely being for large enterprises, curious how people do A/B tests at their respective startups. Do most roll their own? How do you make sure your science is sound?
Re: p-Hacking and False Discovery in A/B Testing
#9Re: p-Hacking and False Discovery in A/B Testing
#10How can A/B testing tools be improved to prevent p-value hacking?
Could it be as simple as declaring your test duration before starting the experiment, and having the tool add an asterisk to your results if you stop the experiment early?