Hack Your Way to Scientific Glory (2016)
projects.fivethirtyeight.com
Hack Your Way to Scientific Glory (2016)
1–10 of 12 posts
Re: Hack Your Way to Scientific Glory (2016)
#2For anyone wondering about the title, it's about p-hacking - picking and choosing variables to suit your objectives. Highly recommend trying the mini-game, it's fun!
Re: Hack Your Way to Scientific Glory (2016)
#3They say a photo is worth a thousand words, and here, this demo is as convincing as a lot of the spilt ink regarding the replication crisis and social psychology. As someone who pursuing a PhD in the social sciences... This could be a useful tool for seniors majoring in the social sciences / who are taking stats classes.
Re: Hack Your Way to Scientific Glory (2016)
#4Sigh. P hacking has long been a meme. Nicholas Cage and shark attacks for example. P values are useful though if the results seem at least plausible or can be explained by some underlying process. Like a low p value between calorie restriction and weight loss.
Re: Hack Your Way to Scientific Glory (2016)
#5I know the exercise was to p-hack, but instead I decided to one-shot my attempt at the most reasonable model from first principals:
- given that we are looking at a national scale, use only national politicians
- use the components from Macroeconomics 101: exclude inflation as that’s on the Fed, exclude stocks as too conflated with FX and international investing alternatives
- don’t needlessly withhold data
Tried one hypothesis, so p-value of 0.04 is accurate. Still OK to explore if you Bonferroni correct the p-Val afterwards
Re: Hack Your Way to Scientific Glory (2016)
#6Sigh. P hacking has long been a meme. Nicholas Cage and shark attacks for example. P values are useful though if the results seem at least plausible or can be explained by some underlying process. Like a low p value between calorie restriction and weight loss.
Shark attacks isn't really p-hacking though, that's about misinterpreting/misrepresenting a real correlation. P-hacking is finding a fake correlation due to chance because you tested a lot of different things in a smallish dataset, and then reporting that correlation as meaningful without mentioning your negative results or properly explaining your entire process.
Re: Hack Your Way to Scientific Glory (2016)
#7An excellent demonstration of how easy it is to manipulate data to find significant results - it brings to light (again) the pervasive issue of data dredging in research, where researchers, intentionally or unintentionally, keep testing hypotheses until they find something publishable, thereby undermining the integrity of scientific findings and highlighting the importance of preregistration and replication studies.
How can we shift the academic incentives away from "publish or perish" toward promoting transparency and rigorous methodology in research? Are any of the current attempts moving the needle?
Re: Hack Your Way to Scientific Glory (2016)
#8An excellent demonstration of how easy it is to manipulate data to find significant results - it brings to light (again) the pervasive issue of data dredging in research, where researchers, intentionally or unintentionally, keep testing hypotheses until they find something publishable, thereby undermining the integrity of scientific findings and highlighting the importance of preregistration and replication studies.…
Current attempts showing promise include initiatives like the Center for Open Science and journals such as eLife and PLOS ONE that emphasize methodological rigor over impact factor. Additionally, policies from organizations like the National Institutes of Health (NIH) mandating data sharing plans are gradually moving the needle toward these goals.
Re: Hack Your Way to Scientific Glory (2016)
#9An excellent demonstration of how easy it is to manipulate data to find significant results - it brings to light (again) the pervasive issue of data dredging in research, where researchers, intentionally or unintentionally, keep testing hypotheses until they find something publishable, thereby undermining the integrity of scientific findings and highlighting the importance of preregistration and replication studies.…
Current attempts showing promise include initiatives like the Center for Open Science and journals such as eLife and PLOS ONE that emphasize methodological rigor over impact factor. Additionally, policies from organizations like the National Institutes of Health (NIH) mandating data sharing plans are gradually moving the needle toward these goals.
eLife is very rigorous, but also very selective. Most submissions don't pass editorial consideration.
I've found their reviews much more detailed and technical than Nature Medicine or Cell.
Re: Hack Your Way to Scientific Glory (2016)
#10Earlier quoted context omitted.
Current attempts showing promise include initiatives like the Center for Open Science and journals such as eLife and PLOS ONE that emphasize methodological rigor over impact factor. Additionally, policies from organizations like the National Institutes of Health (NIH) mandating data sharing plans are gradually moving the needle toward these goals.
eLife is very rigorous, but also very selective. Most submissions don't pass editorial consideration. I've found their reviews much more detailed and technical than Nature Medicine or Cell.
When you say that eLife is "very rigorous," could you elaborate on what specific aspects of their review process or criteria you find particularly stringent, and how those compare to the review processes of Nature/Cell?