P-hacked hypotheses are deceivingly robust (2016) • Hacker News Board

Re: P-hacked hypotheses are deceivingly robust (2016)

#11

Post by knzhou » Sat, Feb 22, 2020, 8:04 AM UTC

lisper wrote: post #6

This problem is going to get more severe as available datasets get bigger and bigger. The more data you have to mine, the more likely you are to find something that looks like a signal but isn't.

At some point, social science must switch to tighter p-value cutoffs and corrections for multiple comparisons. These are the norm in particle physics, which dealt with and resolved exactly this problem back in the 1970s.

Re: P-hacked hypotheses are deceivingly robust (2016)

#12

Post by eganist » Sat, Feb 22, 2020, 9:56 AM UTC

comex wrote: post #10

eganist wrote: post #2

The words "deceivingly" and "deceptively" have the same problem: there's a roughly 50/50 split in polar-opposite interpretations. https://grammarist.com/usage/deceptively/ In this case, does "deceivingly robust" mean they look robust but are fragile? or does it instead mean they look fragile but are robust? This isn't a criticism of you, soundsop. Rather, it's intended to keep pointing at how difficult it can be to c…

Huh. I'm skeptical about that article's dichotomy of "deceptively" meaning either "in appearance but not in reality" or "in reality but not in appearance". I think the most common usage of "deceptively X" is, more broadly, "X in a way that deceives you". That includes "X in reality but not in appearance", but it also includes "X in reality and in appearance, but deceiving you about something else". For example, they…

Mate, I don't know; I'm just going with all the research and linguistic warnings I've read on the word.

https://languagelog.ldc.upenn.edu/nll/?p=3500

https://brians.wsu.edu/2016/05/25/deceptively/

https://www.academia.edu/37488247/The_Deceptively_Simple_Pro...

Shoot, even oxford gives exactly opposite definitions of the word.

https://www.oxfordlearnersdictionaries.com/us/definition/eng...

---

I mean, even when I saw the title of the thread, I had an obligation to click (clickbait I guess?) because I could have interpreted the title as either a warning about p-hacking or an attestation in favor of the practice. In fact, on first glance, I read the title as "P-hacked hypotheses appear less robust than they actually are."

Re: P-hacked hypotheses are deceivingly robust (2016)

#13

Post by ncmncm » Sat, Feb 22, 2020, 5:03 PM UTC

P-hacking is a fine way to winnow through ideas to see what might be interesting to follow up on. There will certainly be false positives, but the real positives will usually be in there, too, if there are any. Determining which is which takes more work, but you need guidance on where to apply that work.

To insist that p-hacking, by itself, implies pseudo-science is fetishism. There is no substitute for understanding what you are doing and why.

Re: P-hacked hypotheses are deceivingly robust (2016)

#14

Post by ebg13 » Sat, Feb 22, 2020, 9:58 PM UTC

eganist wrote: post #2

The words "deceivingly" and "deceptively" have the same problem: there's a roughly 50/50 split in polar-opposite interpretations. https://grammarist.com/usage/deceptively/ In this case, does "deceivingly robust" mean they look robust but are fragile? or does it instead mean they look fragile but are robust? This isn't a criticism of you, soundsop. Rather, it's intended to keep pointing at how difficult it can be to c…

The problem in the title is not the word "deceivingly" but the word "are". It should say "appear" or "seem". The reason is the common understanding of the word "robust" to mean the strong end of the spectrum from weak to strong and not the entire spectrum itself. The only way for the title to work is if you use the other (wrong) meaning of "robust".