P-hacked hypotheses are deceivingly robust (2016)
datacolada.org
P-hacked hypotheses are deceivingly robust (2016)
1–10 of 14 posts
Re: P-hacked hypotheses are deceivingly robust (2016)
#2The words "deceivingly" and "deceptively" have the same problem: there's a roughly 50/50 split in polar-opposite interpretations. https://grammarist.com/usage/deceptively/
In this case, does "deceivingly robust" mean they look robust but are fragile? or does it instead mean they look fragile but are robust?
This isn't a criticism of you, soundsop. Rather, it's intended to keep pointing at how difficult it can be to concisely deliver a message.
---
edit: sounds like the correct interpretation of the title is "P-hacked hypotheses appear more robust than they are."
Re: P-hacked hypotheses are deceivingly robust (2016)
#3The words "deceivingly" and "deceptively" have the same problem: there's a roughly 50/50 split in polar-opposite interpretations. https://grammarist.com/usage/deceptively/ In this case, does "deceivingly robust" mean they look robust but are fragile? or does it instead mean they look fragile but are robust? This isn't a criticism of you, soundsop. Rather, it's intended to keep pointing at how difficult it can be to c…
I think that the correct interpretation is: Statistical robustness tests do not address the problem of p-hacked hypotheses.
Re: P-hacked hypotheses are deceivingly robust (2016)
#4The words "deceivingly" and "deceptively" have the same problem: there's a roughly 50/50 split in polar-opposite interpretations. https://grammarist.com/usage/deceptively/ In this case, does "deceivingly robust" mean they look robust but are fragile? or does it instead mean they look fragile but are robust? This isn't a criticism of you, soundsop. Rather, it's intended to keep pointing at how difficult it can be to c…
I think that the correct interpretation is: Statistical robustness tests do not address the problem of p-hacked hypotheses.
You'd be right, and that's a sentence about halfway down. And should you decide to perform robustness tests on a p-hacked hypothesis, you might be convinced that it's more robust than it is.
Re: P-hacked hypotheses are deceivingly robust (2016)
#5The words "deceivingly" and "deceptively" have the same problem: there's a roughly 50/50 split in polar-opposite interpretations. https://grammarist.com/usage/deceptively/ In this case, does "deceivingly robust" mean they look robust but are fragile? or does it instead mean they look fragile but are robust? This isn't a criticism of you, soundsop. Rather, it's intended to keep pointing at how difficult it can be to c…
Eh, I don't think it's difficult so much as a situation where the writer did not weed out ambiguity for some reason. The fix is simply to use different words, especially since there's nothing particularly technical about this to constrain word choice.
Re: P-hacked hypotheses are deceivingly robust (2016)
#6This problem is going to get more severe as available datasets get bigger and bigger. The more data you have to mine, the more likely you are to find something that looks like a signal but isn't.
Re: P-hacked hypotheses are deceivingly robust (2016)
#7Basically, if you take a p-hacked hypothesis and attempt to use it predictively, it falls apart.
That's kinda ... useful, actually.
It feels like this is sort of the same issue with overfitting in ML. Attempts to use ML results predictively often fail in hilarious ways.
Re: P-hacked hypotheses are deceivingly robust (2016)
#8Basically, if you take a p-hacked hypothesis and attempt to use it predictively , it falls apart. That's kinda ... useful, actually. It feels like this is sort of the same issue with overfitting in ML. Attempts to use ML results predictively often fail in hilarious ways.
Yep, that s also how science works: it predicts the future based on a model. Quick trick to know if it s a science or not: if it has the word science in it it’s not. (Eg social science)
Re: P-hacked hypotheses are deceivingly robust (2016)
#9> Direct replications, testing the same prediction in new studies, are often not feasible with observational data. In experimental psychology it is common to instead run conceptual replications, examining new hypotheses based on the same underlying theory. We should do more of this in non-experimental work. One big advantage is that with rich data sets we can often run conceptual replications on the same data.
I think actually relying on "conceptual replications" in practice is impossible. If the theory is only coincidentally supported by the data, that makes the replication more likely to exceed p The author mentions that problem, but doesn't mention a bigger issue: If you think people are unlikely to publish replications using novel data sets, just imagine how impossibly unlikely it is for people to publish failed replications with the original data set! If you read a "replicated" finding of the same theory using the same data set, you can safely ignore it, because 19 other people probably tried other related "replications" and didn't get them to work.
Re: P-hacked hypotheses are deceivingly robust (2016)
#10The words "deceivingly" and "deceptively" have the same problem: there's a roughly 50/50 split in polar-opposite interpretations. https://grammarist.com/usage/deceptively/ In this case, does "deceivingly robust" mean they look robust but are fragile? or does it instead mean they look fragile but are robust? This isn't a criticism of you, soundsop. Rather, it's intended to keep pointing at how difficult it can be to c…
Huh. I'm skeptical about that article's dichotomy of "deceptively" meaning either "in appearance but not in reality" or "in reality but not in appearance". I think the most common usage of "deceptively X" is, more broadly, "X in a way that deceives you". That includes "X in reality but not in appearance", but it also includes "X in reality and in appearance, but deceiving you about something else".
For example, they used this quote as an example of "in appearance but not in reality":
> It’s no mystery why images of shocking, unremitting violence spring to mind when one hears the deceptively simple term, “D-Day.” [Life]
But the term "D-Day" is simple. It's deceptive because it might wrongly lead you to think the event it refers to is also simple.
Similarly, if something is "deceptively simple-looking", it really is simple-looking; it's just not simple.