Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

If you think psychological science is bad, imagine how bad it was in 1999

statmodeling.stat.columbia.edu

Post reply on HN
Print view View original

101–110 of 238 posts

Re: If you think psychological science is bad, imagine how bad it was in 1999

#101

Post by Blikkentrekker »

username90 wrote: post #98
Blikkentrekker wrote: post #90

Earlier quoted context omitted.

And it starts with that the terms are colloquial and bereft o an actually hard definition. “colloquial”, “roughly”, ”perceived”. — these are not the terms that definitions are made of. The point is that there is no actual hard distinction between “hard science” and “soft science” but there is a hard distinction between an exact theory, and an inexact theory.

There is no exact science, physics isn't an exact science either. You need softer definitions like "Hard" and "Soft" precisely because there are no exact results in any science we have. And it is fine to use soft definitions for these things since categorising scientific fields doesn't need to be a science. A hard science is where replication is expected to never fail, if replication fails once then the theory is thr…

> There is no exact science, physics isn't an exact science either.

Which is why I said “exact theory”.

> You need softer definitions like "Hard" and "Soft" precisely because there are no exact results in any science we have. And it is fine to use soft definitions for these things since categorising scientific fields doesn't need to be a science.

There are exact results every day, but those are not delimited cleanly by “fields”: a theory is exact or it is not.

> A hard science is where replication is expected to never fail, if replication fails once then the theory is thrown out. Applying that to social science studies would seem ridiculous, no social scientist would want that, so they want their field to be soft.

And this criterion is never mentioned at any point in the Wikipedia article linked.

It also seems a useless definition as replication can always fail due to flukes, and the confidence numbers chosen for replication, typically within 0.05, are very arbitrarily chosen.

Whether it is “replicated” or not is a rather arbitrary delimitation of an arbitrarily picked number, and 5% is certainly not improbably low to begin with.

Re: If you think psychological science is bad, imagine how bad it was in 1999

#102

Post by chimpme »

m12k wrote: post #15

When you think about it, it's insane that metascience isn't studied more. We're going to question every little detail about the universe, but we're just going to take on faith that peer-review and publication in journals is an effective method for weeding out "bad" science?

super good point. it is questioned, but the doubts are mostly ignored

Re: If you think psychological science is bad, imagine how bad it was in 1999

#105

Post by PatentlyDC123 »

Is there a method of flagging papers in scientific journals that have been criticized or refuted, e.g., by later studies or proven inability to replicate the data?

In legal research services, like Lexis Nexis or Westlaw, many cases are "flagged" when a later case or statute reverses, narrows, or otherwise affects the earlier case. This system warns lawyers that they may not be able to cite the flagged case in their current work. Of course legal research services also come with their own issues and costs; some of which are likely associated with this system.

Re: If you think psychological science is bad, imagine how bad it was in 1999

#106

Post by ImprobableTruth »

jhap wrote: post #37
logicchains wrote: post #21

Earlier quoted context omitted.

>When you think about it, it's insane that metascience isn't studied more. We're going to question every little detail about the universe, but we're just going to take on faith that peer-review and publication in journals is an effective method for weeding out "bad" science? In a sense we do have this: engineering and finance. Engineering turns good hard science into new tools, machines and weapons, and Finance turns…

> In a sense we do have this: engineering and finance. Engineering turns good hard science into new tools, machines and weapons, and Finance turns good (predictive) soft science into new ways to make money. I think this is a common critique, but I also think it is missing the point. What if the question of interest isn't so easily verifiable like in Engineering? Do we just throw up our hands and give up on those ques…

If there's "good social science" that can't be used to make predictions, what differentiates it from bad social science?

Re: If you think psychological science is bad, imagine how bad it was in 1999

#107

Post by enkid »

99_00 wrote: post #100
enkid wrote: post #8

Earlier quoted context omitted.

Tangent, but the Stanford Prison Experiment is an awful example of science. It was a researcher who wanted to prove a point and created the conditions to collect the data to prove that point. I hate that it's often the only psychological experiment many people are familiar with.

>the Stanford Prison Experiment is an awful example of science. This may be true, but I don't think the evidence you give supports your assertion. >It was a researcher who wanted to prove a point and created the conditions to collect the data to prove that point "prove a point" is the hypothesis "created the conditions" is the experiment "collect the data to prove that point" is the observation

No, an experiment should not be set up to prove a hypothesis, it should be set up to test a hypothesis. In one the hypothesis is falsifiable, and in the other it's not.

Re: If you think psychological science is bad, imagine how bad it was in 1999

#108

Post by jrochkind1 »

hpoe wrote: post #29
logicchains wrote: post #21

Earlier quoted context omitted.

>When you think about it, it's insane that metascience isn't studied more. We're going to question every little detail about the universe, but we're just going to take on faith that peer-review and publication in journals is an effective method for weeding out "bad" science? In a sense we do have this: engineering and finance. Engineering turns good hard science into new tools, machines and weapons, and Finance turns…

I think this is an important point, ultimately good science will produce, verifiable, testable, actionable results. Until you have that no matter how much math you use, how many lab coats you've got, no matter how many journals you publish in you're just sitting there playing with strings.

Telling when this has happened can be less trivial than you would assume though. People (scientists even) were sure that phrenology produced verifiable, testable, actionable results for a generation or two.

In the long run it usually comes out, but the run can be longer than you think, and you may not be where you think in it with regard to any particular current theory. I wonder what things we know all "know" are proven by science will be dismissed by later generations. (I personally guess a lot of genetics-related stuff will be).

(Note that something doesn't need to be verifiable, reliable, or true to be "actionable". You can act on anything...)

Re: If you think psychological science is bad, imagine how bad it was in 1999

#109

Post by pjmorris »

jrochkind1 wrote: post #104

The realities of science as actually practiced rather complicates the religion of "trust the science!" (which usually actually means "trust the scientists!")

“In God we trust. All others must bring data.” - W. Edwards Deming.

Re: If you think psychological science is bad, imagine how bad it was in 1999

#110

Post by enkid »

mateo411 wrote: post #94
enkid wrote: post #8

Earlier quoted context omitted.

Tangent, but the Stanford Prison Experiment is an awful example of science. It was a researcher who wanted to prove a point and created the conditions to collect the data to prove that point. I hate that it's often the only psychological experiment many people are familiar with.

What about the Milgram experiment? This is also a very well known psychological experiment. Was the science behind the Milgram experiment rigorous?

From what I have read (I'm not a psychologist, so take everything I say with a grain of salt), the Milgram experiment had some major ethical issues and doesn't meet modern standards for statistical evidence. There's also accusations that the data may have been manipulated and the results don't directly support the claim that Milgram was making. For example, a large number of participants may have been aware that the actor wasn't really suffering when they administered the punishment. This would seriously skew the results.
Post reply on HN