If you think psychological science is bad, imagine how bad it was in 1999 • Hacker News Board

Re: If you think psychological science is bad, imagine how bad it was in 1999

#131

Post by derbOac » Wed, Jun 16, 2021, 6:55 PM UTC

pacbard wrote: post #70

m12k wrote: post #15

When you think about it, it's insane that metascience isn't studied more. We're going to question every little detail about the universe, but we're just going to take on faith that peer-review and publication in journals is an effective method for weeding out "bad" science?

I saw a chart once that made the point that most research studies can be classified along two axes: rigorousness of methods and popularity of results. Most published papers have either high rigor/low popularity and low rigor/high popularity. The trade-off is in the fact that highly rigorous studies only allow for narrow, unexciting results, while popular studies with flashy results will have to compromise on their ri…

I agree with you mostly, but it's worth noting that modern meta-science had its blossoming in psychology in the 1960s, with the development of meta-analysis (with educational psychology and clinical psychology). Technically the origins are much earlier, in the 30s(?) in statistics, but as a field I think it took off around that time, and spread.

Similarly, the replication crisis was being discussed in a lot of areas, especially in psychology, throughout this time, but was largely ignored until after the Bem ESP study. Registered replications aren't new, nor is concern about meta-science; it's just had renewed focus in recent years for various reasons.

It's not all that surprising to me that meta-science is associated with psychology. After all, not only is psychology often sort of fuzzy (by necessity of its subject matter), but it's the science of human behavior, which I think can lay claim to scientist behavior as well.

I think it's arguably the greatest contribution of psychology to the sciences in general.

Re: If you think psychological science is bad, imagine how bad it was in 1999

#132

Post by rijoja » Wed, Jun 16, 2021, 6:55 PM UTC

munificent wrote: post #24

m12k wrote: post #15

When you think about it, it's insane that metascience isn't studied more. We're going to question every little detail about the universe, but we're just going to take on faith that peer-review and publication in journals is an effective method for weeding out "bad" science?

> it's insane that metascience isn't studied more. It is, probably more than is apparent. See: https://en.wikipedia.org/wiki/Philosophy_of_science https://en.wikipedia.org/wiki/Sociology_of_scientific_knowle... There is an interesting recursive problem here, though: what tools do you use to scientifically analyze the scientific process? Whatever tools you use will themselves be hobbled by the same systemic flaws you…

Yeah so it is not science but rather philosophy and it is called Epistemology and not metascience. Even though I suppose metascience would be an apt description in a way.

Re: If you think psychological science is bad, imagine how bad it was in 1999

#133

Post by akiselev » Wed, Jun 16, 2021, 6:56 PM UTC

ImprobableTruth wrote: post #106

jhap wrote: post #37

Earlier quoted context omitted.

> In a sense we do have this: engineering and finance. Engineering turns good hard science into new tools, machines and weapons, and Finance turns good (predictive) soft science into new ways to make money. I think this is a common critique, but I also think it is missing the point. What if the question of interest isn't so easily verifiable like in Engineering? Do we just throw up our hands and give up on those ques…

If there's "good social science" that can't be used to make predictions, what differentiates it from bad social science?

Its utility [1]. The social sciences study a lot of things that people in group A intuitively understand that group B can be completely ignorant of - say, for example, how to navigate a complex social structure like office politics in a modern workplace. Making any sort of predictions about intangible outcomes where the Hawthorne effect is in full effect is pretty much impossible since group A will respond to the new knowledge gained by group B, in effect changing the system we're trying to predict. Individual's psychologies respond to the changing psychology of the group in nondeterministic ways (at least, relative to our ability to collect data on input variables and internal state).

We can bikeshed what makes something a "science" till the cows come home but the philosophy of science and epistemology were not settled with Bacon and Popper - the end goal has always understanding in the broadest sense. Those studies have value as long as they help someone make sense of and adapt to the social systems they're in. It does mean though that those studies should be approached with extreme caution (see the decades wasted on string theory) and anyone basing their research off past results needs to carefully validate their assumptions.

[1] I think in this case "predictive" as a scientific term of art is too restricting. Social sciences often deal with very personal interactions that appear nondeterministic at the scale of a society but are relatively predictable when applied to a stereotypical office or school setting.

Re: If you think psychological science is bad, imagine how bad it was in 1999

#134

Post by loup-vaillant » Wed, Jun 16, 2021, 6:59 PM UTC

m12k wrote: post #15

When you think about it, it's insane that metascience isn't studied more. We're going to question every little detail about the universe, but we're just going to take on faith that peer-review and publication in journals is an effective method for weeding out "bad" science?

A good start would be something like Probability Theory: the Logic of Science By E. T. Jaynes: http://www.med.mcgill.ca/epidemiology/hanley/bios601/Gaussia...

Also, the difference between a bad method and a good method, is that the good method makes more accurate, better calibrated predictions (that is, using it makes us better gamblers).

Re: If you think psychological science is bad, imagine how bad it was in 1999

#135

Post by rijoja » Wed, Jun 16, 2021, 7:01 PM UTC

bluetomcat wrote: post #30

m12k wrote: post #15

When you think about it, it's insane that metascience isn't studied more. We're going to question every little detail about the universe, but we're just going to take on faith that peer-review and publication in journals is an effective method for weeding out "bad" science?

"People" sciences like sociology, psychology and economics can make incredibly misleading claims because one experiment over a small sample of people at a certain moment in time might seem to support a claim, while the actual reason for the observed results is a factor which is never taken in consideration. On the other hand, conducting those experiments over wider demographics and in different points in time means t…

Yeah so you have to make a difference between empirical science and science here really. Which Max Weber who was one of the pillars of social sciences stated around one hundred years ago.

"As such, he was a key proponent of methodological anti-positivism, arguing for the study of social action through interpretive (rather than empiricist) methods, based on understanding the purpose and meanings that individuals attach to their own actions."

https://en.wikipedia.org/wiki/Max_Weber

edit: This is then further developed by the so called Frankfurt School as Critical Theory.

Re: If you think psychological science is bad, imagine how bad it was in 1999

#136

Post by munificent » Wed, Jun 16, 2021, 7:01 PM UTC

chromaton wrote: post #32

munificent wrote: post #24

Earlier quoted context omitted.

> it's insane that metascience isn't studied more. It is, probably more than is apparent. See: https://en.wikipedia.org/wiki/Philosophy_of_science https://en.wikipedia.org/wiki/Sociology_of_scientific_knowle... There is an interesting recursive problem here, though: what tools do you use to scientifically analyze the scientific process? Whatever tools you use will themselves be hobbled by the same systemic flaws you…

>what tools do you use to scientifically analyze the scientific process? Iterating on existing systems to see if you can get results to converge and also testing new systems to see if they also result in known good values.

It's fun to realize how deeply ingrained the current scientific process is in our way of thinking.

All of these ideas that you tacitly take for granted are itself mutable parts of the scientific process:

* That iterative improvement and hill-climbing is an effective process for improving results.

* That replication of experiments and convergence is a truth-generating enterprise.

* That truth can be expressed numerically.

* That there are some values that are "known good". By what process? According to whom?

To be clear, I don't disagree with those. However, these rules aren't baked into the firmament of the universe. They are processes we humans have chosen to apply in our social process of reaching conensus on truth. In other words, this list here isn't physics, it's technology.

It's entirely possible to imagine a culture whose truth finding bodies don't take for granted one or more of these rules at all. That culture might be more or less effective (again, according to what metrics?), but it would still be well-defined.

Re: If you think psychological science is bad, imagine how bad it was in 1999

#137

Post by ch4s3 » Wed, Jun 16, 2021, 7:07 PM UTC

bingidingi wrote: post #130

cwkoss wrote: post #129

Earlier quoted context omitted.

I majored in psychology in undergrad. A big part of why I didn't look for a psychology focused job is that the science is all so loose. I'd often learn about two different study-backed phenomena is two different classes that somewhat contradicted each other. Or I'd learn in a subsequent class that a previously taught study has been invalidated in one way or another. Almost everything is measured subjectively, so huge…

Think about how loose medical science used to be (and for how long)! leeches, bloodletting, miasma, ridiculous enemas and all sorts of outright nonsense. We've got a lot more mistakes to make, but social sciences will improve too.

To be fair, no one was using statistics and the scientific method to support bloodletting, or miasma theory.

Re: If you think psychological science is bad, imagine how bad it was in 1999

#138

Post by theptip » Wed, Jun 16, 2021, 7:10 PM UTC

naasking wrote: post #11

Tarsul wrote: post #2

I still don't understand why we "believed" studies that claimed knowledge about typical behavior and subconscious decisions while those studies were only based on a single experiment with limited participants, e.g. Stanford Prison Experiment. These type of studies should ALWAYS be looked at critically and not just because they fail to reproduce but because they are based on a very small sample size in a very discreet…

The problem is not just with "belief", but in the process itself. Non-replicable studies are actually cited more than replicable studies: https://advances.sciencemag.org/content/7/21/eabd1705 Science journalists probably are vulnerable to the same thing influences that lead scientists to do this, except they have even less review on their claims and so they become pop culture sound bites. As for why non-replicable re…

> As for why non-replicable results are cited more, I'd speculate that non-replicable results are often more unintuitive and surprising

This seems the likely explanation; I saw a paper recently that showed that lay people can predict what will replicate with above-chance accuracy[1]. I imagine scientists are even better than lay people at this.

So non-replicable results are almost by definition surprising (i.e. they are hypotheses that don't match our current model of how the world works), and surprising results are definitely better news than unsurprising results.

[1](https://journals.sagepub.com/doi/full/10.1177/25152459209196...)

Re: If you think psychological science is bad, imagine how bad it was in 1999

#139

Post by colechristensen » Wed, Jun 16, 2021, 7:11 PM UTC

publicola1990 wrote: post #26

Much of such dubious "science" is packaged by self help authours as miracle principles which are going to set ones life right. Popular books like "Power Of Habit" and Cal Newport's books also seem to fall into this category, and seem to enjoy huge patronage, even in HN.

Most popular "nonfiction" fits into this category, really.

There is a pattern to these books

* Pick a topic

* Decide on a narrative

* Pick a collection of studies to demonstrate that narrative

* Take study conclusions (which are often dubious extrapolations of data) and summarize vaguely adding additional unsupported projections, a handful per chapter

* Publish and promote

It isn't just "self-help" but nearly everything in the nonfiction section that you hear people talking about.

Re: If you think psychological science is bad, imagine how bad it was in 1999

#140

Post by colechristensen » Wed, Jun 16, 2021, 7:12 PM UTC

tern wrote: post #119

publicola1990 wrote: post #26

Much of such dubious "science" is packaged by self help authours as miracle principles which are going to set ones life right. Popular books like "Power Of Habit" and Cal Newport's books also seem to fall into this category, and seem to enjoy huge patronage, even in HN.

Who do you look to for wisdom about how to live your life?

In general, nobody who is trying to sell it to me.