Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Depression-Gene Studies Were Built on Shaky Foundations

theatlantic.com

Post reply on HN
Print view View original

81–90 of 91 posts

Re: Depression-Gene Studies Were Built on Shaky Foundations

#81

Post by rossdavidh »

thanatropism wrote: post #51
mattkrause wrote: post #50

Earlier quoted context omitted.

It’s on a slightly different level. There are strong incentives to find ‘sexy’ results, but this is mostly aimed at getting a paper into Nature, Science, or Cell. Publishing in these journals can have an outsized effect on one’s career, even if the results don’t actually hold up. In contrast, there’s not a huge pay off for getting something into Popular Science or the New York Times science section. Publicity is good…

That's probably a bigger effect. People have a huge trouble accepting that their wonderful science is a social process.

Yes, and also that it's a very economic one. If you need to churn out (and get accepted) research in quantity, then "hold on there, do we know that for sure? let's try to replicate that, with a much bigger sample size..." is downright unwelcome.

Scientists are mostly good people, but they're not angels, and if we put them in a system that rewards the wrong thing, we will get the wrong thing.

Re: Depression-Gene Studies Were Built on Shaky Foundations

#82

Post by mindgam3 »

bobcostas55 wrote: post #73
mindgam3 wrote: post #61

“Nor... should his work be taken to mean that genes don’t affect depression. They do, and with newer, bigger studies, researchers are finally working out which ones do.” This part really needs some context/sources. Anyone know which studies he’s talking about?

Here's one: https://www.nature.com/articles/s41593-018-0326-7

Thanks

Re: Depression-Gene Studies Were Built on Shaky Foundations

#83

Post by atomical »

kerbalspacepro wrote: post #55
atomical wrote: post #13

Earlier quoted context omitted.

I care about reading something written by a professional instead of a Tim Ferriss type. I've found several suspect items in his blog articles. I would like to know his credentials.

You can look them up though? It's not difficult?

Where?

Re: Depression-Gene Studies Were Built on Shaky Foundations

#84

Post by rjf72 »

mattkrause wrote: post #52
rjf72 wrote: post #30

Earlier quoted context omitted.

One factor you cannot ignore is the exponential increase in the number of 'scientists.' In times past, for better and for worse, college was generally relegated to an extremely small section of generally over-performing society. And of these a tiny minority would then go on to pursue the post-grad level education that would culminate in becoming a scientist. In today's society college has become high school 2.0. And…

I have a hard time believing that the issue is a dilution in the “quality” of scientists, but I would agree that ever-increasing competition for funds and jobs has produced some perverse incentives. The consequences for publishing something that’s wrong but not obviously indefensible are often pretty low. On average, it probably just languishes, uncited, in a dusty corner of PubMed. It might even pick up a few citati…

The replication crisis that's plaguing much of the social sciences, but especially psychology, did not cherry pick studies. It started with an effort to replicate studies only from high impact well regarded journals in psychology. [1] It found that 64% of the studies could not be replicated, leading to the curious outcome that if you assumed the literal and exact opposite of what you read in psychology (e.g. - what is said to be statistically significant, is not) - you would tend to be substantially more accurately informed than those who believe the 'science.' [1]

But more to our discussion, two of the journals from which studies were chosen were Psychological Science - impact factor 6.128, and the Journal of Personality and Social Psychology - impact factor 5.733. The replication success rate for those journals was 38% and 23% respectively. I'm certain you know, but impact factor is the yearly average number of citations for each article published in a journal. A high impact factor is generally anything above about 2. These are among the crème de la crème of psychology, and they're worthless.

As you mention pubmed, preclinical research is also a field with just an absolutely abysmal replication rate. And once again these are not cherry picked. In an internal replication study Amgen, one of the world's largest biotech companies, alongside researchers from MD Anderson, one of the world's premier cancer hospitals, were only able to replicate 11% of landmark hematology and oncology papers. [2] Needless to say those papers, and their now unsupported conclusions, were acted upon in some cases.

-----

All that said I do completely agree with you that the current system of publish or perish is playing into this, but your characterization of the current state of bad science is inaccurate. Bad science is becoming ubiquitous. However, I'm not as optimistic that there is any clean solution. There are currently about 400 players in the NBA. If you increased that 4,000 what would you expect to happen to the mean quality and the lowest common denominators? Suddenly somebody who would normally not even make it into the NBA is a first round pick. And science is a skill like any other that relies on outliers to drive it forward. We now have a system that's mostly just shoveling people through it and outputting 'scientists' for commercial gain. The output of this system is, in my opinion, fundamentally harming the entire system of science and education. And this is a downward spiral because now these individuals of overall lower quality are working as the mentors and advisers for the next generation of scientists, and actively 'educating' the current generation of doe-eyed students. This is something that will get worse, not better, over time.

[1] - https://en.wikipedia.org/wiki/Replication_crisis#Psychology_...

[2] - https://www.taconic.com/taconic-insights/quality/replication...

[3] - http://graphics8.nytimes.com/packages/pdf/education/harvarde...

Re: Depression-Gene Studies Were Built on Shaky Foundations

#85

Post by unityByFreedom »

skybrian wrote: post #56
unityByFreedom wrote: post #28

Earlier quoted context omitted.

> We're discussing his criticism of how a ton of "real psychiatrists" got some major things wrong No, you replied to me. My comment was it is strange that this Atlantic article cites him as "the psychiatrist Scott Alexander" because that is a pen name, and it isn't verifiable that he is a psychiatrist due to his anonymity. Your refutation of my point is "it doesn't matter that it isn't verifiable", but that was part…

You're assuming without evidence that Ed Yong didn't verify his credentials. How do you know? Maybe ask?

Again, the author cited him as "the psychiatrist Scott Alexander".

If he did verify credentials, he ought to have mentioned the name is an alias.

Re: Depression-Gene Studies Were Built on Shaky Foundations

#86

Post by unityByFreedom »

DataWorker wrote: post #40
unityByFreedom wrote: post #28

Earlier quoted context omitted.

> We're discussing his criticism of how a ton of "real psychiatrists" got some major things wrong No, you replied to me. My comment was it is strange that this Atlantic article cites him as "the psychiatrist Scott Alexander" because that is a pen name, and it isn't verifiable that he is a psychiatrist due to his anonymity. Your refutation of my point is "it doesn't matter that it isn't verifiable", but that was part…

Perhaps publish or parish applies to journalism as well. Factchecking costs money and not everyone has ssc’s budget.

> Factchecking costs money

Authors who do not fact-check are writing fiction. That is not journalism.

> not everyone has ssc’s budget

What budget? It's one or a handful of people writing a blog. I'm sure the budget of The Atlantic, founded in the 1800s, dwarfs that of SSC.

Re: Depression-Gene Studies Were Built on Shaky Foundations

#87

Post by AstralStorm »

rjf72 wrote: post #84
mattkrause wrote: post #52

Earlier quoted context omitted.

I have a hard time believing that the issue is a dilution in the “quality” of scientists, but I would agree that ever-increasing competition for funds and jobs has produced some perverse incentives. The consequences for publishing something that’s wrong but not obviously indefensible are often pretty low. On average, it probably just languishes, uncited, in a dusty corner of PubMed. It might even pick up a few citati…

The replication crisis that's plaguing much of the social sciences, but especially psychology, did not cherry pick studies. It started with an effort to replicate studies only from high impact well regarded journals in psychology. [1] It found that 64% of the studies could not be replicated, leading to the curious outcome that if you assumed the literal and exact opposite of what you read in psychology (e.g. - what i…

Speaking of replication, my personal experience in very narrow field of audio DSP which is easy to test gave results of 9 papers impossible to implement mostly due to missing key details, 6 more where results only apply in specific test signals (total failure in reality), 3 where performance was overstated by over 12 dB in real samples. 8 were really good and detailed. Two had actual used test code available, one has it in printed form. None with the code were any good. :D

(IEEE database around 2005 in noise reduction, echo cancellation and speaker separation or detection.)

Re: Depression-Gene Studies Were Built on Shaky Foundations

#88

Post by AstralStorm »

searine wrote: post #48
AstralStorm wrote: post #6

Statistics. Garbage in, garbage out. This shows why you have to explicitly quantify statistical power and error model for your study design. Tools are good, but not good enough to do it for you. Then, do calculate effect size instead of a binary answer. You will get answer with a digit in fifth place with a proper size unbiased sample. Note that for proper genome study, the sample size for observational studies is in…

>Statistics. Garbage in, garbage out. The problem isn't the statistics, and it isn't the solution either. Its the diagnosis data. How do you quantify if someone is depressed? How depressed are they? How do you compare two people depression? It is an incredibly nebulous disease.

That's part of "garbage in". Just because you can put a number on it does not mean the number is well scaled, validated or in any way meaningful.

Re: Depression-Gene Studies Were Built on Shaky Foundations

#89

Post by AstralStorm »

rossdavidh wrote: post #81
thanatropism wrote: post #51

Earlier quoted context omitted.

That's probably a bigger effect. People have a huge trouble accepting that their wonderful science is a social process.

Yes, and also that it's a very economic one. If you need to churn out (and get accepted) research in quantity, then "hold on there, do we know that for sure? let's try to replicate that, with a much bigger sample size..." is downright unwelcome. Scientists are mostly good people, but they're not angels, and if we put them in a system that rewards the wrong thing, we will get the wrong thing.

Unfortunately neither economics nor sociology are good enough science branches to even fix themselves. ;)

Re: Depression-Gene Studies Were Built on Shaky Foundations

#90

Post by skybrian »

unityByFreedom wrote: post #85
skybrian wrote: post #56

Earlier quoted context omitted.

You're assuming without evidence that Ed Yong didn't verify his credentials. How do you know? Maybe ask?

Again, the author cited him as "the psychiatrist Scott Alexander". If he did verify credentials, he ought to have mentioned the name is an alias.

He could have mentioned it, but I don't understand why you think it's important?

The authors of the actual scientific study are Richard Border and his co-authors. Scott Alexander is just a well-known blogger who gets credit for writing about it in a vivid way that got people's attention. He's not a primary source, so for the purposes of this article, it doesn't really matter whether he's using an alias or even whether he's a psychiatrist. (It's not a credential in the relevant field anyway.) You can verify the quotes by following the link.

Calling up other scientists in the field and asking questions about a scientific paper is how science writers verify a science article, and Ed Yong did that.

Post reply on HN