Live data from Hacker News

Scientific Regress

firstthings.com

11–20 of 56 posts

Re: Scientific Regress

#11
post #10
post #9

Earlier quoted context omitted.

Hi, I'm the author. Just wanted to pop in and say: (1) I originally had the thing stuffed full of citations and links, but since they wanted to print it in the paper magazine, we stripped all those out. If you want a reference for any particular claim, I'm happy to provide. (2) First Things didn't pressure me to make any content or editorial modifications to the article. The sole exception was one or two more technic…

It might be helpful to compile a list of those citations online (your own website, perhaps?). This is the sort of essay that might get shared around a whole bunch....it would be a shame for its impact to be lessened because of the inevitable judgment people will have for the hosting publication. It may be a perfect example of the "secular bias" the publication references, but I too groaned inwardly when I went to che…

That's a great point, I'll try to put something together.

Re: Scientific Regress

#12
>But it adds to this a pinch of glib frivolity and a dash of unembarrassed ignorance. Its rhetorical tics include a forced enthusiasm (a search on Twitter for the hashtag “#sciencedancing” speaks volumes) and a penchant for profanity. Here in Silicon Valley, one can scarcely go a day without seeing a t-shirt reading “Science: It works, b—es!” The hero of the recent popular movie The Martian boasts that he will “science the sh— out of” a situation. One of the largest groups on Facebook is titled “I f—ing love Science!” (a name which, combined with the group’s penchant for posting scarcely any actual scientific material but a lot of pictures of natural phenomena, has prompted more than one actual scientist of my acquaintance to mutter under her breath, “What you truly love is pictures”). Some of the Cult’s leaders like to play dress-up as scientists—Bill Nye and Neil deGrasse Tyson are two particularly prominent examples— but hardly any of them have contributed any research results of note. Rather, Cult leadership trends heavily in the direction of educators, popularizers, and journalists.

As much as I think the essay is great, this paragraph is a terrible ad-hominem argument for the existence of a "cult of science" or why it is dangerous. It doesn't matter if people wear xkcd shirts, Matt Damon uses profanity with "science", or if people like a facebook page. Laypeople having fun with "science" does not make a cult. A better argument would be to show how people cargo-cult science, and why it causes problems with science. The author fails to do both.

IFL is dangerous not because it is only pictures of scientific phenomena, but that bullshit is constantly posted on the page to be consumed by laypeople. The cult of science is dangerous because it has failed to teach people how to properly evaluate claims, and believe anything with the word "study" in it. Woo-pushers have appropriated the vocabulary of science indistinguishable to a layperson. Andrew Wakefield resurrected long dead pandemics by falsely publishing in the Lancet. This is what makes the cult of science dangerous, not the word "bitches".

Re: Scientific Regress

#13
TLDR: some scientists are doing science by critiquing what some other scientists did, using statistics. The authors are a wee bit confused about the fact that science is needed to weed out the 'bad' science from the 'good' science.

Re: Scientific Regress

#14
Certainly, the reproducibility crisis in social science and cancer biology is important, and deserves attention. But this article mostly strikes me as profoundly overstated. Yes, science is done by humans, and is thus imperfect. Yes, science's approach to truth is nonmonotonic. (I think most working scientists would consider this obvious.) Yes, scientists have various incentives to hype surprising results, cut corners, and even cheat. Yes, a lot of bad science gets published. But clearly science is making progress, since we clearly know more than we did 5, 10, 20, or 50 years ago. Are there things that could be done to improve the way science is conducted? Almost certainly (e.g. preregistering experimental trials, requiring power analysis in published papers), but this article is remarkable quiet about such things. In the end, what is author even really saying? That lay people should trust scientists less than they do? That doesn't seem correct to me. For instance, on climate change, it seems that normal people trust scientists a good deal less than they should, instead placing their trust in talking heads, politicians, and people who are unambiguously paid by oil companies to advocate on their behalf. So: yes, science as practiced in 2016 is imperfect. But what exactly do you propose replacing it with?

Re: Scientific Regress

#15

Certainly, the reproducibility crisis in social science and cancer biology is important, and deserves attention. But this article mostly strikes me as profoundly overstated. Yes, science is done by humans, and is thus imperfect. Yes, science's approach to truth is nonmonotonic. (I think most working scientists would consider this obvious.) Yes, scientists have various incentives to hype surprising results, cut corner…

It's academia that is broken, which is being referred to here as 'science'. What do we replace it with? Organizations that do not damage actual science in order perpetuate themselves, obviously.

Re: Scientific Regress

#16

Certainly, the reproducibility crisis in social science and cancer biology is important, and deserves attention. But this article mostly strikes me as profoundly overstated. Yes, science is done by humans, and is thus imperfect. Yes, science's approach to truth is nonmonotonic. (I think most working scientists would consider this obvious.) Yes, scientists have various incentives to hype surprising results, cut corner…

It's academia that is broken, which is being referred to here as 'science'. What do we replace it with? Organizations that do not damage actual science in order perpetuate themselves, obviously.

In my experience, academia's brokenness has been greatly exaggerated. All medium-to-large human organizations are broken to some extent. And exactly what are the concrete outlines of these mythical "organizations that do not damage actual science in order perpetuate themselves" that you propose to replace academia with?

Re: Scientific Regress

#17

TLDR: some scientists are doing science by critiquing what some other scientists did, using statistics. The authors are a wee bit confused about the fact that science is needed to weed out the 'bad' science from the 'good' science.

Please, this is not even close to the content of the article.

Re: Scientific Regress

#18
post #8

I think this is largely a case of 'moloch' as described by Scott Alexander: http://slatestarcodex.com/2014/07/30/meditations-on-moloch/ In that long (but excellent essay) Scott points out that self organizing systems like human society can sometimes arrange themselves in horrible local minimas which can be very difficult to escape. I think the way modern science is organized is decidedly suboptimal. Note that I'm tal…

A more relevant post by Scott is The Control Group is Out of Control [1]. The starting off point is that parapsychology has high-quality studies with positive results, but we're pretty sure parapsychology is wrong, and this bodes poorly for "high-quality" studies in other fields. But then it just keeps going and going... by the end I always feel like lowering the acceptable p-value ceiling by a factor of 1000, and also like that would be a joke laughed off by the nature of the problem.

1: http://slatestarcodex.com/2014/04/28/the-control-group-is-ou...

Re: Scientific Regress

#19
I agree that there are real structural issues, especially in certain fields, and especially involving the incentives for funding and publishing. However, I think that the overall impression gives some mis-characterization, at least from my perspective as a molecular biologist / geneticist.

First, I think that the most prominent studies are the most likely to have issues; generally they're doing something new, often with new methodology. It's here where extrapolations tend to be made that are the most dangerous. It's also not surprising that social sciences have such issues; they don't have rigorous tools (i.e. genetics) that can effectively ground their work. It makes compounding issues much harder to catch.

Related is the re-testing issue; in many fields, subsequent work will catch errors in previous work. If you work with a mutant and then someone else works with it, they'll see if it behaves differently than expected from the previous work. Germplasm travels, and it's the ultimate arbiter of truth. This usually does lead to further scrutiny and fixing the issue. The real problem that I've observed isn't that the errors aren't caught, but that a formal retraction isn't always done. Sometimes it just gets contradicted in a subsequent paper (and often with some relish) without ever resulting in a retraction. The editors of that journal clearly have a responsibility here that they are failing to uphold.

However, despite these faults, it's quite clear to those of us with 'boots on the ground' that you can't hide from the data; as long as you're using solid genetics and doing 'real' experiments (e.g. western blots, in situ/immuno-localization, simple gels/pcr, etc.) you can only hide for so long. The exception is if no one keeps working on it, in which case it's probably not that interesting to begin with. This also leads to a deep suspicion of bioinformatics among geneticists because we see how often things go wrong, and what is needed to make it right. Fortunately, genetics can still be used to great effect.

Ultimately, I'm not suspicious of the large body of work in my field. Most of it is based on extremely solid forward genetics that withstands lots of testing. Even now, great value is placed on these 'old school' methods because of how robust they are known to be.

Re: Scientific Regress

#20

I am a post-doctoral research scientist that has wanted to be a scientist since early childhood (6-7 years old). The structure of how science is practiced seems to have changed substantially in that time frame, not for the better. Some of my childhood naivete plays a role in those perceptions, but many older scientists share that perspective. A lot of the problems in modern science revolve around the growing focus on…

to normalize grant scores by previous grant funding to the PI. i.e., (X papers of Y impact)/Z dollars of funding over the past 10 years.

One way to bypass this would be for larger organizations to internally incentivize their researchers to cite their own organizations papers in preference to, or in addition to, other organizations papers. Thus the larger the organization, the more it can press down on the PI scales.

Alternately, even if it's not a several large organizations, smaller organizations can incentivize citing partner papers in the same way.

Which is just restating the concept that if you measure something, and then base monetary rewards off that measurement, then that number will be hyper-optimized (even if it's not a good predictor of future performance).

I don't have a good alternative, but just mentioning that there are easy circumventions (like you mentioned with double blinding)

Post reply on HN