Live data from Hacker News

To an alarming degree, science is not self-correcting

economist.com

21–30 of 145 posts

Re: To an alarming degree, science is not self-correcting

#21
post #18

The Structure of Scientific Revolutions by Thomas Kuhn has some interesting commentary on how invalid scientific models of our world are eventually (and painfully) discharged. It's the book that coined the term "paradigm shift". This article tends to focus more on bad science, but I think the book is still at least partially relevant.

I'm not a fan of Kuhn, largely because he can be read as saying that, indeed, scientific models that get "overthrown" in a paradigm shift are therefore "invalid". (In the extreme, this leads to complete relativism, the idea that there is no such thing as a "valid" model at all. Kuhn himself has waffled on this point: sometimes he says he didn't really mean to take that extreme view, but other times he says things tha…

[deleted]

Re: To an alarming degree, science is not self-correcting

#22
The guy who wrote Zen and The Art of Motorcycle Maintenance had a problem with the scientific method too:

"Through multiplication upon multiplication of facts, information, theories and hypotheses it is science itself that is leading mankind from single absolute truths to multiple, indeterminate, relative ones. The major producer of the social chaos, the indeterminacy of thoughts and values that rational knowledge is supposed to eliminate, is none other than science itself."

I used to think (10 years ago) he was wrong or concerned about something that would happen far in the future beyond my lifespan.

Re: To an alarming degree, science is not self-correcting

#23
I haven't read the article, yet (in a bit of a hurry ATM...), but a crucial part of science is (independent) reproduction and verification.

Any number of things stand in the way of this: Funding and funding restrictions, counter-productive IP laws, "cult of celebrity" (to the detriment of those quiet souls who "quiety grind away"), etc.

I've been screwed over more than once by "medical science" and medical "best practices", so I have been somewhat sensitized to incongruities in this area. A recent, emerging big one: Statins, and more so the premise upon which they became ubiquitous, stand upon increasingly shaky ground -- at least with regard to their current, widespread prescription and use.

E.g. http://www.peoplespharmacy.com/2013/03/09/895-the-great-chol...

Re: To an alarming degree, science is not self-correcting

#24
What a weird article. I agree with nearly every individual fact within it, but after reading it, I still don't know the point of the piece. I don't want to jump to conclusions, but given that this piece is mostly criticism of science the part of me that associates "conservatives" with "anti-intellectual reactionaries" suspects a hidden agenda to undermine science or give climate change "skeptics" something to point at and yell about [1].

Do scientists get funding to replicate research? No, not usually.

Does replication still happen? Yes replication of a previous work is often step 0 of new research. It happens all the time.

Do replication results get published? No, not usually. Unless it's a clear rebuttal of a famous paper.

Does this matter? Debatable. Science proceeds slowly, and bad papers tend to be forgotten unless they're easily replicable. It just takes time.

Does this mean that we can't trust science? Absolutely not. You just can't trust individual papers, prima facie. This is something that scientists are supposed to learn as undergrads. Just because it's in a journal (even a "good" journal, like Science or Nature -- some would say especially if it's in one of those journals) doesn't mean it's right.

What does this tell us about armchair science? It's utterly useless. The reason that you have to be a professional scientist is not because professional scientists are smarter, but because they have a huge depth of knowledge about a particular field. Any working scientist can point out published, cited papers in their field that are total crap. But you can't tell that a paper might be crap unless you've spent years reading all of the literature in the field. Yet people on the internet persist in thinking that they can cherry-pick a single paper from arXiv or PubMed, and make sweeping conclusions about a field.

[1] indeed, the usual HN climate-change critics are in this thread, pointing at and yelling about how science is "broken".

Re: To an alarming degree, science is not self-correcting

#25

This is an excellent overview article on an important topic. It mentions many of the most influential authors on the topic of accuracy of scientific publications. Hacker News readers may enjoy "Warning Signs in Experimental Design and Interpretation"[1] by Peter Norvig, a LISP hacker who is now director of research at Google, on how to interpret scientific research. Norvig's essay is my all-time favorite link to shar…

Norvig's essay is an absolute must-read. You may learn more counter-intuitive facts in the next hour than in the last week. For convenience to mobile users, here's the link again: http://norvig.com/experiment-design.html

Re: To an alarming degree, science is not self-correcting

#26
post #15

Earlier quoted context omitted.

So there's an interesting topic hidden in your comment. What is truth? I don't mean to ask that in a flippant way. I mean to ask it in a way that has an answer - or at least a reliable set of answers. Of course this is a philosophical debate of long standing, but it has a lot of relevance for today. What do you mean by truth? How do you attain to truth? Is truth what is objectively perceived? If so, how do you determ…

In my experience, when most engineering-oriented people use the word truth, they (wittingly or unwittling) mean the pragmatic definition of "allows us to make stronger-than-previous predictive models about something". Most of the argument in philosophy of science circles strikes me as meta-interesting, not eminently relevant to the practice of science. Especially when dealing with psychology, a lot of those "what, ex…

That's a reasonable approach when talking with people who have sort of day to day experience with data and systems. Unfortunately, this doesn't per se hold true when talking with people from other disciplines. Some people hold truth as a platonic thing: it's either true or not true, and new research disproving old things implies falsehood, not a less accurate model. Tackling the elephant in that room, I'm not even talking about religious discussions, just normal discussions on physics research being found out.

A historian could speak more pertinently to this, but my understanding is that in 50s-60s "middle" USA, scientists were pretty much treated as oracles of divinity by a great deal of the common population.

Re: To an alarming degree, science is not self-correcting

#27
And this is why http://xkcd.com/836/ could have a fourth scene at the graveyard. Top hat man and his girlfriend stand by his tombstone. Girlfriend says to top hat man something like, "that study turned out to be bogus" or "he was in the control group"

Re: To an alarming degree, science is not self-correcting

#28
post #15

The truth machine is broken. How do we fix it? And why is it that whenever I see a list of the "top 10 most important problems" to solve, this isn't on it? Most educated people take the veracity of published science as given, and we clearly know that's a false assumption. I know we need more transparency in science - sharing of data and code, and negative results. But institutionally, I don't know how we get there wi…

So there's an interesting topic hidden in your comment. What is truth? I don't mean to ask that in a flippant way. I mean to ask it in a way that has an answer - or at least a reliable set of answers. Of course this is a philosophical debate of long standing, but it has a lot of relevance for today. What do you mean by truth? How do you attain to truth? Is truth what is objectively perceived? If so, how do you determ…

Actually, I don't think we have to go down that rabbit hole. The current journal system is built such that a published paper is essentially a claim that this result is reproducible and statistically significant in some sense. A philosopher can and should then go and question what exactly that means in practice, whether it's the best standard, etc etc, but in the meantime, it is also acceptable to take the system's claims and observe that they do not match the reality of the journal system on its own terms, without any particular need to dig into philosophy.

In fact, I'm personally not convinced it's the best standard, but, nevertheless, it is a good start to hold journal papers to the standards they proclaim.

Worry about what the best theory is when they show some ability to implement any theory!

Re: To an alarming degree, science is not self-correcting

#29

The truth machine is broken. How do we fix it? And why is it that whenever I see a list of the "top 10 most important problems" to solve, this isn't on it? Most educated people take the veracity of published science as given, and we clearly know that's a false assumption. I know we need more transparency in science - sharing of data and code, and negative results. But institutionally, I don't know how we get there wi…

Move to a bayesian approach for experiment design?

Re: To an alarming degree, science is not self-correcting

#30
post #18

The Structure of Scientific Revolutions by Thomas Kuhn has some interesting commentary on how invalid scientific models of our world are eventually (and painfully) discharged. It's the book that coined the term "paradigm shift". This article tends to focus more on bad science, but I think the book is still at least partially relevant.

I'm not a fan of Kuhn, largely because he can be read as saying that, indeed, scientific models that get "overthrown" in a paradigm shift are therefore "invalid". (In the extreme, this leads to complete relativism, the idea that there is no such thing as a "valid" model at all. Kuhn himself has waffled on this point: sometimes he says he didn't really mean to take that extreme view, but other times he says things tha…

I was under the impression that models are thrown out in a paradigm shift because they aren't robust enough to account for edge cases that start out as weird (and ignored) outliers to the original model. These outliers eventually become glaringly apparent to the mainstream. Ideally, the new model resulting from a paradigm shift supercedes the old model while modeling the edge cases that the older model was unable to account for. Don't take my word for it, though. It's been a long time since I've read the book ;-).
Post reply on HN