Live data from Hacker News

What has happened down here is the winds have changed

andrewgelman.com

71–80 of 187 posts

Re: What has happened down here is the winds have changed

#71
Excerpt from post on Robin Hanson's blog covers this well[0]:

"In the rest of society, however, we often both try to hire people who seem to show off the highest related abilities, and we let those most prestigious people have a lot of discretion in how the job is structured. For example, we let the most prestigious doctors tell us how medicine should be run, the most prestigious lawyers tells us how law should be run, the most prestigious finance professionals tell us how the financial system should work, and the most prestigious academics tell us how to run schools and research.

This can go very wrong! Imagine that we wanted research progress, and that we let the most prestigious researchers pick research topics and methods. To show off their abilities, they may pick topics and methods that most reduce the noise in estimating abilities. For example, they may pick mathematical methods, and topics that are well suited to such methods. And many of them may crowd around the same few topics, like runners at a race. These choices would succeed in helping the most able researchers to show that they are in fact the most able. But the actual research that results might not be very useful at producing research progress.

Of course if we don’t really care about research progress, or students learning, or medical effectiveness, etc., if what we mainly care about is just affiliating with the most impressive folks, well then all this isn’t much of a problem. But if we do care about these things, then unthinkingly presuming that the most prestigious people are the best to tell us how to do things, that can go very very wrong."

The winds may have changed, but landscape erodes very slowly…

[0] http://www.overcomingbias.com/2016/06/beware-prestige-based-...

Re: What has happened down here is the winds have changed

#72
> In short, Fiske doesn’t like when people use social media to publish negative comments on published research.

This doesn't at all match Fiske's article. As people on HN know very well, there's a difference between, on one hand, serious, constructive criticism; and on the other, the trolling, abuse, and just endless barrage of nonsense that buries any signal in noise, all of which is common online. It's clear Fiske is talking about the latter; the following is very recognizable:

... uncurated, unfiltered trash-talk, In the most extreme examples, online vigilantes are attacking individuals. Self-appointed data police are volunteering critiques of such personal ferocity and relentless frequency that they resemble a denial-of-service attack ...

It's not discussion, it's abuse, and we know well that the latter have tried to hide behind the former as a justification. And we also are very familiar now with the consequences:

Only what's crashing are people. The unmoderated attacks create collateral damage to targets' careers and well-being, with no accountability for the bullies. Our colleagues at all career stages are leaving the field because of sheer adversarial viciousness.

----

Then our author fabricates and attributes ideas to Fiske that she never mentions or even alludes to:

> She’s implicitly following what I’ve sometimes called the research incumbency rule: that, once an article is published in some approved venue, it should be taken as truth.

The word 'implicitly' is not a license to fabricate the rest of the sentence. And he goes on to criticize the whole field, despite saying at the beginning,

> it’s pretty much all about internal goings-on within the field of psychology (careers, tenure, smear tactics, people trying to protect their labs, public-speaking sponsors, career-stage vulnerability), and I don’t know anything about this, as I’m an outsider to psychology and I’ve seen very little of this sort of thing ...

I don't know anything about this; enough said. Also, his characterization of her article makes me wonder if he read it; her article is about trolling and abuse, and the things he mentions are only second-order effects.

The above behaviors are all familiar, and also telling is another habit we'll recognize:

> I was inclined to just keep my response short and sweet, but then it seemed worth the trouble to give some context.

Maybe we found one of the trolls.

Re: What has happened down here is the winds have changed

#73
> once an article is published in some approved venue, it should be taken as truth

And I agree with the author that it shouldn't.

A good point is made about the arguments for releasing study findings in blogs, where discussions can be had in comments and perhaps the content of the blog amended as needed.

I definitely think that the old way of publishing studies needs to change. They need to allow discussion. Those that publish the studies and those that read them need to critique and possibly conclusions be amended. And to get this to happen at an intellectual level, students and others in academia and research should be funded more by taxes and possibly be required to do critiques and duplicate other studies to some extent.

For too long we've assumed the bodies of accumulated knowledge in the sciences to be fact without dispute; that's not science. Science is all about finding models that work and gathering data and trying to draw conclusions from it. Science was never meant to be about facts; it is all about understanding. Though some understanding may be innate, much of human understanding comes from experience. While we can build upon previous "knowledge", time over time we've seen what we held to be science fact proven false, e.g. earth is flat -> it's round but the sun goes around it -> it goes around the sun, and flies spontaneous generate from feces -> flies lay eggs in feces. If we'd never allow criticism, we'd never have progressed.

Science should be about keeping an open mind and accepting there is much we don't know, even if you believe and have faith in something. Einstein believed in God, and Hawking is an atheist. Both are scientists, and both have had theories proven and disproven. We are human, and we must continually strive for understanding knowing some ideas may be right, some may be wrong, and that ideas will change.

Re: What has happened down here is the winds have changed

#74
post #59

Good scientists should welcome this challenge with open arms, sunlight all of their data and code in a repository, and show how proper statistically rigorous research is conducted. That's an elevator right to the top of the field. If I wrote code that messed up, I'd patch it. If someone else points out a bug? Even better, now I don't have to find it myself. But I would definitely not feel attacked or nor would I blam…

> sunlight all of their data and code This is not always possible. For example, if you are working with proprietary subscription-only datasets where the researcher has no redistribution rights.

[deleted]

Re: What has happened down here is the winds have changed

#75
post #59

Good scientists should welcome this challenge with open arms, sunlight all of their data and code in a repository, and show how proper statistically rigorous research is conducted. That's an elevator right to the top of the field. If I wrote code that messed up, I'd patch it. If someone else points out a bug? Even better, now I don't have to find it myself. But I would definitely not feel attacked or nor would I blam…

> Good scientists should welcome this challenge with open arms

They should welcome online trolling, abuse, and personal attacks? That is what Fiske criticizes in her article, not serious criticism and debate, which she explicitly welcomes and encourages.

Re: What has happened down here is the winds have changed

#76
post #59

Good scientists should welcome this challenge with open arms, sunlight all of their data and code in a repository, and show how proper statistically rigorous research is conducted. That's an elevator right to the top of the field. If I wrote code that messed up, I'd patch it. If someone else points out a bug? Even better, now I don't have to find it myself. But I would definitely not feel attacked or nor would I blam…

> sunlight all of their data and code This is not always possible. For example, if you are working with proprietary subscription-only datasets where the researcher has no redistribution rights.

Not directly relevant to the more widespread issue of how people should provide and react to criticism of scienctific work.

Re: What has happened down here is the winds have changed

#77
post #37

First, I don't understand why rapid feedback is bad. Communication adapts to new mediums all the time, and academia goes with it. The arXiv pre-print system is a great example of a field adapting well to a new technology. Secondly, addressing the article's main point, I'm being told that social media comments about their studies are so traumatic and detrimental that they leave their careers. That's a problem with the…

> I don't understand why rapid feedback is bad

She doesn't say rapid feedback is bad. She says online personal attacks and abuse are bad.

Re: What has happened down here is the winds have changed

#78
post #59

Good scientists should welcome this challenge with open arms, sunlight all of their data and code in a repository, and show how proper statistically rigorous research is conducted. That's an elevator right to the top of the field. If I wrote code that messed up, I'd patch it. If someone else points out a bug? Even better, now I don't have to find it myself. But I would definitely not feel attacked or nor would I blam…

> sunlight all of their data and code This is not always possible. For example, if you are working with proprietary subscription-only datasets where the researcher has no redistribution rights.

Presumably however other researchers could subscribe to the same source... third-party proprietary datasets, if non-partisan wrt research results, are still better than first-party proprietary datasets, right?

Re: What has happened down here is the winds have changed

#79
post #59

Good scientists should welcome this challenge with open arms, sunlight all of their data and code in a repository, and show how proper statistically rigorous research is conducted. That's an elevator right to the top of the field. If I wrote code that messed up, I'd patch it. If someone else points out a bug? Even better, now I don't have to find it myself. But I would definitely not feel attacked or nor would I blam…

> Good scientists should welcome this challenge with open arms They should welcome online trolling, abuse, and personal attacks? That is what Fiske criticizes in her article, not serious criticism and debate, which she explicitly welcomes and encourages.

Fiske is pulling a classic Motte and Bailey strategy.

http://slatestarcodex.com/2014/07/07/social-justice-and-word...

The Motte is that we should condemn personal attacks, trolling, and abuse. This is completely uncontroversial and nobody sane would disagree.

The Bailey is that criticism made outside of the peer review process is somehow terrorism, which is so ludicrous that ones reaction should involve spirited laughter.

Re: What has happened down here is the winds have changed

#80

> once an article is published in some approved venue, it should be taken as truth And I agree with the author that it shouldn't. A good point is made about the arguments for releasing study findings in blogs, where discussions can be had in comments and perhaps the content of the blog amended as needed. I definitely think that the old way of publishing studies needs to change. They need to allow discussion. Those th…

> For too long we've assumed the bodies of accumulated knowledge in the sciences to be fact without dispute

Who is this "we" who has assumed such a thing? Certainly no working scientist I know would agree that a result being published means it is necessarily correct. Nevertheless, some knowledge of the world scientists have accumulated has very strong support, and those who challenge it have invariably turned out to be in the wrong.

Post reply on HN