Why the world of scientific research needs to be disrupted • Hacker News Board

Re: Why the world of scientific research needs to be disrupted

#61

Post by dantheman » Tue, Nov 08, 2011, 9:14 PM UTC

mechanical_fish wrote: post #37

dantheman wrote: post #23

Earlier quoted context omitted.

Speed is important for a number of factors, mistakes cost less if they can be quickly corrected. So being risky and publishing about your failures becomes useful (no one publishes failures...). Speed also reduces duplicate work. Peer review is slow because publishing and retracting used to take a long time, now it is instant. And as for the quality of the reviewer, perhaps it would make sense for the journal to emplo…

Peer review is slow because publishing and retracting used to take a long time, now it is instant. Have you ever done a peer review? It took me hours to do each one. To do it well, you have to stare at each procedure and each conclusion and try to imagine what could be going wrong. You may also have to read a lot of literature, both in general (so that you can recognize the difference between something genuinely new…

I'm not saying that peer review doesn't take a long time to do it right, but I'm saying that perhaps a new job can be created who's job is to review and stay current on the literature. I'm not saying one person can know everything, but that there may be a better way. We have more scientists and papers being published every year, and I'm making the argument that we've outgrown current procedures. This is not an editorial job, this job does exactly what current peer reviewers do -- but it's their full time job.

Could I be wrong, sure -- but if the field is so big that a dedicated person can't keep on top of it then there is no chance for a scientist to do that either. The biggest issue is whether there is an incentive for someone with the skills to do it; why would I want to just review the work of others when I have the skills to do my own.

Re: Why the world of scientific research needs to be disrupted

#62

Post by sausagefeet » Tue, Nov 08, 2011, 9:15 PM UTC

mechanical_fish wrote: post #37

dantheman wrote: post #23

Earlier quoted context omitted.

Speed is important for a number of factors, mistakes cost less if they can be quickly corrected. So being risky and publishing about your failures becomes useful (no one publishes failures...). Speed also reduces duplicate work. Peer review is slow because publishing and retracting used to take a long time, now it is instant. And as for the quality of the reviewer, perhaps it would make sense for the journal to emplo…

Peer review is slow because publishing and retracting used to take a long time, now it is instant. Have you ever done a peer review? It took me hours to do each one. To do it well, you have to stare at each procedure and each conclusion and try to imagine what could be going wrong. You may also have to read a lot of literature, both in general (so that you can recognize the difference between something genuinely new…

Journals should probably pay people to review works (well, really it will be included in a submission cost). But many people are resistent to the idea.

Re: Why the world of scientific research needs to be disrupted

#63

Post by kd0amg » Tue, Nov 08, 2011, 9:39 PM UTC

quanticle wrote: post #42

_delirium wrote: post #33

Earlier quoted context omitted.

On #2, it varies by field, but I agree somewhat, in that I generally think there's already a significant problem of people being too quick-on-the-draw without first familiarizing themselves with the literature. There is a lot of stuff published that rehashes (without citation) existing published results or minor variations on them, sometimes from decades earlier, that the authors and reviewers just didn't happen to b…

How much of that is because research from the 1970s is effectively locked away behind ACM and IEEE paywalls? How much research gets rehashed because the researcher can't find the result he or she is looking for via the atrocious search tools provided by the likes of Elsevier? I think that more openness in scientific publication will in fact mitigate the problem of people being quick-on-the-draw, as contradictory find…

>> I notice this particularly in computer science, where it seems nobody is capable of locating or reading journal articles published before around 1995, a handful of canonical "classic" papers excepted.

How much of that is because research from the 1970s is effectively locked away behind ACM and IEEE paywalls?

If you're serious about keeping up with research, a couple hundred dollars every year to get through a paywall doesn't sound so bad. The real problem with the stuff from the 1970s and earlier is that it's not readily available even to paying members. When I went looking for an article from 1970, the ACM Digital Library had an entry for it but not the document itself.

Re: Why the world of scientific research needs to be disrupted

#64

Post by lutorm » Tue, Nov 08, 2011, 9:41 PM UTC

zeratul wrote: post #53

mironathetin wrote: post #12

Two comments: 1. Scientists publish in slow peer reviewed journals, because scientific publication has to be peer reviewed. The reviewers are a very small selection of other scientists, who are expert in the field of the publication. As long as the reviewers don't signal white smoke, a publication is no science by definition. This way of work is important to keep quality excellent. We certainly don't need more quickl…

The article makes at lest one important point. There are only two incentives for a scientist: publications and grant money. Getting published fast reduces quality of research. Also, the focus on publications hinders other research outcomes like publicly available data or software. Industry or start-ups may not make full use of publications that attack the same problem over and over again but they surely can improve w…

There are only two incentives for a scientist: publications and grant money.

I disagree. There are three incentives: first, getting a permanent job, second getting grant money, and third, doing good work. The two former are unfortunately sometimes in opposition to the third.

Re: Why the world of scientific research needs to be disrupted

#65

Post by nickpinkston » Tue, Nov 08, 2011, 9:43 PM UTC

markkat wrote: post #11

nickpinkston wrote: post #8

Earlier quoted context omitted.

Yea, the real question is how to show their efforts in an open system so they can get tenure without fear of being "scooped" by being open

Bingo. There is good reason to fear being scooped. It can cost you funding, and thus, your job.

For sure - I wish we could find ways of seeing whose "code" was "forked" and thus whose shoulders everyone is collectively standing one - the some of that is your advances in the field right?

The current system uses paper publishing (# / impact factor) as a proxy for this, but it produces a lot of perverse incentives. If we found a way to see whose concepts were being used and when - people would instead throwing out any idea they'd have hoping that it be picked up (even unknowingly) by other researchers so they can get some of the credit.

If this could be done, would it fix this status / scooped problem, and is there actually a way to replicate the "fork" structure of something like GitHub? Unfortunately, science is both: 1.) hard to pull in-depth semantic data out of (I'm guessing a lot of nuance) and 2.) not formally coded like programming language.

Re: Why the world of scientific research needs to be disrupted

#66

Post by jeltz » Tue, Nov 08, 2011, 9:49 PM UTC

zeratul wrote: post #53

mironathetin wrote: post #12

Two comments: 1. Scientists publish in slow peer reviewed journals, because scientific publication has to be peer reviewed. The reviewers are a very small selection of other scientists, who are expert in the field of the publication. As long as the reviewers don't signal white smoke, a publication is no science by definition. This way of work is important to keep quality excellent. We certainly don't need more quickl…

The article makes at lest one important point. There are only two incentives for a scientist: publications and grant money. Getting published fast reduces quality of research. Also, the focus on publications hinders other research outcomes like publicly available data or software. Industry or start-ups may not make full use of publications that attack the same problem over and over again but they surely can improve w…

NIH are among the proponents of Open access so they should be one of the easier organisations to convince. I believe all articles resulting from NIH sponsored research must now be published in Open access papers.

Re: Why the world of scientific research needs to be disrupted

#67

Post by chalst » Tue, Nov 08, 2011, 9:51 PM UTC

dantheman wrote: post #61

mechanical_fish wrote: post #37

Earlier quoted context omitted.

Peer review is slow because publishing and retracting used to take a long time, now it is instant. Have you ever done a peer review? It took me hours to do each one. To do it well, you have to stare at each procedure and each conclusion and try to imagine what could be going wrong. You may also have to read a lot of literature, both in general (so that you can recognize the difference between something genuinely new…

I'm not saying that peer review doesn't take a long time to do it right, but I'm saying that perhaps a new job can be created who's job is to review and stay current on the literature. I'm not saying one person can know everything, but that there may be a better way. We have more scientists and papers being published every year, and I'm making the argument that we've outgrown current procedures. This is not an editor…

The point is that the skill set to be able to do science and to judge scientific manuscripts are very close together. Keeping on top of one is keeping on top of the other.

Re: Why the world of scientific research needs to be disrupted

#68

Post by chalst » Tue, Nov 08, 2011, 9:56 PM UTC

Note that the nature of research varies quite a bit from discipline to discipline. Physics has a preprint-oriented culture that reduces the conflict between individual interests and the wider ends of research.

Re: Why the world of scientific research needs to be disrupted

#69

Post by kd0amg » Tue, Nov 08, 2011, 10:11 PM UTC

khafra wrote: post #58

cube13 wrote: post #54

Earlier quoted context omitted.

>In practice, no journals ever publish replications, so nobody wastes their time reproducing others' work when they could be working on something publishable or their next grant proposal. Uh, this is exactly how Science works. It's not worth publishing the exact same results of the exact same experiment by multiple people. That only adds noise to the discussion. If I arrive to the same conclusion after running the ex…

Not publishing positive replications is just as bad a problem as not publishing negative original results. We have things like BigTable and Hadoop now; if 100 laboratories repeat an experiment and publish their results, that just means we can raise our confidence in the result by the sum of their likelihood ratios. Getting more data improves the accuracy your results even better than using more sophisticated algorith…

I remember seeing an article suggesting that the incentives related to publication (specifically in medicine) set us up for conditions where a majority of published results are wrong, and we have no way of knowing it:

Effectively requiring a positive result for publication means two phenomena can be the subject of multiple studies, with the one fluke that finds correlation being the one that gets published. At that point, we only get corrected if someone actually attempts to replicate the result, but replication effort may well be seen as a waste of resources.

Re: Why the world of scientific research needs to be disrupted

#70

Post by snowwrestler » Tue, Nov 08, 2011, 10:46 PM UTC

cube13 wrote: post #25

dantheman wrote: post #23

Earlier quoted context omitted.

Speed is important for a number of factors, mistakes cost less if they can be quickly corrected. So being risky and publishing about your failures becomes useful (no one publishes failures...). Speed also reduces duplicate work. Peer review is slow because publishing and retracting used to take a long time, now it is instant. And as for the quality of the reviewer, perhaps it would make sense for the journal to emplo…

>Speed is important for a number of factors, mistakes cost less if they can be quickly corrected. So being risky and publishing about your failures becomes useful (no one publishes failures...). Speed also reduces duplicate work. You're missing the point of scientific publication. The entire point of publication is so others can reproduce your work. Speed doesn't work if you need a billion data points taken over 20 y…

I'd make a slight correction and say that the point of publication is not so others can reproduce your work, it is so others can confirm your results.

Exact reproduction can be problematic in science because if there was a flaw in the original experimental design or method, an exact reproduction could "confirm" that same flawed result. It's better if other scientists can learn enough to understand the result, and design their own experiments to confirm or disprove it.

A super simple example is if I drop something and then report a value for gravitational acceleration. But what if I dropped a feather? If you simply reproduce the experiment, you'll get the same (wrong) result. Whereas if you select your object to drop, there's a better chance you'll pick something denser and get a different result.