Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why the world of scientific research needs to be disrupted

gigaom.com

Post reply on HN
Print view View original

71–80 of 86 posts

Re: Why the world of scientific research needs to be disrupted

#71

Post by snowwrestler »

gwern wrote: post #55
mironathetin wrote: post #12

Two comments: 1. Scientists publish in slow peer reviewed journals, because scientific publication has to be peer reviewed. The reviewers are a very small selection of other scientists, who are expert in the field of the publication. As long as the reviewers don't signal white smoke, a publication is no science by definition. This way of work is important to keep quality excellent. We certainly don't need more quickl…

Why do we need the process of peer review? Peer review is not robust against even low levels of collusion ( http://arxiv.org/abs/1008.4324v1 ). Scientists who win the Nobel Prize find their other work suddenly being heavily cited ( http://www.nature.com/news/2011/110506/full/news.2011.270.ht... ), suggesting either that the community either badly failed in recognizing the work's true value or that they are now suckin…

Your conclusion seems to be: why do we need peer review if the results are this bad?

But an alternative perspective might be: think of how much worse things would be _without_ peer review.

Re: Why the world of scientific research needs to be disrupted

#72

Post by davidblondeau »

grep2 wrote: post #51

Actually, there are quite a few points where modern technologies would allow for a significant streamlining of research processes. Off the top of my head (as I work in the field): 1. data acquisition - you won't believe in how many labs devices with a computer interface (gpib or whatever) are running in standalone mode, costing hours in grad student hours where parameters are changed and values are read by hand - in…

My company, Collaborative Drug Discovery (https://www.collaborativedrug.com) has been developing and offering a secure collaborative data sharing environment for drug discovery data (chemistry and SAR data). There are other tools in the space as well though labs and companies are slow to change. The sector is very secretive and closed so it takes time for habits to change.

Some factors have been promoting more collaboration and data sharing:

* The increasing cost of research

* Specialization and the emergence of micro-biotech (5 people biotech startups)

* Foundations like the Bill and Melinda Gates Foundation that push for more collaboration amongst recipients of their grants (disclaimer: Collaborative Drug Discovery has received grants from the Gates foundation as well)

Re: Why the world of scientific research needs to be disrupted

#73

Post by impendia »

kd0amg wrote: post #63
quanticle wrote: post #42

Earlier quoted context omitted.

How much of that is because research from the 1970s is effectively locked away behind ACM and IEEE paywalls? How much research gets rehashed because the researcher can't find the result he or she is looking for via the atrocious search tools provided by the likes of Elsevier? I think that more openness in scientific publication will in fact mitigate the problem of people being quick-on-the-draw, as contradictory find…

>> I notice this particularly in computer science, where it seems nobody is capable of locating or reading journal articles published before around 1995, a handful of canonical "classic" papers excepted. How much of that is because research from the 1970s is effectively locked away behind ACM and IEEE paywalls? If you're serious about keeping up with research, a couple hundred dollars every year to get through a payw…

"A couple hundred dollars"? I see that you are not familiar with the pricing schemes of Elsevier and others.

Re: Why the world of scientific research needs to be disrupted

#74

Post by impendia »

cube13 wrote: post #25
dantheman wrote: post #23

Earlier quoted context omitted.

Speed is important for a number of factors, mistakes cost less if they can be quickly corrected. So being risky and publishing about your failures becomes useful (no one publishes failures...). Speed also reduces duplicate work. Peer review is slow because publishing and retracting used to take a long time, now it is instant. And as for the quality of the reviewer, perhaps it would make sense for the journal to emplo…

>Speed is important for a number of factors, mistakes cost less if they can be quickly corrected. So being risky and publishing about your failures becomes useful (no one publishes failures...). Speed also reduces duplicate work. You're missing the point of scientific publication. The entire point of publication is so others can reproduce your work. Speed doesn't work if you need a billion data points taken over 20 y…

Speaking as an academic mathematician, I don't think it is so small.

If the paper is groundbreaking, everyone will want to read it anyway.

If it is so-so, an expert will be able to determine if it is correct by skimming it pretty quickly.

Re: Why the world of scientific research needs to be disrupted

#75

Post by khafra »

PotatoEngineer wrote: post #56
angersock wrote: post #36

Earlier quoted context omitted.

Out of curiosity, is there some place to locate a clear listing of the seminal works in your field? I'm genuinely curious how much of the difficulty here is actual breadth/depth of the subject matter, and how much of the difficulty is due to some systemic inefficiency in the way research is published and consumed.

The trick is usually to look at the references section of a paper. If there's an old paper in there (say, >10 years old), then it's probably seminal. If you see the same paper in a lot of reference sections, then it's probably seminal. So: to know what's seminal so you can skip reading a bunch of papers, you need to read a bunch of papers. Right.

Sounds like a machine learning problem.

Re: Why the world of scientific research needs to be disrupted

#76

Post by kd0amg »

impendia wrote: post #73
kd0amg wrote: post #63

Earlier quoted context omitted.

>> I notice this particularly in computer science, where it seems nobody is capable of locating or reading journal articles published before around 1995, a handful of canonical "classic" papers excepted. How much of that is because research from the 1970s is effectively locked away behind ACM and IEEE paywalls? If you're serious about keeping up with research, a couple hundred dollars every year to get through a payw…

"A couple hundred dollars"? I see that you are not familiar with the pricing schemes of Elsevier and others.

I thought what I quoted was about ACM/IEEE. $198 per year (less than that for students) to the ACM gets you unlimited access to their online archives. I didn't check the IEEE rate, as I rarely encounter a paper I have to get from them instead of from ACM -- is it much more? What am I missing out on by not checking Elsevier's stash?

Re: Why the world of scientific research needs to be disrupted

#77

Post by kd0amg »

eru wrote: post #24
beambot wrote: post #15

Earlier quoted context omitted.

To add to that, 3. Scientists (particularly academics) should focus on a smaller number of solid, flushed-out papers rather than seeking out the minimum publishable unit. The latter seems to be a byproduct of the tenure process (IMO) and just adds noise.

How about you focus on publishing diffs to an overview (or so) instead of self-contained articles?

What do you do with the details too small to belong in an overview?

Re: Why the world of scientific research needs to be disrupted

#78

Post by impendia »

kd0amg wrote: post #76
impendia wrote: post #73

Earlier quoted context omitted.

"A couple hundred dollars"? I see that you are not familiar with the pricing schemes of Elsevier and others.

I thought what I quoted was about ACM/IEEE. $198 per year (less than that for students) to the ACM gets you unlimited access to their online archives. I didn't check the IEEE rate, as I rarely encounter a paper I have to get from them instead of from ACM -- is it much more? What am I missing out on by not checking Elsevier's stash?

I am in math rather than CS/EE. I am not precisely sure, and indeed Elsevier goes to some effort to make their pricing complicated, but I am pretty sure it runs well into the thousands.

I am a professor at a reasonably good state university, having just come from Stanford. At Stanford they subscribed to everything, and here our department picks and chooses so that I constantly run into paywalls despite a university subscription.

I don't know how much it would be to upgrade to Stanford-level access, but if it were $200 a professor I assume they'd do it. (Certainly I'd pay $200 from my salary for that.) I'm guessing high four or low five figures per prof in the department.

I'm guessing ACM/IEEE are nonprofits? Kudos to them for making their prices reasonable. There are some professional organizations in math (e.g. the AMS) that do something similar. But unfortunately a lot of our journals are published by for-profit companies.

Re: Why the world of scientific research needs to be disrupted

#79

Post by mironathetin »

gwern wrote: post #55
mironathetin wrote: post #12

Two comments: 1. Scientists publish in slow peer reviewed journals, because scientific publication has to be peer reviewed. The reviewers are a very small selection of other scientists, who are expert in the field of the publication. As long as the reviewers don't signal white smoke, a publication is no science by definition. This way of work is important to keep quality excellent. We certainly don't need more quickl…

Why do we need the process of peer review? Peer review is not robust against even low levels of collusion ( http://arxiv.org/abs/1008.4324v1 ). Scientists who win the Nobel Prize find their other work suddenly being heavily cited ( http://www.nature.com/news/2011/110506/full/news.2011.270.ht... ), suggesting either that the community either badly failed in recognizing the work's true value or that they are now suckin…

In order to check, if what you write makes sense, we have to read at least all texts you link to, probably also links in these text. This will take perhaps one day, if done accurately. Do you prefer that all readers spend the time, or wouldn't it be an advantage, if a few knowledgeable experts would do the job and signal to everybody: what he writes makes sense?

Sure, peer review is not flawless. But flawed procedures can either be replaced or improved. Improvement keeps what is good and changes the rest. Complete replacement may come with new problems.

Don't think that you can escape vanity where people are involved. Not even in science. There are also always people who make use of system errors. As long as that is only a few percent, we do pretty well.

And yes, I think snowwrestler also has a very good point.

Re: Why the world of scientific research needs to be disrupted

#80

Post by gwern »

snowwrestler wrote: post #71
gwern wrote: post #55

Earlier quoted context omitted.

Why do we need the process of peer review? Peer review is not robust against even low levels of collusion ( http://arxiv.org/abs/1008.4324v1 ). Scientists who win the Nobel Prize find their other work suddenly being heavily cited ( http://www.nature.com/news/2011/110506/full/news.2011.270.ht... ), suggesting either that the community either badly failed in recognizing the work's true value or that they are now suckin…

Your conclusion seems to be: why do we need peer review if the results are this bad? But an alternative perspective might be: think of how much worse things would be _without_ peer review.

> But an alternative perspective might be: think of how much worse things would be _without_ peer review.

Hey, I have this elephant repellant for sale. I know it works awesome because I haven't ever seen any elephants around here. (I also cited historical data about the absence of peer review working, you know, pretty well.)

Post reply on HN