Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Redesigning the Scientific Paper

theatlantic.com

Post reply on HN
Print view View original

21–30 of 110 posts

Re: Redesigning the Scientific Paper

#21

Post by bjourne »

jfaucett wrote: post #9

> These programs tend to be both so sloppily written and so central to the results that it’s contributed to a replication crisis, or put another way, a failure of the paper to perform its most basic task: to report what you’ve actually discovered, clearly enough that someone else can discover it for themselves. This is the crux of the of the problem IMHO - at least for the fields I study (AI/ML). Replicating the resu…

I've had a paper peer reviewed. It was ultimately rejected but I can't help but suspect that by making all my code publicly available, I hurt my chances of publication. The reviewers comments were about my coding style, my choice of build tool (I didn't use make, but something else which is just as easy to use), the choice of C vs C++...

It's like best practices for computer security -- always strive to minimize the attack surface. :) Without source code there is much less stuff to criticize!

Re: Redesigning the Scientific Paper

#22

Post by ibeckermayer »

chongli wrote: post #13
WhompingWindows wrote: post #8

Earlier quoted context omitted.

It's not only about funding agencies, though. Researchers need to be trained in the use of a GitHub like system - as a former graduate student, I knew NO ONE using Github. Furthermore, after working in scientific publishing where it was MS Office or bust, I doubt you will convince Elsevier and the publishing giants that GitHub for Science is the way to go. How do these huge companies make money if scientists are shar…

I doubt you will convince Elsevier and the publishing giants Who says we need to convince them? How about we leave them behind? They are rentiers, gatekeeping society's access to publicly funded scientific knowledge. I can't think of a reason why society should allow this hostage situation to continue.

Exactly. These rent-seeking gate keepers should be sent to the dustbin of history where they belong

Re: Redesigning the Scientific Paper

#24

Post by tensor »

This title is horrible hyperbole. Science is more than just machine learning. Hell, even if we just constrain ourselves to "computer science" probably half of it is just math, for which the scientific paper is definitely not "obsolete" nor even deficient in any way.

But outside of computer science you need laboratories to replicate experiments. Scientific papers are perfectly fine vehicles to record the necessary information to replicate experiments in this setting. Historically appendices are used for the extended details. And yes, replication is hard, but it's part of science.

Re: Redesigning the Scientific Paper

#25

Post by anigbrowl »

chongli wrote: post #13
WhompingWindows wrote: post #8

Earlier quoted context omitted.

It's not only about funding agencies, though. Researchers need to be trained in the use of a GitHub like system - as a former graduate student, I knew NO ONE using Github. Furthermore, after working in scientific publishing where it was MS Office or bust, I doubt you will convince Elsevier and the publishing giants that GitHub for Science is the way to go. How do these huge companies make money if scientists are shar…

I doubt you will convince Elsevier and the publishing giants Who says we need to convince them? How about we leave them behind? They are rentiers, gatekeeping society's access to publicly funded scientific knowledge. I can't think of a reason why society should allow this hostage situation to continue.

OK, but you need to figure out a way to take over the universities then because scientists are also animals who need food and shelter etc. etc. and depend on grants, stipends, and salaries to buy those things.

I'm not saying this to be dismissive - I'm strongly in favor of faculties organizing to unseat administrators from their privileged positions. It's baffling to me that bright minds on campuses complain at length about the state of higher education but seem oddly averse to doing anything about it.

Re: Redesigning the Scientific Paper

#26

Post by denzil_correa »

jfaucett wrote: post #9

> These programs tend to be both so sloppily written and so central to the results that it’s contributed to a replication crisis, or put another way, a failure of the paper to perform its most basic task: to report what you’ve actually discovered, clearly enough that someone else can discover it for themselves. This is the crux of the of the problem IMHO - at least for the fields I study (AI/ML). Replicating the resu…

> I don't really know what the solution is since apparently most academics have been perfectly fine with the status quo.

Simple - change the incentives. Currently, academics are evaluated based on paper publications not "actual code". If you want code and data to be shipped, create enough incentive for them and you'd see the change.

Re: Redesigning the Scientific Paper

#27

Post by rotskoff »

bjourne wrote: post #21
jfaucett wrote: post #9

> These programs tend to be both so sloppily written and so central to the results that it’s contributed to a replication crisis, or put another way, a failure of the paper to perform its most basic task: to report what you’ve actually discovered, clearly enough that someone else can discover it for themselves. This is the crux of the of the problem IMHO - at least for the fields I study (AI/ML). Replicating the resu…

I've had a paper peer reviewed. It was ultimately rejected but I can't help but suspect that by making all my code publicly available, I hurt my chances of publication. The reviewers comments were about my coding style, my choice of build tool (I didn't use make, but something else which is just as easy to use), the choice of C vs C++... It's like best practices for computer security -- always strive to minimize the…

I don't work primarily in computer science, but rather in math/physics, and both as a reviewer and as an author, I have only seen a positive impact for sharing code. When I review, if code is made available, it is easy for me to see the details of a model or a calculation, which I really appreciate. When I am writing a paper and developing a model, knowing that I will make my code available ensures that I write things in a clear, transferable, and understandable way (which ultimately ends up being quite beneficial to me).

Re: Redesigning the Scientific Paper

#28

Post by mschuetz »

bjourne wrote: post #21
jfaucett wrote: post #9

> These programs tend to be both so sloppily written and so central to the results that it’s contributed to a replication crisis, or put another way, a failure of the paper to perform its most basic task: to report what you’ve actually discovered, clearly enough that someone else can discover it for themselves. This is the crux of the of the problem IMHO - at least for the fields I study (AI/ML). Replicating the resu…

I've had a paper peer reviewed. It was ultimately rejected but I can't help but suspect that by making all my code publicly available, I hurt my chances of publication. The reviewers comments were about my coding style, my choice of build tool (I didn't use make, but something else which is just as easy to use), the choice of C vs C++... It's like best practices for computer security -- always strive to minimize the…

> It's like best practices for computer security -- always strive to minimize the attack surface.

I suspect that's also why some papers are unnecesarely verbose and describe simple things as complicated as possible. Can't criticize something that can't be understood.

Re: Redesigning the Scientific Paper

#29

Post by 52-6F-62 »

bjourne wrote: post #21
jfaucett wrote: post #9

> These programs tend to be both so sloppily written and so central to the results that it’s contributed to a replication crisis, or put another way, a failure of the paper to perform its most basic task: to report what you’ve actually discovered, clearly enough that someone else can discover it for themselves. This is the crux of the of the problem IMHO - at least for the fields I study (AI/ML). Replicating the resu…

I've had a paper peer reviewed. It was ultimately rejected but I can't help but suspect that by making all my code publicly available, I hurt my chances of publication. The reviewers comments were about my coding style, my choice of build tool (I didn't use make, but something else which is just as easy to use), the choice of C vs C++... It's like best practices for computer security -- always strive to minimize the…

That's terribly unfortunate.

I'm entirely open to being shown otherwise, but working in that field seems more akin to working in physics than in software engineering (those are engineering problems, after all, not computer science; and sometimes they're only opinions)— and being critiqued for that in an ML/AI paper would be like critiquing a physics paper over the author's coding style— it is misdirected, IMO.

They could be squashing some legitimately good work by being too heavy handed around coding style and build process.

Re: Redesigning the Scientific Paper

#30

Post by majos »

jfaucett wrote: post #9

> These programs tend to be both so sloppily written and so central to the results that it’s contributed to a replication crisis, or put another way, a failure of the paper to perform its most basic task: to report what you’ve actually discovered, clearly enough that someone else can discover it for themselves. This is the crux of the of the problem IMHO - at least for the fields I study (AI/ML). Replicating the resu…

I would not say that most academics are "perfectly fine with the status quo". But I would say that most academics have enough competing interests taking their time away from research that they're uninterested in taking on another one with such uncertain payoff.

In a way bringing about the kind of change you reference in scientific publishing would actually be a pretty significant research accomplishment -- the field would be that much better for your efforts! But the road to get there is filled with political wrangling, talking to and serving on committees, probably forming dedicated organizations and painstakingly getting buy-in. This is not something you can realistically achieve without probably a good career's worth of political capital in your field and the drive and people skills to make it happen.

Until it does happen, making your own lab adhere to these standards is admirable but with unfortunately limited upside. I'm not saying the status quo is good, just that there are reasons for it still being the status quo.

Post reply on HN