Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Redesigning the Scientific Paper

theatlantic.com

Post reply on HN
Print view View original

1–10 of 110 posts

Re: Redesigning the Scientific Paper

#2

Post by hprotagonist »

as a practical matter, papers will remain relevant as long as they are the metric by which grant applications and tenure decisions are made.

as a philosophical matter, for computation heavy fields, i would love to see literate programming tools become de rigeur in the peer-reviewed distribution of results. In some fields (AI) this basically happens already — the blog post with code snippets and a link to arxiv at the end is a pretty common thing now.

Re: Redesigning the Scientific Paper

#8

Post by WhompingWindows »

jostmey wrote: post #3

We need GitHub for science. But that's not enough. It needs to be combined with a mechanism for peer-review and publishing that funding agencies will find acceptable--that's the key.

It's not only about funding agencies, though. Researchers need to be trained in the use of a GitHub like system - as a former graduate student, I knew NO ONE using Github. Furthermore, after working in scientific publishing where it was MS Office or bust, I doubt you will convince Elsevier and the publishing giants that GitHub for Science is the way to go. How do these huge companies make money if scientists are sharing their work for free online? Meanwhile, you have scientists themselves, many of whom are required to hit certain numbers of articles in certain "prestige tiers" of journals in order to get tenure. If you're an extremely busy junior faculty member who will be FIRED when tenure review comes up if you don't have 1 paper per year, will you risk it on a GitHub scheme? I doubt that.

These problems are not unique to science. Take any large group of humans, be it government, the military, a company, a set of companies: they will coalesce onto the use of a certain number of tools, and it will take a LOT of energy and work to switch tools. Getting an entire industry to switch tools is extremely hard, perhaps downright impossible, and I'm not sure if it can be done in the rather slow-moving world of academic science. It simply won't be possible to enact a huge change on the system with so much institutional momentum built up.

It seems to me the best solution is to make incremental changes toward a github like system, training new scientists in grad programs to have better practices. Maybe we can't get researchers to post their full data set for all the public, but perhaps it can be made available to non-competitors, those who researchers aren't actively competing with for the same grants. Maybe researchers can be required by journals to post the peer-review comments/responses, as well as the drafts of the article, alongside the article itself. Maybe we can have researchers posting extremely detailed code/methodological information alongside the article, without forcing them to give over the entire dataset to other researchers?

Finally, maybe the whole incentive structure of academic science needs to change: maybe articles/publishing metrics should be de-valued in favor of teaching skills, mentorship ability, and collaboration within and across disciplines?

Re: Redesigning the Scientific Paper

#9

Post by jfaucett »

> These programs tend to be both so sloppily written and so central to the results that it’s contributed to a replication crisis, or put another way, a failure of the paper to perform its most basic task: to report what you’ve actually discovered, clearly enough that someone else can discover it for themselves.

This is the crux of the of the problem IMHO - at least for the fields I study (AI/ML). Replicating the results in papers I read, is way harder than it needs be, i.e. for these fields it should just be fire up a jupyter notebook and download the actual dataset they used (much harder than it seems to actually get your hands on). Very few papers actually contain links to all of this in a final polished manner so that it's #1 understandable and #2 repeatable.

Honestly, I'd much rather have your actual code and data that you used to get your results than read through the research paper if I had to choose (assuming the paper is not pure theory) - but instead there is a disproportionate focus on paper quality over "project quality" at least IMHO.

I don't really know what the solution is since apparently most academics have been perfectly fine with the status quo. I feel like we could build a much better system if we redefined our goals, since I don't think the current system is optimal for disseminating knowledge or finding and fixing mistakes in research or even generally working in a fast iterative process.

Re: Redesigning the Scientific Paper

#10

Post by dang »

awll wrote: post #7

This reads like an ad for Mathematica

Swipes like this break the site guidelines, including these important ones:

"Please respond to the strongest plausible interpretation of what someone says, not a weaker one that's easier to criticize. Assume good faith."

"Please don't post shallow dismissals, especially of other people's work. A good critical comment teaches us something."

Would you please (re-)read https://news.ycombinator.com/newsguidelines.html and not post like this here?

Post reply on HN