Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Redesigning the Scientific Paper

theatlantic.com

Post reply on HN
Print view View original

91–100 of 110 posts

Re: Redesigning the Scientific Paper

#91

Post by p1esk »

currymj wrote: post #75
p1esk wrote: post #66

Earlier quoted context omitted.

I also work in AI/ML field (deep learning), and usually I don't care if the paper has corresponding code or not. I read papers to find good ideas. If I find it, I can implement it myself. I rarely need more than a couple of days to test an idea (e.g. Hinton's capsules model took 4-5 hours to implement). The benefits of own implementation should be obvious. If something important is missing or does not make sense, I u…

this is the right way to go about things if you have certain goals, for sure. sometimes you need to replicate exactly the same training method, on exactly the same data — for instance if you want to use it as a baseline on a known dataset. then it becomes really important to have the code, because while an adequate replication might be easy, it takes a lot of trial and error to get perfectly the same model.

Sure, but if the code for some result is not available, I feel free to report whatever result I got implementing their method. I’m also perfectly fine with using “couldn’t reproduce” phrase in my papers.

Re: Redesigning the Scientific Paper

#92

Post by Ace17 »

mettamage wrote: post #62

I love science, but I have a lot of issues with it lately. I'm going to express some of them since they're related to this topic. The basic function of a scientific paper is understanding and reproducibility (inspired by jfaucett his comment). I wonder, is reproducibility necessary? Is it even possible when things get really complex? Isn't consensus enough? I feel in the field of psychology (and most social sciences)…

> I wonder, is reproducibility necessary? Is it even possible when things get really complex? Isn't consensus enough?

If my results are not reproductible, then I'm basically asking for your trust. So now, instead of actually doing the experiment, there's an incentive for me to forge my results, And they don't have to match reality anymore, by the way.

Gotta go ; back to working on my paper about psychic powers.

Re: Redesigning the Scientific Paper

#93

Post by TeMPOraL »

neuromantik8086 wrote: post #88
bcoughlan wrote: post #33

I used to work as a software developer for a research institute. I wanted to open source our research code and tools, and the department head was in favour of it because it would raise the profile of the research unit. There were two forces working against us. First many of the grants came from governments, and a stipulation was that we would devote some resources to helping startups commercialise the output of the r…

> Papers are the currency of academics, and all metrics for grants and careers hinge off it. It hinders cooperation and fosters a cynical environment of trying to game the metrics to secure a future in academics. I honestly can't contemplate who in their right mind would want "a future in academics" where academia is defined as a constant stream of metric gaming rather than actually accomplishing what you originally…

Imagine it like this: maybe you joined your field (be it software or academia) with dreams of making the world better. So it shouldn't matter to you who does the problem solving, as long as the problems get solved and the world gets better in a timely manner. But then, you've found a gig solving an important problem, with ok-ish pay. And you've also found yourself with dreams of a home, a spouse, maybe children. And suddenly, it starts to matter to you whether or not you get paid, so that you can fulfill your dreams. Maybe someone else could do your important work better than you. But then you won't have money to support your family. So it's gaming time, and now you end up focusing on "a future in ".

Re: Redesigning the Scientific Paper

#94

Post by akvadrako »

JorgeGT wrote: post #39

> How to integrate billions of base pairs of genomic data, and 10 times that amount of proteomic data, and historical patient data, and the results of pharmacological screens into a coherent account of how somebody got sick and what to do to make them better? How to make actionable an endless stream of new temperature and precipitation data, and oceanographic and volcanic and seismic data? How to build, and make sens…

Of course people will be able to read your Mathematica notebook in 150 years. They will just load it into a 2020-era Mathematica-engine.

The problem of software packaging will be solved by then, at least well enough to trivially emulate any of the popular environments we have today.

Re: Redesigning the Scientific Paper

#95

Post by Al-Khwarizmi »

jfaucett wrote: post #9

> These programs tend to be both so sloppily written and so central to the results that it’s contributed to a replication crisis, or put another way, a failure of the paper to perform its most basic task: to report what you’ve actually discovered, clearly enough that someone else can discover it for themselves. This is the crux of the of the problem IMHO - at least for the fields I study (AI/ML). Replicating the resu…

From an ideal point of view, I agree with your criticism. Probably most honest academics would, as we all have had frustrations after spending a lot of time trying to reproduce someone else's research. But it is very difficult to solve this problem.

Peer-review takes a large amount of time from most academics, time that is totally unpaid. With the status quo, we are OK with that - it's a service we do to each other (we need our own papers reviewed, after all) and reviewing also has the advantage of finding new ideas sooner. Although precisely in AI/ML, many academics are currently complaining: due to the rapid expansion of the field, the peer-review load has gone beyond acceptable in many cases. For the last AAAI conference I had to review 6 papers in a not too long deadline. In the last 3 months I have reviewed like 40 or so papers, and I'm very far from being a top-tier star in my field, there's people who are probably getting much more review requests (although they're probably saying no to some if they want to keep sanity).

Reviewing code and data seriously can take, how long? I would estimate an order of magnitude more than reviewing a conventional research paper in PDF.

So currently, the situation is that if you post a link to source code you may get some positive reaction in the reviews, but in 99% of the cases reviewers are not going to actually look at the code (or at least not beyond a cursory look to see if it seems coherent at a first glance) because there is just no time.

Unless we fix this, I don't think we will see papers really focusing on the code and data, regardless of good intentions.

Re: Redesigning the Scientific Paper

#96

Post by Al-Khwarizmi »

bjourne wrote: post #21
jfaucett wrote: post #9

> These programs tend to be both so sloppily written and so central to the results that it’s contributed to a replication crisis, or put another way, a failure of the paper to perform its most basic task: to report what you’ve actually discovered, clearly enough that someone else can discover it for themselves. This is the crux of the of the problem IMHO - at least for the fields I study (AI/ML). Replicating the resu…

I've had a paper peer reviewed. It was ultimately rejected but I can't help but suspect that by making all my code publicly available, I hurt my chances of publication. The reviewers comments were about my coding style, my choice of build tool (I didn't use make, but something else which is just as easy to use), the choice of C vs C++... It's like best practices for computer security -- always strive to minimize the…

I don't know what field you work on but this would be very atypical in mine.

While it's true that minimizing the attack surface is something that can work in papers, in my field reviewers typically don't look at the code. Many of my papers include code or links to it, and I haven't ever had a comment about it in reviews.

Re: Redesigning the Scientific Paper

#97

Post by bcoughlan »

neuromantik8086 wrote: post #88
bcoughlan wrote: post #33

I used to work as a software developer for a research institute. I wanted to open source our research code and tools, and the department head was in favour of it because it would raise the profile of the research unit. There were two forces working against us. First many of the grants came from governments, and a stipulation was that we would devote some resources to helping startups commercialise the output of the r…

> Papers are the currency of academics, and all metrics for grants and careers hinge off it. It hinders cooperation and fosters a cynical environment of trying to game the metrics to secure a future in academics. I honestly can't contemplate who in their right mind would want "a future in academics" where academia is defined as a constant stream of metric gaming rather than actually accomplishing what you originally…

Cynicism is rife in every industry and walk of life.

But I think most people tell themselves "once I get the PhD... once I have a permanent position... once I have tenure..." and by that stage they're institutionalized.

Re: Redesigning the Scientific Paper

#98

Post by henrikeh »

Myrmornis wrote: post #84

How well does it work to version control Mathematica notebooks in git? For example, is it possible to get meaningful textual diffs when comparing two versions of a mathematica notebook, and can git compress them enough to keep repo size down? With iPython this is also an issue -- tracking code in JSON is much less clean than tracking code in text files. It's interesting that Mathematica and iPython both left code-as-…

I use Mathematica daily and frequently store large-ish notebooks in Git. The format is textual, but the diffs are filled with a lot of noise.

Re: Redesigning the Scientific Paper

#99

Post by nonbel »

p1esk wrote: post #80
nonbel wrote: post #79

Earlier quoted context omitted.

http://jmlr.org/papers/v15/srivastava14a.html

You linked to the original dropout paper. What’s “inverse dropout”?

It is just the description of an idea I came up with without any implementation.

I was leaving it purposefully vague, just do the "inverse" of what it says in that paper.

Re: Redesigning the Scientific Paper

#100

Post by zrobotics »

RangerScience wrote: post #57
jfaucett wrote: post #9

> These programs tend to be both so sloppily written and so central to the results that it’s contributed to a replication crisis, or put another way, a failure of the paper to perform its most basic task: to report what you’ve actually discovered, clearly enough that someone else can discover it for themselves. This is the crux of the of the problem IMHO - at least for the fields I study (AI/ML). Replicating the resu…

Seems like this could pair well with the journal crisis and suggestions to implement a blockchain journal: Your paper cannot be accepted by the journal unless it includes executable code; the results of which are then injected into the "paper" view...? So, basically - a paper consists of what it takes to replicate the paper, and the blockchain journal's first step is running the replication. This would be problematic…

So where, exactly, is a block chain required here? Everything you listed could just as easily be a requirement set by the journal, after all. I mean, every journal has at least some requirements already (at the very least, nearly all require publishing in a specific language). So aside from jumping on the blockchain bandwagon just because that's the new exciting thing, what value is added here?

Good God, I'm getting tired of every single thing needing use the magic word 'blockchain' ATM.

Post reply on HN