Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Redesigning the Scientific Paper

theatlantic.com

Post reply on HN
Print view View original

61–70 of 110 posts

Re: Redesigning the Scientific Paper

#61

Post by ltratt »

seanmcdirmid wrote: post #47
bjourne wrote: post #21

Earlier quoted context omitted.

I've had a paper peer reviewed. It was ultimately rejected but I can't help but suspect that by making all my code publicly available, I hurt my chances of publication. The reviewers comments were about my coding style, my choice of build tool (I didn't use make, but something else which is just as easy to use), the choice of C vs C++... It's like best practices for computer security -- always strive to minimize the…

This is why artifact reviews should be separated from publication review. Artifact reviews are notoriously horrible, with reviewers inexperienced in it simply bike shedding and picking at straws. At best, artifact reviews should simply be checking for reproducibility (i.e. can they get the code to run at all).

I'm very surprised to hear this: I've submitted several artefacts; co-run an AEC for a (small-medium) conference; and spoken to a lot of people about it. I've heard virtually nothing negative until your post. Indeed, the artefact reviews I've received have nearly all been thorough and considered (one review was slightly nitpicky, but that's one review out of 10-12). For paper reviews, on the other hand, I'm very happy if 1/2 of reviews are thorough and considered. Bear in mind that most of the artefact reviewers also implicitly review the paper, and you get some idea of how good a job they do.

My main bugbear with the whole thing is the incorrect spelling of "artefact". And when that's my main bugbear... well, things aren't too bad!

Re: Redesigning the Scientific Paper

#62

Post by mettamage »

I love science, but I have a lot of issues with it lately. I'm going to express some of them since they're related to this topic.

The basic function of a scientific paper is understanding and reproducibility (inspired by jfaucett his comment).

I wonder, is reproducibility necessary? Is it even possible when things get really complex? Isn't consensus enough? I feel in the field of psychology (and most social sciences) that is what happens. I suppose consensus can be easily gamed by publication bias and a whole slew of other things. So I suppose as jfaucett puts it, a "discover for yourself" type of thing should still be there. I wonder how qualitative research could be saved and if you could call it science. In Dutch it is all called "wetenschap" and "weten" means to know.

But how should we go about design then? HCI papers use a a lot of design of design that is never justified. The paper is like: we build a system, it improved our user metrics. But is there any intuition or theory written down as to why they designed something a certain way? Not really.

I suppose one strong way to get reproducibility is by getting all the inputs needed. In a psychology study this means getting a dataset. Correlations are fuzzy but if I get the same answers out of the same dataset, then the claims must be true for that particular dataset.

Regarding design and qualitative studies, maybe, film everything? The general themes that everybody would agree upon watching everything would be the reproducible part of it?

Ok, I'll stop. The whole idea of that a paper needs to satisfy the criterion of reproducibility confuses me when I look at what science is nowadays.

Re: Redesigning the Scientific Paper

#63

Post by mettamage »

Regarding the article itself: Brett Victor is amazing and so is Strogatz. They are both my heroes actually.

But I do think there is a difference between scientific professionals communicating amongst each other and scientific communication to the public. And if mathematicians understood Strogatz his paper at the time when it was published, and there were enough mathematicians to disseminate the knowledge, then should you require that you create algorithms as animations?

Part of the reason why mathematicians and computer scientists (as researchers) conceive of new algorithms in the first place is because a lot of them are very strong in visualizing algorithms and 'being their own computer'.

Though, if a scientist wants to appeal to a broader group of scientists, then I'd recommend her or him to use every educational tool possible. For example, they could create an interactive blogpost a la parable of the polygons[1] and link that in their paper.

On an unrelated note, it is such a pity ncase isn't mentioned at all in this article!

Also related is explorabl.es, not everything is science communication in an interactive way but a lot of it is[2].

[1] http://ncase.me/polygons/ [2] http://explorabl.es/

Re: Redesigning the Scientific Paper

#64

Post by electricslpnsld »

dsacco wrote: post #45
contravariant wrote: post #20

Earlier quoted context omitted.

I assume you don't just means using Github for version control of scientific papers, because that sounds pretty pointless to me. As far as tracking incremental improvements over time goes, I think it'll be hard to do better than our current method of including references to papers. It's impossible to track ideas the same way tracking code works (which itself is limited for similar reasons). It would be nice if you co…

> I assume you don't just means using Github for version control of scientific papers, because that sounds pretty pointless to me. Why? I recently wrote a paper on GitHub. I loved it.

> I recently wrote a paper on GitHub.

When you have a paper deadline in two weeks and four authors are furiously hacking in edits left and right, Git is absolutely invaluable when writing a paper. I can't fathom how people handled merges of written text without a version control system.

Re: Redesigning the Scientific Paper

#65

Post by jacobolus »

tensor wrote: post #24

This title is horrible hyperbole. Science is more than just machine learning. Hell, even if we just constrain ourselves to "computer science" probably half of it is just math, for which the scientific paper is definitely not "obsolete" nor even deficient in any way. But outside of computer science you need laboratories to replicate experiments. Scientific papers are perfectly fine vehicles to record the necessary inf…

What makes you think math papers wouldn’t benefit from having interactive bits in the middle? Obviously isn’t relevant for all math papers, but often would be extremely helpful. I read a lot of technical papers, and it is quite frequent that I will need to spend an hour or two decoding some formal mathematical statements whose basic idea/intuition could be more clearly conveyed pictorially in a few minutes.

Of course, making interactive diagrams often takes dramatically more work than sketching pictures with a pen (or just writing down equations), and mathematicians are not typically trained to do it, so it would be an uphill slog for many. But I would love it if there was more funding/prestige/etc. available for mathematicians to make their papers more accessible by adding better visuals.

Re: Redesigning the Scientific Paper

#66

Post by p1esk »

jfaucett wrote: post #9

> These programs tend to be both so sloppily written and so central to the results that it’s contributed to a replication crisis, or put another way, a failure of the paper to perform its most basic task: to report what you’ve actually discovered, clearly enough that someone else can discover it for themselves. This is the crux of the of the problem IMHO - at least for the fields I study (AI/ML). Replicating the resu…

I also work in AI/ML field (deep learning), and usually I don't care if the paper has corresponding code or not. I read papers to find good ideas. If I find it, I can implement it myself. I rarely need more than a couple of days to test an idea (e.g. Hinton's capsules model took 4-5 hours to implement). The benefits of own implementation should be obvious.

If something important is missing or does not make sense, I usually just email the first author. Usually they respond within a couple of days, and unlike looking at code, I can also get an explanation of why they did it that way.

In fact, I don't even usually care that much about stated results (such as improvements in state of the art).

Things that matter are: deep insight into a problem, new angle to look at something, discovery of a new phenomenon, high quality explanation, practical tricks to save resources, and comprehensive prior/related work review. That's why I read papers.

Re: Redesigning the Scientific Paper

#67

Post by nonbel »

p1esk wrote: post #66
jfaucett wrote: post #9

> These programs tend to be both so sloppily written and so central to the results that it’s contributed to a replication crisis, or put another way, a failure of the paper to perform its most basic task: to report what you’ve actually discovered, clearly enough that someone else can discover it for themselves. This is the crux of the of the problem IMHO - at least for the fields I study (AI/ML). Replicating the resu…

I also work in AI/ML field (deep learning), and usually I don't care if the paper has corresponding code or not. I read papers to find good ideas. If I find it, I can implement it myself. I rarely need more than a couple of days to test an idea (e.g. Hinton's capsules model took 4-5 hours to implement). The benefits of own implementation should be obvious. If something important is missing or does not make sense, I u…

You seem to be exceptionally well funded, and/or have few deadline constraints. Your strategy will only work until you get spammed with "good ideas".

Re: Redesigning the Scientific Paper

#68

Post by p1esk »

nonbel wrote: post #67
p1esk wrote: post #66

Earlier quoted context omitted.

I also work in AI/ML field (deep learning), and usually I don't care if the paper has corresponding code or not. I read papers to find good ideas. If I find it, I can implement it myself. I rarely need more than a couple of days to test an idea (e.g. Hinton's capsules model took 4-5 hours to implement). The benefits of own implementation should be obvious. If something important is missing or does not make sense, I u…

You seem to be exceptionally well funded, and/or have few deadline constraints. Your strategy will only work until you get spammed with "good ideas".

You seem to be exceptionally well funded, and/or have few deadline constraints

I wish! :)

you get spammed with "good ideas"

Again, I wish!

In the subfield I'm focused on at the moment (efficient mapping of NN algorithms to specialized hardware, low precision computation, model compression) I don't see good ideas very often (fewer than one good paper a week). Previously I worked on music generation - also didn't really feel spammed with good ideas.

Re: Redesigning the Scientific Paper

#69

Post by therein »

eli_gottlieb wrote: post #43
bjourne wrote: post #21

Earlier quoted context omitted.

I've had a paper peer reviewed. It was ultimately rejected but I can't help but suspect that by making all my code publicly available, I hurt my chances of publication. The reviewers comments were about my coding style, my choice of build tool (I didn't use make, but something else which is just as easy to use), the choice of C vs C++... It's like best practices for computer security -- always strive to minimize the…

>The reviewers comments were about my coding style, my choice of build tool (I didn't use make, but something else which is just as easy to use), the choice of C vs C++... I don't know precisely what field you submitted in, but this is maliciously bad reviewing practice. You should have submitted a rebuttal and written to the editor calling the relevance of such "reviews" into question.

Yeah, just reading about that made my blood boil. Especially if it was a biology paper, I don't know what I would have done...

Re: Redesigning the Scientific Paper

#70

Post by nonbel »

p1esk wrote: post #68
nonbel wrote: post #67

Earlier quoted context omitted.

You seem to be exceptionally well funded, and/or have few deadline constraints. Your strategy will only work until you get spammed with "good ideas".

You seem to be exceptionally well funded, and/or have few deadline constraints I wish! :) you get spammed with "good ideas" Again, I wish! In the subfield I'm focused on at the moment (efficient mapping of NN algorithms to specialized hardware, low precision computation, model compression) I don't see good ideas very often (fewer than one good paper a week). Previously I worked on music generation - also didn't reall…

I don't mean this to be adversarial, but what exactly is it you do that would not be sped up by checking someone else's results directly before fiddling around and then trying out your own implementation?
Post reply on HN