Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Redesigning the Scientific Paper

theatlantic.com

Post reply on HN
Print view View original

41–50 of 110 posts

Re: Redesigning the Scientific Paper

#41

Post by sideshowb »

iamdave wrote: post #38
mschuetz wrote: post #28

Earlier quoted context omitted.

> It's like best practices for computer security -- always strive to minimize the attack surface. I suspect that's also why some papers are unnecesarely verbose and describe simple things as complicated as possible. Can't criticize something that can't be understood.

Then why submit it for peer review at all?

Because we need peer reviewed papers on our CVs!

I also detest simple things made complex, though. In my experience (with has covered electronics, epidemiology and geography) reviewers tend to pick up on obtuse issues in text but miss glaring errors in the math. It's sad, and you can see why someone less than scrupulous would exploit that tendency by over complicating things. That said I think plenty of authors are honest but just not very clear thinkers!

Re: Redesigning the Scientific Paper

#42

Post by prepend »

amelius wrote: post #40
jfaucett wrote: post #9

> These programs tend to be both so sloppily written and so central to the results that it’s contributed to a replication crisis, or put another way, a failure of the paper to perform its most basic task: to report what you’ve actually discovered, clearly enough that someone else can discover it for themselves. This is the crux of the of the problem IMHO - at least for the fields I study (AI/ML). Replicating the resu…

> there is a disproportionate focus on paper quality over "project quality" One problem is bitrot. Stuff that runs now is not guaranteed to work in 1 or 2 years, let alone 10 years. Even more so when it runs on fancy hardware, like GPUs.

This is one of the main reasons to require source release. Open source software is much more likely to run in 10 years. It’s actually useful to package everything together into a container or VM so all the packages are there too.

I work with some genome guys and they have this problem as their sequencers basically turn over in a year or two the advances are so fast. So they have to maintain the specimen as well as all the software versions they used for analysis. It’s a pain, but otherwise nothing is reproducible.

Re: Redesigning the Scientific Paper

#43

Post by eli_gottlieb »

bjourne wrote: post #21
jfaucett wrote: post #9

> These programs tend to be both so sloppily written and so central to the results that it’s contributed to a replication crisis, or put another way, a failure of the paper to perform its most basic task: to report what you’ve actually discovered, clearly enough that someone else can discover it for themselves. This is the crux of the of the problem IMHO - at least for the fields I study (AI/ML). Replicating the resu…

I've had a paper peer reviewed. It was ultimately rejected but I can't help but suspect that by making all my code publicly available, I hurt my chances of publication. The reviewers comments were about my coding style, my choice of build tool (I didn't use make, but something else which is just as easy to use), the choice of C vs C++... It's like best practices for computer security -- always strive to minimize the…

>The reviewers comments were about my coding style, my choice of build tool (I didn't use make, but something else which is just as easy to use), the choice of C vs C++...

I don't know precisely what field you submitted in, but this is maliciously bad reviewing practice. You should have submitted a rebuttal and written to the editor calling the relevance of such "reviews" into question.

Re: Redesigning the Scientific Paper

#44

Post by sophacles »

This is a really interesting article. The use of jupyter as a publication mechanism is a really neat idea! I think this path will be fruitful, and I am all for it. I do think however that some low-hanging fruit should be addressed in parallel - stuff that makes looking through the existing work a total pain:

* Date of publication and dates of research should be required in every paper. It's really difficult to trace out the research path if you start from google or random papers you find in various archive searches. Yes that info can be present but often its in metadata where the PDF is linked rather than the PDF itself. Even worse is the "pubname, vol, issue" info rather than a year that you get... now I have to track down when the publication started publishing, how they mark off volumes and so on. I just want to know when the thing was published.

* Software versions used - if you are telling me about kernel modules or plugins/interfaces to existing software, I need to know the version to make my stuff work. Again - eventually it can be tracked down, but running a 'git bisect' on some source tree to find out when the code listings will compile is not OK.

* actual permalinks to data, code, and other supplimental information. Some 3rd party escrow service is not a terrible idea even. I hate trying to track down something from a paper only to find the link is dead and the info is no longer available or has moved a several hour google journey away.

Re: Redesigning the Scientific Paper

#45

Post by dsacco »

contravariant wrote: post #20
jostmey wrote: post #3

We need GitHub for science. But that's not enough. It needs to be combined with a mechanism for peer-review and publishing that funding agencies will find acceptable--that's the key.

I assume you don't just means using Github for version control of scientific papers, because that sounds pretty pointless to me. As far as tracking incremental improvements over time goes, I think it'll be hard to do better than our current method of including references to papers. It's impossible to track ideas the same way tracking code works (which itself is limited for similar reasons). It would be nice if you co…

> I assume you don't just means using Github for version control of scientific papers, because that sounds pretty pointless to me.

Why? I recently wrote a paper on GitHub. I loved it.

Re: Redesigning the Scientific Paper

#46

Post by bonoboTP »

bjourne wrote: post #21
jfaucett wrote: post #9

> These programs tend to be both so sloppily written and so central to the results that it’s contributed to a replication crisis, or put another way, a failure of the paper to perform its most basic task: to report what you’ve actually discovered, clearly enough that someone else can discover it for themselves. This is the crux of the of the problem IMHO - at least for the fields I study (AI/ML). Replicating the resu…

I've had a paper peer reviewed. It was ultimately rejected but I can't help but suspect that by making all my code publicly available, I hurt my chances of publication. The reviewers comments were about my coding style, my choice of build tool (I didn't use make, but something else which is just as easy to use), the choice of C vs C++... It's like best practices for computer security -- always strive to minimize the…

In machine learning / computer vision people often release their code after the paper is already accepted.

Time before the submission deadline is usually used to do more experiments and write text, not to polish the code. And after the deadline there is no hurry. What people (who want to share code) consider important is to release it till a bit before the actual conference (but this doesn't transfer to journal-based fields).

Re: Redesigning the Scientific Paper

#47

Post by seanmcdirmid »

bjourne wrote: post #21
jfaucett wrote: post #9

> These programs tend to be both so sloppily written and so central to the results that it’s contributed to a replication crisis, or put another way, a failure of the paper to perform its most basic task: to report what you’ve actually discovered, clearly enough that someone else can discover it for themselves. This is the crux of the of the problem IMHO - at least for the fields I study (AI/ML). Replicating the resu…

I've had a paper peer reviewed. It was ultimately rejected but I can't help but suspect that by making all my code publicly available, I hurt my chances of publication. The reviewers comments were about my coding style, my choice of build tool (I didn't use make, but something else which is just as easy to use), the choice of C vs C++... It's like best practices for computer security -- always strive to minimize the…

This is why artifact reviews should be separated from publication review. Artifact reviews are notoriously horrible, with reviewers inexperienced in it simply bike shedding and picking at straws. At best, artifact reviews should simply be checking for reproducibility (i.e. can they get the code to run at all).

Re: Redesigning the Scientific Paper

#48

Post by yiyus »

I do not agree that the scientific paper needs to be replaced. It should be complemented with the help of new tools, that is a very good thing, but I still want the article.

I work everyday with papers from decades ago, and I hope people will work with my papers in the future. How can I guarantee that researches of 2050 will be able to run my Jupyter notebooks?

Moreover, it is not uncommon to not be able to publish source code. I can write about models and algorithms, but I am not allowed to publish the code I write for some projects.

Re: Redesigning the Scientific Paper

#49

Post by laderach »

Sometimes you don't need to redo everything from scratch to change things.

There are a number of problems in scientific publishing. Two big ones are:

1) Distribution hurdles and paywalls imposed by rent seeking journals - who knows how much this has prevented innovation and scientific advancement in the last 20 years

2) Easily replicating experiments / easily verifying accuracy and significance of results - this is related to for instance making data used in research more easily accessible and making it easier to spot p-value hacking

Fixing these might not require a completely new format for papers. Or it could. I can envision solutions both ways.

I really like what the folks from Fermat's Library have been doing. They have been developing tools that are actually useful at the present time and push us in the right direction. I use their arXiv chrome extension https://fermatslibrary.com/librarian all the time for extracting references and bibtex. At the same time they are playing with entirely new concepts - they just posted a neat article on medium about a new unit for academic publishing https://medium.com/@fermatslibrary/a-new-unit-of-academic-pu...

Re: Redesigning the Scientific Paper

#50

Post by [deleted] »

anigbrowl wrote: post #25
chongli wrote: post #13

Earlier quoted context omitted.

I doubt you will convince Elsevier and the publishing giants Who says we need to convince them? How about we leave them behind? They are rentiers, gatekeeping society's access to publicly funded scientific knowledge. I can't think of a reason why society should allow this hostage situation to continue.

OK, but you need to figure out a way to take over the universities then because scientists are also animals who need food and shelter etc. etc. and depend on grants, stipends, and salaries to buy those things. I'm not saying this to be dismissive - I'm strongly in favor of faculties organizing to unseat administrators from their privileged positions. It's baffling to me that bright minds on campuses complain at lengt…

[deleted]
Post reply on HN