Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Scientific Paper is Obsolete (2018)

theatlantic.com

Post reply on HN
Print view View original

41–50 of 175 posts

Re: The Scientific Paper is Obsolete (2018)

#41

Post by musicale »

johnsutor wrote: post #39

I feel like the website paperswithcode.com addresses this very well, especially with their feature "quick start in Colab". For example, here's the top paper on the website as of now: https://paperswithcode.com/paper/towards-real-world-blind-fa... . Instead of going through the process of cloning a repo, initializing a fresh Anaconda environment from scratch, reading through nebulous, haphazard documentation about how…

It's really disappointing that technical societies like the ACM and IEEE haven't done this already.

For many journals and conferences there isn't even a way to submit the code or other digital artifacts with the PDF. A few have badging for whether digital artifacts are provided and whether the results have been reproduced or repeated by others - steps in the right direction at least.

As much as I intensely dislike their practices of overcharging for journals and milking digital library subscriptions to fund administrative overhead, the technical societies are technically non-profits and exist to serve their members and the research and professional community. This is really something they should be doing.

Re: The Scientific Paper is Obsolete (2018)

#42

Post by a-dub »

what came first? notebooks in mathematica or knuth's ideas on literate programming[1] ?

regarding notebooks themselves, i feel like they're a high concept idea but i've yet to see them really click for me in practice. i find the small cells for code to be extremely unergonomic and that the interspersal of code and plots to be distracting from both the code and the plots (although pretty fantastic for demonstrating high level features of a library, programming language or environment).

on a more fundamental level, i completely agree that mathematical notation is lossy, and that it takes a lot of skill to go from some arcane notation to an actual sense of what the relationships are- but, it requires no specific functioning technology to do so. i can review a paper from 100 years ago and understand it, where running a computer program from 20 years ago can be a challenge at best.

i think that additional high touch experiences for data exploration and teaching are fantastic ideas, but i also think that maybe the base level of communication should be kept simple; both for the purposes of maintaining accessibility and history. where the linux kernel developers insist on 78 column listservs, maybe scientists should insist on camera ready documents when it comes time to share.

i think that everyone agrees that better science would come from full data and code being supplied with publications, but interop is quite difficult as-is keeping code alive. i suppose the big question is: does it make sense to move science towards how software is done, where every bit of code is actively maintained over the years to avoid code rot, or does it make sense to come up with a scheme of freezing and archiving computing environments used in science so those in the future may be able to reproduce results or errors as they see fit. (something like, every paper must ship with a vm image for a widely available architecture that includes no proprietary code and all data used for results)

interesting questions. how to fundamentally change scientific communication such that it is enriched with data and code properly is a harder/organizational problem that i think many have tried to solve (not to mention how this ties into another problem in science- idea validation/replication and knowledge rot). building software systems for exploration and data analysis (ie; computer as partner in exploration) sounds much more fun and likely to produce useful results!

[1] https://en.wikipedia.org/wiki/Literate_programming

Re: The Scientific Paper is Obsolete (2018)

#43

Post by multilogit39 »

As an academically employed scientist of 20 years, the notion that scientific communication suddenly needs better standards puzzles me. The core research curriculum of nearly every scientific field I’ve seen, STEM or otherwise, is that the data needed for replication are non-negotiable. A paper that doesn’t include it would be table rejected by any editor. Or one would hope. This is taught at the UNDERgraduate level, for heaven’s sake.

The thought clusters emerging from the recent “replication crisis” are a fascinating rabbit hole to crawl into. If you stay near the surface, you will find mostly young scholars cheerleading open science as the obvious solution to replication difficulties. The concepts of pre-registering your study, committing to sharing data, and publishing online are all various components of this idea, varying in their necessity by the author’s devotion to their cause.

But there are several downsides to such a system that aren’t immediately obvious. For example, does the skill set of the successful scientist broaden to include how skilled they are at poaching ideas from public data that wasn’t immediately seen by their authors?

Some of the more recent criticisms invoked the spectre of “platform capitalism”, and suggested the Facebook and Linkedin-ification of science by dumping all its data on a centralized platform would likely have a net negative effect.

This article was written in 2018, and most of the discussions I’ve read since then have suggested that the open science initiative has failed despite the rapid penetration of Jupyter and visualization tools in the scientific process. Perhaps, like most things, the unseen market will pick and choose the good out of the dubious.

Re: The Scientific Paper is Obsolete (2018)

#44

Post by [deleted] »

queuebert wrote: post #11
robotresearcher wrote: post #6

Earlier quoted context omitted.

Do you do so? If not, why not?

I do share code that way, but the traditional ivory tower standards by which I am judged require "refereed journal publications" in high impact factor traditional journals. I'm trying to fight back against that, largely unsuccessfully. What would help me is to have the old geezers consider GitHub issues, PRs, and commits as a type of citation and to have a better way of tracking when my code gets used by others that…

[deleted]

Re: The Scientific Paper is Obsolete (2018)

#45

Post by d110af5ccf »

chrisseaton wrote: post #36
vlunkr wrote: post #33

Earlier quoted context omitted.

Not if the code is wrong, and therefore the conclusion may be wrong. I'm no scientist, but I don't think the point of scientific papers is to get unfounded ideas out into the world.

I can list many major influential papers in computer science that described an idea and didn't really give any concrete code, where we're still using the idea today. For example the paper on polymorphic inline caching, which is the key idea for the performance of many programming languages today, just described the idea, and didn't present any code. How was it evaluated? People sat and thought about it. Holds up toda…

I think you're talking past each other. Both are true under different circumstances. In some cases an abstract idea is the important takeaway. In other cases the central point of a paper is to present conclusions that were arrived at based on analysis of some dataset. If the code used to generate or analyze the dataset is wrong then conclusions based on it likely worthless.

Re: The Scientific Paper is Obsolete (2018)

#46

Post by jccooper »

chrisseaton wrote: post #21
rtkaratekid wrote: post #3

Earlier quoted context omitted.

I think this almost every time I read the paper. It’s like Linus’ “show me the code.” I just want papers now to “show me the data and the code.” And include a discussion about why these results are important. I think it’s a great time for the scientific community to improve transparency on these fronts. Sincerely, someone who reads a lot of research but contributes none because I’m an amateur. Edit: when I say data,…

> I just want papers now to “show me the data and the code.” But the code is secondary to the idea. The idea and the discussion around how it was arrived at and what it means is the key thing. The code is just there to implement it. You could code the same idea ten different ways.

Some papers are about an idea, but others (perhaps most others) are about results. And the results are very much dependent on the data and how you analyzed it.

Re: The Scientific Paper is Obsolete (2018)

#47

Post by jvanderbot »

fxtentacle wrote: post #29
chrisseaton wrote: post #21

Earlier quoted context omitted.

> I just want papers now to “show me the data and the code.” But the code is secondary to the idea. The idea and the discussion around how it was arrived at and what it means is the key thing. The code is just there to implement it. You could code the same idea ten different ways.

Yes but chances are it only appears to work because the analysis code has bugs. So first i want to check the code and that it works before i put effort into understanding the idea.

So let me clarify.

You're saying that the majority of CS papers only appear to work because the analysis code has bugs?

And that checking the code (presumably also the analysis code) is easier than "understanding the idea"?

Neither of those ring true to me, but your mileage may vary.

Re: The Scientific Paper is Obsolete (2018)

#48

Post by tossracct8384 »

chrisseaton wrote: post #21
rtkaratekid wrote: post #3

Earlier quoted context omitted.

I think this almost every time I read the paper. It’s like Linus’ “show me the code.” I just want papers now to “show me the data and the code.” And include a discussion about why these results are important. I think it’s a great time for the scientific community to improve transparency on these fronts. Sincerely, someone who reads a lot of research but contributes none because I’m an amateur. Edit: when I say data,…

> I just want papers now to “show me the data and the code.” But the code is secondary to the idea. The idea and the discussion around how it was arrived at and what it means is the key thing. The code is just there to implement it. You could code the same idea ten different ways.

Throwaway because I have “skin in the game”…

Yep but we should still show we did actually simulate our idea, and the methodology that gave rise to the simulation. Not because of the code but to test at all a simulation we describe actually outputs what we propose

Not everyone is a programmer but they could find one to confirm, or better yet, invalidate code my team relied on

Re: The Scientific Paper is Obsolete (2018)

#49

Post by d110af5ccf »

Mezzie wrote: post #30
d110af5ccf wrote: post #16

Earlier quoted context omitted.

> the references at the end of the paper should be split into two sections: stuff that was right and stuff that was wrong I've seen stuff like this said before but I don't think it would work. Most citations are mixed in my experience. A few objections, a bunch of stuff you aren't commenting on, and some things you're building on. Or you agree with the raw data but completely disagree with the interpretation. Others…

Yes. We should record all of this, and turn them into easily browsable graphs/hypertext to easily assemble sets of papers to read/look into. At the very least things like 'background reading', 'further reading', 'supporting evidence' and 'addressed arguments' would be useful. 'We' meaning the librarians and archivists. You guys actually researching have more than enough to do.

Actually I think that's an intriguing idea for how to improve citations. Instead of a single citation, have multiple citations that include a region of text as well as basic categorization of how the citation is being used in that instance.

I'm not sure if it would prove feasible in practice. It seems like it would aid the writing process in some cases by helping the author keep track of details. But in other cases maintaining all that metadata would become too much of a burden while writing, so it would get put off, and then it would all fall apart.

Very interesting to think about!

Re: The Scientific Paper is Obsolete (2018)

#50

Post by lowbloodsugar »

chrisseaton wrote: post #31
LudwigNagasena wrote: post #26

Earlier quoted context omitted.

The code is there to show that the idea is worth the discussion around.

Would the idea be valuable without the code? Yes.

If I've learned anything in my career it is that no, ideas are not valuable. There are vastly more bad ideas than good ideas. What makes an idea valuable is validation. Papers aren't to present ideas: papers are to present ideas that have been validated. We proposed an idea, we went and ran some experiments or gathered data some other way, and we concluded the idea was valid (or not valid). The point of this discussion here is that ideas that require huge amounts of computer effort to validate are prone to bugs. The conclusions cannot be relied upon to be validated without having the software available so that it, too, can be validated.
Post reply on HN