Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Scientific Paper is Obsolete (2018)

theatlantic.com

Post reply on HN
Print view View original

131–140 of 175 posts

Re: The Scientific Paper is Obsolete (2018)

#131

Post by seoaeu »

d0mine wrote: post #93
jvanderbot wrote: post #47

Earlier quoted context omitted.

So let me clarify. You're saying that the majority of CS papers only appear to work because the analysis code has bugs? And that checking the code (presumably also the analysis code) is easier than "understanding the idea"? Neither of those ring true to me, but your mileage may vary.

Yes, a significant proportion of all scientific research is not reproducible e.g., "Over half of psychology studies fail reproducibility test" https://www.nature.com/articles/nature.2015.18248 Cancer research: "scientific findings were confirmed in only 6 (11%) cases." https://www.nature.com/articles/483531a

Those aren’t CS papers, social science and biology research have constraints that CS does not. I haven’t seen any evidence that there is anywhere close to that level of issue here. A couple of conferences adopted artifact review where an independent reviewer attempts to reproduce the experiments listed in the paper. Nearly all papers that participate do end up passing

Re: The Scientific Paper is Obsolete (2018)

#132

Post by seoaeu »

BeetleB wrote: post #56
d110af5ccf wrote: post #9

Earlier quoted context omitted.

> As a practicing scientist > version-controlled Jupyter notebooks That's awfully field specific. It probably wouldn't work for most of STEM. Even for ML I shudder to imagine trying to make sense of the inevitable monstrosities. Writing a paper is part of the thinking process. It forces the author to sit down and work through things in an orderly manner and they're still often difficult to read. I'm definitely in fav…

> It forces the author to sit down and work through things in an orderly manner and they're still often difficult to read. As a former academian: Papers are difficult to read primarily because the academic community does not value making them easier to read - no other reason. You may hear things like "papers should be written for other experts", but even that doesn't hold up to scrutiny. They typically spend 99% of t…

> They typically spend 99% of their research time on the actual research, and less than 1%

This isn’t anywhere close to my experience. 2-3 days of writing per year seems like a wild underestimate for any academic I know. I’d maybe believe only 20% of time spent writing, but for some folks even that’s probably way too low

Re: The Scientific Paper is Obsolete (2018)

#133

Post by bloak »

chongli wrote: post #32
netizen-936824 wrote: post #8

Earlier quoted context omitted.

Raw data can be on the order of terabytes, not that it can't be shared but this is a real barrier when it comes to raw data

The whole point of the field of statistics is that you can carry out statistical tests and analysis on a sample; you don’t need all of the data.

I think in astronomy they generate tens of terabytes per night and an experiment may involve automatically searching through the data for instances of something rare, like one star almost exactly behind another star, or an imminent supernova, or whatever. To test the program that does the searching you need the raw data, which until recently, at least, was stored on magnetic tape because they don't need random access to it: they read through all the archived data once per month (say) and apply all current experiments to it, so whenever you submit a new experiment you get the results back one month later.

I like the idea of publishing the data with the paper but it's not feasible in every case.

Re: The Scientific Paper is Obsolete (2018)

#134

Post by necovek »

eesmith wrote: post #118
bloaf wrote: post #77

Earlier quoted context omitted.

You should always want to have the underlying code available. Without the exact procedures they used to process their data, the only kind of "using their conclusions" you can do is the superficial "take it at face value" kind. So many important details get hand-waved away in papers that say things like "we used the well known blahblahblah method to analyze the data." If you do it right, the code should in no way inte…

I think I can convince you otherwise. If I publish a paper saying I have an algorithm which can factor large composites, and in the paper publish the factors to all of the RSA numbers listed at https://en.wikipedia.org/wiki/RSA_Factoring_Challenge , then I think people will take it seriously, and not consider it at the superficial level. Even if I don't publish the algorithm. ("Because of the security implications of…

> If I publish a paper saying I have an algorithm which can factor large composites, and in the paper publish the factors to all of the RSA numbers

If some factors of those numbers are also large composites, without access to a good algorithm, nobody can truly verify your claims.

If not and you include all of those factors in an easily digestible way for computers to process (let's call that "code"), it will be easy for anyone to reproduce your results (run that code which multiplies all the factors and gets the resulting RSA numbers).

With code, they could easily check that there's not an error in your verification method too (eg. large number multiplication broken).

This would achieve both goals: you'd withhold your algorithm for security reasons, and your results would be easier to verify.

Edit: but to be honest, I think withholding the research is a bit of a special case. You are doing it on purpose, and you can easily offer a service to prove your algorithm works (eg. imagine a "factoring" web service that instantly gives you a hash of the resulting sequence of factors, and then only mails you the actual sequence in two days).

Re: The Scientific Paper is Obsolete (2018)

#135

Post by tpoacher »

I don't disagree with the main point of the article, but I think it underplays the extent to which most publications being information-poor is the fault of the medium rather than the low standard of writing that we've become accustomed to and complacent with over the years.

This is not too unlike maintainable code. The code platform itself matters to some extent, but far less than the extent to which the author wrote with maintainability in mind.

Re: The Scientific Paper is Obsolete (2018)

#136

Post by unionpivo »

chrisseaton wrote: post #53
lowbloodsugar wrote: post #50

Earlier quoted context omitted.

If I've learned anything in my career it is that no, ideas are not valuable. There are vastly more bad ideas than good ideas. What makes an idea valuable is validation. Papers aren't to present ideas: papers are to present ideas that have been validated. We proposed an idea, we went and ran some experiments or gathered data some other way, and we concluded the idea was valid (or not valid). The point of this discussi…

But there are so many valuable papers in CS that just presented an idea. If you ignored them you’d be ignorant of how to do 90% of modern engineering. Likely the tech stack you use is built on a tower of ‘just idea’ papers.

> But there are so many valuable papers in CS that just presented an idea. If you ignored them you’d be ignorant of how to do 90% of modern engineering.

You are right. But that is why we are having this discussions, so we can improve situation.

Having even bad code (and corresponding data) available is always better than not. You can always just ignore it, and read the papers like today.

Honestly I am ok with just zip file of project directory that you have anyway, with hopefully list of versions of os, libs and programs used.

We could do a lot better than just a zip file, but that would be a nice start.

Re: The Scientific Paper is Obsolete (2018)

#137

Post by zmb_ »

netizen-936824 wrote: post #8
rtkaratekid wrote: post #3

Earlier quoted context omitted.

I think this almost every time I read the paper. It’s like Linus’ “show me the code.” I just want papers now to “show me the data and the code.” And include a discussion about why these results are important. I think it’s a great time for the scientific community to improve transparency on these fronts. Sincerely, someone who reads a lot of research but contributes none because I’m an amateur. Edit: when I say data,…

Raw data can be on the order of terabytes, not that it can't be shared but this is a real barrier when it comes to raw data

There are also legal and privacy concerns. I've worked on a few research papers where exactly one researcher had access to the data under a very strict NDA. And even they did not get full access to the raw data, only the ability to run vetted code against it and some subsets for development.

This is because the datasets were subscriber logs from mobile operators. They are both highly privacy sensitive and contain sensitive business knowledge. There is no way they will ever get published, even in some anonymized form.

Ultimately it always comes down to trust. You need to convince your peer reviewers to trust you that you have correctly done what you have claimed to have done. Of course, even when you publish datasets, you need to convince the peer reviewers to trust you that you didn't fake the data.

Re: The Scientific Paper is Obsolete (2018)

#138

Post by bborud »

tpoacher wrote: post #135

I don't disagree with the main point of the article, but I think it underplays the extent to which most publications being information-poor is the fault of the medium rather than the low standard of writing that we've become accustomed to and complacent with over the years. This is not too unlike maintainable code. The code platform itself matters to some extent, but far less than the extent to which the author wrote…

Years ago I did a lot of original research in a field that wasn't very well developed at the time. However, that research took place in the context of a startup, not academia. I was rewarded for producing solutions that worked - not for publishing papers.

I was approached by a few academics about publishing what I'd worked on, but I never did. I never did because I did consume large stacks of papers every month, and I absolutely hated the pompous, obfuscated portioning out of ideas fragment by fragment. It was an unnecessarily time consuming, and often quite useless way of sharing information. Especially since source code often wasn't part of what was published so a lot of important information got lost (which I guess was the entire point of not publishing code).

I particularly remember a 4-5 page paper that was so poorly written it took me a couple of readings (weeks apart) to realize that it described something I too had worked on. How bad is a paper when it is so obfuscated that it takes effort to recognize something you have worked on too?

I wasn't interested in wasting time dressing up my notes in drag. And if my notes as they were were not good enough, well, then someone else would surely do the same work independently and publish something at some point. Lots of the things I worked on inevitably were described by other people.

I have a love-hate relationship to scientific papers for the simple reason that they sometimes aren't really about science, but about scoring points in academia and certain types of research organizations. Yes, a lot of interesting goodies are published, but my god there is a lot of garbage that gets published. Not least because people in academia are incentivized to get as many papers as possible out of what ought to be a single publishable unit.

If we incentivize authors to spam us, they will spam us.

Re: The Scientific Paper is Obsolete (2018)

#139

Post by periheli0n »

As a publishing scientist myself I would have hoped to read more about how I can actually publish Jupyter Notebooks in a way that is recognised academically. That's at least what the title implied for me.

But the article is actually about Mathematica vs. Jupyter notebooks. Still, it's well researched and very interesting.

Nevertheless, the question how to publish better remains open. I for one think that some progress could already be made if ArXiv published html articles by default, rather than those unwieldy PDFs that really only work best when printed on paper.

Re: The Scientific Paper is Obsolete (2018)

#140

Post by jamessb »

ansgri wrote: post #111
nextos wrote: post #66

Earlier quoted context omitted.

EMBL-EBI and others had some RDF-related effort to provide machine readable abstracts, which I thought was a really cool idea. IMHO, the biggest problem with papers is politics and reviews. In many top journals like Nature there's no double-blind review (actually in Nature it's now optional but big groups never use it). And even if there was double-blind review, referees have no skin in the game. So the usual outcome…

It’s not really possible to conduct double-blind reviews in most cases: authors or at least the group can often be easily guessed from the list of references, “in our previous work…”, and research domain and approach in general.

Sensible anonymisation policies prevent people from referring to "our previous work" in submissions - e.g., the policy for CHI [1] states:

> We do expect that authors leave citations to their previous work unanonymized so that reviewers can ensure that all previous research has been taken into account by the authors. However, authors are required to cite their own work in the third person, e.g., avoid “As described in our previous work [10], … ” and use instead “As described by [10], …”

However, it is true that things like choice of research questions, approach, and equipment used can be quite suggestive of the authors' identity.

[1]: https://chi2020.acm.org/authors/papers/chi-anonymisation-pol...

Post reply on HN