Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

New academic journal only publishes 'unsurprising' research rejected by others

cbc.ca

Post reply on HN
Print view View original

171–180 of 207 posts

Re: New academic journal only publishes 'unsurprising' research rejected by others

#171

Post by PeterStuer »

choxi wrote: post #32

Why don’t research groups just publish to their own websites or directories like arxiv? What’s the role of an academic journal in 2020? Honest question, I’d love to see more blogging from hard science academics but I’m wondering if there’s a reason why that’s challenging or if it’s just academic culture. We should have a Substack/OnlyFans for scientists.

(At least it used to be because) the paper's publication score, which is relevant for your quantified objectives (universities run by administrators and all), is weighted by the journal's impact score, so e.g. one paper in Nature is 'worth' more than 10 in conference proceedings.

Most people do publish 'working drafts', in practice the same paper submitted to the journal, on their own sites and to archives.

Re: New academic journal only publishes 'unsurprising' research rejected by others

#172

Post by skosuri »

bnegreve wrote: post #97
skosuri wrote: post #89

Earlier quoted context omitted.

1. Other than your time, why not preprint it? 2. As an aside, I can't tell you how many times I've tried to work on stuff, it ends up working, and then I find papers and people saying what we did would never work. Sometimes the ignorance is good.

> then I find papers and people saying what we did would never work. Seriously? What kind of scientific paper makes such claims?

To be fair, it's mostly people that do this because negative results publishing is rare. There have definitely been papers saying this though; plenty of shade thrown at basically every new method in its infancy with papers saying why they won't work (you can find plenty of academic papers dunking on human genome project and shotgun sequencing; next-gen sequencing; talens/crispr's, gene therapy, immunotherapy, ai, etc).

Re: New academic journal only publishes 'unsurprising' research rejected by others

#173

Post by matthewdgreen »

smnrchrds wrote: post #104
bumby wrote: post #29

Earlier quoted context omitted.

My experience: I proposed a thesis to an advisor who deemed it unlikely to work. He ran it by a colleague who came to the same conclusion. I went to a different major and pursued the same thesis and invited the previous faculty who initially turned it down to the defense because it was relevant to their field. It was frustrating to hear them voice their opinions in the defense that they felt, “of course it world work…

Since the first and second interactions were 4 or more years apart, it is entirely conceivable that the field had moved enough during those years to warrant a genuine change in opinion.

It's also possible that they were just trying to protect a student from investing years into a project that they deemed to have a low (but perhaps non-zero) chance of success. This is a thing that good advisors should do to protect their students from career-wasting wild goose chases.

The fact that the two interactions were very different with four years and a completed thesis between them doesn't surprise me at all. My own embarrassing story is that I advised Jason Donenfeld to submit his WireGuard paper to NDSS, forgot about the meeting entirely after a few months, then complained (in retrospect, unfairly) when NDSS accepted it. Advisors do stupid, embarrassing, forgetful things all the time. The OP's story isn't even a misdemeanor.

Re: New academic journal only publishes 'unsurprising' research rejected by others

#174

Post by owenshen24 »

mikk14 wrote: post #141
learnstats2 wrote: post #138

Earlier quoted context omitted.

I understand that people generally mean 'null results' here, by your definition, when they say 'null results'. That's the intent. Null results are also important . Suppression of null results allows for p-hacking and confirmation biases to creep into research, and greatly reduces the power of literature reviews.

True, but I'm not really arguing against what you're saying. It is true that, when you have a positive (or a negative!) result you should also report on the nulls you obtained on the way (most likely in the supplementary materials) as a compendium of the result, to put it into context. What I'm arguing against is publishing a null result as a stand-alone publication. This creates the illusion of it being somehow a "r…

I see. Thanks for adding to the clarification. I think that the presentation of nulls as "results" can definitely be disingenuous. Ideally, science would have a better database to keep track of what people find, where we could add nulls in a way that doesn't highlight their "importance". As the person above says, reporting nulls is still useful to prevent p-hacking and publication bias.

(Of course, ideally I think we'd be better off focusing on reporting the data in a Bayesian approach, but that hasn't really gotten traction in the broader community.)

Re: New academic journal only publishes 'unsurprising' research rejected by others

#175

Post by derefr »

adminprof wrote: post #74
derefr wrote: post #60

Earlier quoted context omitted.

Getting that benefit only requires double-blinding during the review process. There’s no reason that both sides of the blinding cannot be removed (and also revealed to third parties) after the review process is complete.

You can't think of any reason? What about retaliation for being the less enthusiastic reviewer? What about the opposite, that you become known for being an easy reviewer who missed some obvious flaws?

You're still assuming that this happens right away. How about if the blinding is removed after 50 years? Then people studying the history of science would have the data, but it would have no impact on the careers of the people involved.

Re: New academic journal only publishes 'unsurprising' research rejected by others

#177

Post by derefr »

smabie wrote: post #81
derefr wrote: post #60

Earlier quoted context omitted.

Getting that benefit only requires double-blinding during the review process. There’s no reason that both sides of the blinding cannot be removed (and also revealed to third parties) after the review process is complete.

I can't think of any real upside of that, but can think of a lot of downside. Humans, and especially academics (I've noticed a trend in my work environments: the less money everyone makes, the more power is sublimated into inane dick measuring contests) are petty, and less personal it is, the better.

How about if the anonymity is "de-blinded" only partially, replacing it with pseudonymity?

Imagine a system like this:

1. The reviewers on your paper not only have to collaborate on a decision to accept/reject, but also write an opinion (like a https://en.wikipedia.org/wiki/Legal_opinion) about your paper, individual to themselves, after the consensus to accept/reject is reached (so some of them will likely be https://en.wikipedia.org/wiki/Dissenting_opinion s.)

2. The reviewers are each assigned a global permanent pseudonymous identifier—a UUID, basically—known only to them and some "Society for the Advancement of the Scientific Process in Academia" organization.

3. Every vote a peer-reviewer makes, and also every opinion they write about a paper, must be registered with the same academic-process org, whose job is then to collate and publish them to the Internet under the reviewer's pseudonymous identifier.

You'd be able to use such a website to both 1. audit the peer-review process for a given paper; and 2. cross-reference a given peer-reviewer's votes/opinions.

Additionally, the standards body itself could use the cross-referencing ability to normalize peer-reviewer votes, ala how the Netflix Prize recommendation systems normalized votes by a person's interpretation of the star ratings. (They'd have to ask peer-reviewers to vote with something more fine-grained than a binary pass/fail, but that'd be an easy change.)

The only thing I would worry about in such a system, is that academics might not want the negative opinions of the peer-reviewers on their paper to pop up when random other people plug the paper's DOI into Google Scholar, because a dissent on an accepted paper might unduly impact the paper's impact-factor.

Re: New academic journal only publishes 'unsurprising' research rejected by others

#178

Post by jiggunjer »

mikk14 wrote: post #120
owenshen24 wrote: post #19

Earlier quoted context omitted.

That sucks to hear. Null results are important, as you say, if only to dissuade others from doing the same. See also the "file-drawer problem" ( https://en.wikipedia.org/wiki/Publication_bias ). Also, with regards to the incentives in the field and the lack of null results, there's always Ioannidis's classic work ( https://journals.plos.org/plosmedicine/article?id=10.1371/jo... ).

I disagree. Negative results are important. Null results are of very limited interest. The two are worlds apart. A null result simply means you tried something and it didn't work. But you don't know why. You haven't proven it didn't work. There are literally millions of reasons why something might not work. For instance, you could try to use compound X to cure disease Y, observe no effect, and conclude that X doesn't…

I think you need to rework your definitions. Avoid using the word proven. Most of the time science proves things false. You can't prove anything to be true.

The difference between a null and a negative is just that a negative is an interesting null. In your null example, to create a proper negative you'd probably report several compound synthesis methods instead of one. You'd probably also want to use more mice/data in your analysis.

Re: New academic journal only publishes 'unsurprising' research rejected by others

#179

Post by ianhorn »

sullyj3 wrote: post #117
bumby wrote: post #29

Earlier quoted context omitted.

My experience: I proposed a thesis to an advisor who deemed it unlikely to work. He ran it by a colleague who came to the same conclusion. I went to a different major and pursued the same thesis and invited the previous faculty who initially turned it down to the defense because it was relevant to their field. It was frustrating to hear them voice their opinions in the defense that they felt, “of course it world work…

https://en.wikipedia.org/wiki/Hindsight_bias

I've been noticing a variant of this in myself lately. Maybe I think about some problem and stew on it and think to myself that it's probably not possible, or at least I can't come up with ideas. Then I hear that somebody else has made progress and suddenly I have a bunch of ideas. Somehow switching from "how could this work" or "can this work" to "how did they do it" leads me down entirely different paths.

I've been trying to get better at recognizing the bias and switching viewpoints without the external push.

Re: New academic journal only publishes 'unsurprising' research rejected by others

#180

Post by montalbano »

bumby wrote: post #29
kraetzin wrote: post #14

Half of my PhD thesis is considered "unpublishable" because, after doing the work, my supervisors felt it's actually "unsurprising" that it didn't work out. We took methods that had been exploited to improve on previous results for over a decade to their logical extreme, and found that this method no longer leads to improvements. After doing the work it seems obvious. A paper on the subject would almost be considered…

My experience: I proposed a thesis to an advisor who deemed it unlikely to work. He ran it by a colleague who came to the same conclusion. I went to a different major and pursued the same thesis and invited the previous faculty who initially turned it down to the defense because it was relevant to their field. It was frustrating to hear them voice their opinions in the defense that they felt, “of course it world work…

I believe this quote from J.B.S. Haldane is relevant here:

I suppose the process of acceptance will pass through the usual four stages:

(i) This is worthless nonsense;

(ii) This is an interesting, but perverse, point of view;

(iii) This is true, but quite unimportant;

(iv) I always said so

Post reply on HN