Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

New academic journal only publishes 'unsurprising' research rejected by others

cbc.ca

Post reply on HN
Print view View original

181–190 of 207 posts

Re: New academic journal only publishes 'unsurprising' research rejected by others

#181

Post by JamesBarney »

kraetzin wrote: post #14

Half of my PhD thesis is considered "unpublishable" because, after doing the work, my supervisors felt it's actually "unsurprising" that it didn't work out. We took methods that had been exploited to improve on previous results for over a decade to their logical extreme, and found that this method no longer leads to improvements. After doing the work it seems obvious. A paper on the subject would almost be considered…

What's even worse though is this creates enormous pressure to tweak results such that the findings are publishable.

I know one person who basically couldn't get their PhD because they couldn't reproduce another experiment and after several years of trying is pretty much certain the original results were faked in order to be publishable.

Re: New academic journal only publishes 'unsurprising' research rejected by others

#182

Post by ianhorn »

mikk14 wrote: post #120
owenshen24 wrote: post #19

Earlier quoted context omitted.

That sucks to hear. Null results are important, as you say, if only to dissuade others from doing the same. See also the "file-drawer problem" ( https://en.wikipedia.org/wiki/Publication_bias ). Also, with regards to the incentives in the field and the lack of null results, there's always Ioannidis's classic work ( https://journals.plos.org/plosmedicine/article?id=10.1371/jo... ).

I disagree. Negative results are important. Null results are of very limited interest. The two are worlds apart. A null result simply means you tried something and it didn't work. But you don't know why. You haven't proven it didn't work. There are literally millions of reasons why something might not work. For instance, you could try to use compound X to cure disease Y, observe no effect, and conclude that X doesn't…

> I disagree. Negative results are important. Null results are of very limited interest. The two are worlds apart.

I agree they're different but, but disagree that they're worlds apart. There's a spectrum between them, caused by uncertainty and statistics. If I say the average treatment effect of my new drug is probably somewhere between -x and +y, it could be a negative result or a null result. It's the fuzzy line between statistically insignificant and materially insignificant.

Maybe I only had two patients per experimental cell, so I barely learned anything. The drug's treatment effect on lifespan is between -30 years and +10 years. It's "null" in that we didn't learn much of anything.

Maybe I had a billion patients per cell and I learned that the average treatment effect on lifespan is between -0.001 days and +0.1 days. It's "negative" in that we learned the drug doesn't materially affect lifespan.

The position we seem to be in is that most conventional experiments are powered with a moderate effect size at 80%, meaning that many of our null-or-negative (-x, +y) results will be right around the region where it's unclear whether results are null or negative.

Re: New academic journal only publishes 'unsurprising' research rejected by others

#183

Post by cestith »

elliekelly wrote: post #159
MaxBarraclough wrote: post #130

Earlier quoted context omitted.

It's one of the three of his books I own, I'm afraid I can't easily narrow it down further. (I own all three as audiobooks, and I recall Simon Jones reading it, but it turns out he read all three.) • Predictably Irrational • The (Honest) Truth About Dishonesty • The Upside of Irrationality

Tangentially related I wish there were a way to “search” within audio books. Once you’ve finished the book its almost impossible to figure out where a specific chapter or passage is if you’d like to go back.

The semantic data format people have had a point all along. Just because digital audiobooks are inspired by books on cassette is no reason the data format can't support all sorts of metadata. We could have a format for written and read aloud works that highlights every word in the text on a screen as it's read when used in the proper player software, with user notations, bookmarks, indexes, an completely searchable by full text.

Re: New academic journal only publishes 'unsurprising' research rejected by others

#184

Post by jfengel »

cestith wrote: post #183
elliekelly wrote: post #159

Earlier quoted context omitted.

Tangentially related I wish there were a way to “search” within audio books. Once you’ve finished the book its almost impossible to figure out where a specific chapter or passage is if you’d like to go back.

The semantic data format people have had a point all along. Just because digital audiobooks are inspired by books on cassette is no reason the data format can't support all sorts of metadata. We could have a format for written and read aloud works that highlights every word in the text on a screen as it's read when used in the proper player software, with user notations, bookmarks, indexes, an completely searchable b…

Kindle supports this with Whispersync. I don't know how the file format works.

Re: New academic journal only publishes 'unsurprising' research rejected by others

#185

Post by akiselev »

montalbano wrote: post #180
bumby wrote: post #29

Earlier quoted context omitted.

My experience: I proposed a thesis to an advisor who deemed it unlikely to work. He ran it by a colleague who came to the same conclusion. I went to a different major and pursued the same thesis and invited the previous faculty who initially turned it down to the defense because it was relevant to their field. It was frustrating to hear them voice their opinions in the defense that they felt, “of course it world work…

I believe this quote from J.B.S. Haldane is relevant here: I suppose the process of acceptance will pass through the usual four stages: (i) This is worthless nonsense; (ii) This is an interesting, but perverse, point of view; (iii) This is true, but quite unimportant; (iv) I always said so

It's actually pretty impressive when a scientist goes through the full cycle, especially if they're already at the top of their field. Usually, they never make it past (ii) hence the Planck principle: "science advances one funeral at a time" (see The Structure of Scientific Revolutions by Kuhn)

Re: New academic journal only publishes 'unsurprising' research rejected by others

#186

Post by arendtio »

sullyj3 wrote: post #117
bumby wrote: post #29

Earlier quoted context omitted.

My experience: I proposed a thesis to an advisor who deemed it unlikely to work. He ran it by a colleague who came to the same conclusion. I went to a different major and pursued the same thesis and invited the previous faculty who initially turned it down to the defense because it was relevant to their field. It was frustrating to hear them voice their opinions in the defense that they felt, “of course it world work…

https://en.wikipedia.org/wiki/Hindsight_bias

I like the linked Egg of Columbus:

https://en.wikipedia.org/wiki/Egg_of_Columbus

Re: New academic journal only publishes 'unsurprising' research rejected by others

#187

Post by xenocyon »

choxi wrote: post #32

Why don’t research groups just publish to their own websites or directories like arxiv? What’s the role of an academic journal in 2020? Honest question, I’d love to see more blogging from hard science academics but I’m wondering if there’s a reason why that’s challenging or if it’s just academic culture. We should have a Substack/OnlyFans for scientists.

There's a pragmatic reason and a serious reason:

1. Pragmatic: you don't get tenure for publishing to your website.

2. Serious: peer review, despite its flaws, is a crucial safety check in science and it's not wise to side-step it.

Re: New academic journal only publishes 'unsurprising' research rejected by others

#188

Post by Vorh »

choxi wrote: post #32

Why don’t research groups just publish to their own websites or directories like arxiv? What’s the role of an academic journal in 2020? Honest question, I’d love to see more blogging from hard science academics but I’m wondering if there’s a reason why that’s challenging or if it’s just academic culture. We should have a Substack/OnlyFans for scientists.

> We should have a... OnlyFans for scientists.

Somewhat relevant Dilbert strip: https://dilbert.com/strip/1993-04-09

Re: New academic journal only publishes 'unsurprising' research rejected by others

#189

Post by LolWolf »

mikk14 wrote: post #120
owenshen24 wrote: post #19

Earlier quoted context omitted.

That sucks to hear. Null results are important, as you say, if only to dissuade others from doing the same. See also the "file-drawer problem" ( https://en.wikipedia.org/wiki/Publication_bias ). Also, with regards to the incentives in the field and the lack of null results, there's always Ioannidis's classic work ( https://journals.plos.org/plosmedicine/article?id=10.1371/jo... ).

I disagree. Negative results are important. Null results are of very limited interest. The two are worlds apart. A null result simply means you tried something and it didn't work. But you don't know why. You haven't proven it didn't work. There are literally millions of reasons why something might not work. For instance, you could try to use compound X to cure disease Y, observe no effect, and conclude that X doesn't…

I generally agree in the sense that "null results" should not be published as "results." But, especially in the experimental sciences, I think it would be an incredible (and very useful) feat of work to have well-documented experiments that turned out to be ultimately null or failed, to prevent others from doing the same. (Or, on the other hand, to have people improve on the given methods in order to get a positive/negative result in some specific sense. For example, photonics returning to lithium niobate platforms, which were essentially abandoned in the 80s, but has had incredible successes lately. I'm sure there's been a lot of replicated work here.)

Of course, the problem with all of this is that there really aren't very good incentives to accurately and carefully report null experimental results (except as a kind of "folk knowledge" within a given lab) which would limit its general usefulness. But the "platonic ideal," so to speak, of a null result journal I think would be relatively useful.

Post reply on HN