Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

New academic journal only publishes 'unsurprising' research rejected by others

cbc.ca

Post reply on HN
Print view View original

131–140 of 207 posts

Re: New academic journal only publishes 'unsurprising' research rejected by others

#131

Post by shrimpx »

tralarpa wrote: post #126
mikk14 wrote: post #120

Earlier quoted context omitted.

I disagree. Negative results are important. Null results are of very limited interest. The two are worlds apart. A null result simply means you tried something and it didn't work. But you don't know why. You haven't proven it didn't work. There are literally millions of reasons why something might not work. For instance, you could try to use compound X to cure disease Y, observe no effect, and conclude that X doesn't…

> Negative results are important. Null results are of very limited interest. Correct. There is a highly cited paper in CS where the author showed that a mathematical model that was widely used in research didn't actually work (anymore) in reality. That paper was the starting point of a lot of new research in that field.

Can you add a citation for that paper?

Re: New academic journal only publishes 'unsurprising' research rejected by others

#132

Post by realradicalwash »

this is nice, and a clear step forward. but more is needed to fix a semi-broken system. because publication bias is one thing, but there is also something like a _reception bias_.

there are already null result journals and workshops. it is increasingly possible to publish negative results / unsurprising results.

still, imv those results are less likely get cited. because negative results are often messy that can be hard to understand. and people don't want to pick up a paper and then not understand. so it's no surprise that positive results, certainly the way they are framed, often neatly come together. the 'stories' they tell you are often easy to grasp mono-causal stories. it's almost like you read them and you feel good about yourself, because you feel like you suddenly understand a difficult problem.

and then, which paper will you cite in your own work? the paper you understood, that has that catchy title? or the one that you struggled through, the one that painted a picture of a far more complex, messy reality?

this kind of reception bias will be very hard to fix. it takes more critical editors, reviewers, and readers to fix this.

Re: New academic journal only publishes 'unsurprising' research rejected by others

#133

Post by rozab »

MaxBarraclough wrote: post #121
sullyj3 wrote: post #117

Earlier quoted context omitted.

https://en.wikipedia.org/wiki/Hindsight_bias

I recall Dan Ariely mentioning this in one of his books. His field is psychology/behavioural economics, a field where very often either outcome of an experiment can seem obvious after the fact. (Questions like Do newborn babies have an intuitive understanding of gravity? ) As I recall, he restructured his lectures, asking upfront for a show of hands as to which outcome everyone anticipated, before the big reveal. Aft…

> Do newborn babies have an intuitive understanding of gravity?

I really had no idea about this one, but how it's studied is very interesting.

https://www.livescience.com/18101-infants-grasp-gravity.html

Re: New academic journal only publishes 'unsurprising' research rejected by others

#135

Post by AdmiralGinge »

Cacti wrote: post #34
kraetzin wrote: post #28

Earlier quoted context omitted.

This is normal practice. To publish in a respectable journal you are charged £1000+. To publish your paper as open access, you can be charged another ~£1000 for the privledge (IEEE).

Depends on the field. For example, practically all of machine learning and related fields are $0 to publish and are open access by default. It’s been a huge boon for everyone except I guess traditional publishers.

>It’s been a huge boon for everyone except I guess traditional publishers.

My heart bleeds for them!

Re: New academic journal only publishes 'unsurprising' research rejected by others

#136

Post by pbourke »

shp0ngle wrote: post #95
user32556 wrote: post #75

Earlier quoted context omitted.

IMO, nowadays top-tier conference paper tends to focus too much on telling an interesting story. This makes researchers only show the surprising result in the paper. The unsurprising one is hardly mentioned. However, the uninteresting part would definitely help others to not make the same mistake. The uninteresting result is still result (and contribution), isn't it?

Yeah, that's the TED Talk effect, as some people put it You cannot make a TED Talk about something that people already know

What if you presented it with a bunch of single-word slides and had a compelling frame? “What my 10 years among uncontacted tribes in the Amazon taught me about the boiling point of water”

Re: New academic journal only publishes 'unsurprising' research rejected by others

#138

Post by learnstats2 »

mikk14 wrote: post #120
owenshen24 wrote: post #19

Earlier quoted context omitted.

That sucks to hear. Null results are important, as you say, if only to dissuade others from doing the same. See also the "file-drawer problem" ( https://en.wikipedia.org/wiki/Publication_bias ). Also, with regards to the incentives in the field and the lack of null results, there's always Ioannidis's classic work ( https://journals.plos.org/plosmedicine/article?id=10.1371/jo... ).

I disagree. Negative results are important. Null results are of very limited interest. The two are worlds apart. A null result simply means you tried something and it didn't work. But you don't know why. You haven't proven it didn't work. There are literally millions of reasons why something might not work. For instance, you could try to use compound X to cure disease Y, observe no effect, and conclude that X doesn't…

I understand that people generally mean 'null results' here, by your definition, when they say 'null results'. That's the intent.

Null results are also important.

Suppression of null results allows for p-hacking and confirmation biases to creep into research, and greatly reduces the power of literature reviews.

Re: New academic journal only publishes 'unsurprising' research rejected by others

#139

Post by korijn »

kraetzin wrote: post #14

Half of my PhD thesis is considered "unpublishable" because, after doing the work, my supervisors felt it's actually "unsurprising" that it didn't work out. We took methods that had been exploited to improve on previous results for over a decade to their logical extreme, and found that this method no longer leads to improvements. After doing the work it seems obvious. A paper on the subject would almost be considered…

Same experience for my master's thesis.

Re: New academic journal only publishes 'unsurprising' research rejected by others

#140

Post by bobloblaw45 »

bumby wrote: post #29
kraetzin wrote: post #14

Half of my PhD thesis is considered "unpublishable" because, after doing the work, my supervisors felt it's actually "unsurprising" that it didn't work out. We took methods that had been exploited to improve on previous results for over a decade to their logical extreme, and found that this method no longer leads to improvements. After doing the work it seems obvious. A paper on the subject would almost be considered…

My experience: I proposed a thesis to an advisor who deemed it unlikely to work. He ran it by a colleague who came to the same conclusion. I went to a different major and pursued the same thesis and invited the previous faculty who initially turned it down to the defense because it was relevant to their field. It was frustrating to hear them voice their opinions in the defense that they felt, “of course it world work…

I would have challenged them to a duel.
Post reply on HN