Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Is novel research worth doing? Evidence from peer review

pnas.org

Post reply on HN
Print view View original

11–20 of 54 posts

Re: Is novel research worth doing? Evidence from peer review

#11

Post by bildung »

isaacfrond wrote: post #5
PlasmonOwl wrote: post #3

Bullshit. Journals select for novelty, this is true. But this is novelty in a trifling fashion, like teaching a dog to roll over. The entire system, at least in the UK is set up to actively discourage risky research entirely. Doing something truly new is discouraged unless you are truly a senior or leader. By then, you've probably lost most of your imagination anyway.

[citation needed]

My background is in humanities but the dynamics are the same: Chasing the goal of objectivity, easily measurable datapoints like number of publications and number of citations are used for quantifying impact of a researcher. This incentivizes researching/publishing low hanging fruit, because actually working on deep or unusual approaches means sinking more time into the endevour while risking less or no publishable results. For people trying to earn a living in academic institutions this would be stupid.

Re: Is novel research worth doing? Evidence from peer review

#12

Post by bjornsing »

> We find that higher novelty is consistently associated with higher acceptance; submissions in the top novelty quintile are 6.5 percentage points more likely than bottom quintile ones to get accepted.

Ok, so “novel” research is 6.5% more likely to be accepted if submitted, but how likely is it to even be submitted? I would argue that doing “novel” research is much riskier at the earlier stages, and that you need to look at the whole process.

Here’s a thought experiment to perhaps better explain what I mean: Let’s say you’re playing roulette, but with slightly modified rules. If you bet on a color (red or black) you have a chance to double your money as usual. But if you bet on a single number and get it right you only make 6.5% more, or 2.13 your money. Would you bet on a color (i.e. be conservative) or a single number (i.e. try to be “novel”)?

Re: Is novel research worth doing? Evidence from peer review

#13

Post by jraph »

bjornsing wrote: post #12

> We find that higher novelty is consistently associated with higher acceptance; submissions in the top novelty quintile are 6.5 percentage points more likely than bottom quintile ones to get accepted. Ok, so “novel” research is 6.5% more likely to be accepted if submitted, but how likely is it to even be submitted? I would argue that doing “novel” research is much riskier at the earlier stages, and that you need to…

I've heard some researcher saying that a way to manage this is to dedicate a part of your time to risky research, and the rest to less risky research.

So you still publish, but also get to explore "interesting" stuff (that also needs to be researched). You are covered by your less risky activity.

So, a single number 20%-30% of the time, and a color 70%-80% of the time. And your risky activity could become your comfortable zone if it takes.

Getting published is not the only worthy thing. The content itself and how you enjoy the thing too.

Re: Is novel research worth doing? Evidence from peer review

#14

Post by headsoup »

bjornsing wrote: post #12

> We find that higher novelty is consistently associated with higher acceptance; submissions in the top novelty quintile are 6.5 percentage points more likely than bottom quintile ones to get accepted. Ok, so “novel” research is 6.5% more likely to be accepted if submitted, but how likely is it to even be submitted? I would argue that doing “novel” research is much riskier at the earlier stages, and that you need to…

I think it would be much more interesting to see a study between novel research produced under different education models. E.g. government funded vs industry funded vs user pays.

Re: Is novel research worth doing? Evidence from peer review

#15

Post by nerdponx »

rscho wrote: post #8
PlasmonOwl wrote: post #3

Bullshit. Journals select for novelty, this is true. But this is novelty in a trifling fashion, like teaching a dog to roll over. The entire system, at least in the UK is set up to actively discourage risky research entirely. Doing something truly new is discouraged unless you are truly a senior or leader. By then, you've probably lost most of your imagination anyway.

As a clinical researcher, I wholeheartedly agree. Even worse, meta analysis is comparatively so well cited (even shitty ones) that doing anything else, such as original research, will be suboptimal since the reduced time investment allows you to publish nonstop. Truly, meta analysis has become a blood-ducking parasite of medical science. I know many young researchers who've specialized in meta analysis for that reaso…

Isn't there tremendous real value in meta-analysis? The problem seems to lie in how credit is allocated (via citations).

Re: Is novel research worth doing? Evidence from peer review

#16

Post by n4r9 »

nerdponx wrote: post #15
rscho wrote: post #8

Earlier quoted context omitted.

As a clinical researcher, I wholeheartedly agree. Even worse, meta analysis is comparatively so well cited (even shitty ones) that doing anything else, such as original research, will be suboptimal since the reduced time investment allows you to publish nonstop. Truly, meta analysis has become a blood-ducking parasite of medical science. I know many young researchers who've specialized in meta analysis for that reaso…

Isn't there tremendous real value in meta-analysis? The problem seems to lie in how credit is allocated (via citations).

I'd argue the problem is a combination of citations and short-termism. We don't give researchers the space to do something novel, because institutions demand results in the short term. If you give someone a tangible, quantifiable demand A and a nebulous, low-priority demand B, they will spend nearly 100% of their time chasing A. Truly novel results can take months or years of hard focus; every grant proposal, teaching duty, and routine publication is a distraction.

Re: Is novel research worth doing? Evidence from peer review

#17

Post by ThereIsNoWorry »

PlasmonOwl wrote: post #3

Bullshit. Journals select for novelty, this is true. But this is novelty in a trifling fashion, like teaching a dog to roll over. The entire system, at least in the UK is set up to actively discourage risky research entirely. Doing something truly new is discouraged unless you are truly a senior or leader. By then, you've probably lost most of your imagination anyway.

That was my conclusion after publishing a paper during my BSc. Academia is full of petty politics. Professors that gatekeep whole fields on a "like"-basis, reviewers arguing with "not enough experiments done" and questioning uniqueness and novelty in ways that just don't make any sense (to me). It is a very bureaucratic process that cannot possibly be good for innovation and progress. Then you get incentives of fields that don't make any progress to keep the hype train going so their jobs don't become meaningless over night. It just keeps going. So, I obviously declined to do a PhD and went into industry. At the bare minimum, playing politics in industry leads to vastly better (pay) outcomes than in academia.

Re: Is novel research worth doing? Evidence from peer review

#18

Post by Frost1x »

PlasmonOwl wrote: post #3

Bullshit. Journals select for novelty, this is true. But this is novelty in a trifling fashion, like teaching a dog to roll over. The entire system, at least in the UK is set up to actively discourage risky research entirely. Doing something truly new is discouraged unless you are truly a senior or leader. By then, you've probably lost most of your imagination anyway.

My anecdata from the US agrees. The best part is when funding agencies require proposal reviewers to actually send their critiques back on research proposals. I've read through many reviewers replies to proposals designed for specifically set risky/novel research funding and the responses are sometimes so conservatively comical without a truly valid argument you can't help but laugh.

I've also worked at research organizations specifically touted as being highly independent with freedom to explore novel paths and the story is basically the same. Ultimately, someone somewhere holds the purse strings and conservatism kicks in. You at minimum need to spin your research to fit popular trend keyword language or make it apply to these areas.

Financial structures ultimately dictate the lack of support for novelty. Basically, novel research has freedom only when it's independently funded, which limits the scale of most novel research. No one wants to take risk, they want to market taking high risk while taking low risk and selling mediocrity. I think the fundamental underlying issue we have across multiple societies is that we no longer proportionately reward risk anymore and that's why people seek low risk everywhere. Taking high risk often has limited opportunity for reward, even if you are successful, so why bother? You'll net more success taking low risks and only fools take high risk.

Re: Is novel research worth doing? Evidence from peer review

#19

Post by auggierose »

In the end, trying to find a perfect system for new and original research (I like new and original much better than "novel") is not possible, because new and original research does not follow any rules. All you can do is to build enough slack into the system so that people can do whatever the hell they are interested in, in addition to their measured duties. But then how do you admit people into this system? If it is based on objective measurements, you will mostly get people who will do great at their measured duties, but will mostly not be able to use the slack to produce new and original research; instead they will turn the provided slack into objectively measurable activities. Bar unlimited resources, I don't see a good way out of this conundrum. Do you?

Re: Is novel research worth doing? Evidence from peer review

#20

Post by aleks224 »

auggierose wrote: post #19

In the end, trying to find a perfect system for new and original research (I like new and original much better than "novel") is not possible, because new and original research does not follow any rules. All you can do is to build enough slack into the system so that people can do whatever the hell they are interested in, in addition to their measured duties. But then how do you admit people into this system? If it is…

I believe the culprit is the assumption the grants/tenure-track systems make about applicants/assistant professors. That assumption is negative, as if new hires will try _not to do research_. It's also about the number of grad school offerings, it just seems so huge at the moment, which again forces introducing such metrics on what is considered "success" in academia.
Post reply on HN