Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Is novel research worth doing? Evidence from peer review

pnas.org

Post reply on HN
Print view View original

21–30 of 54 posts

Re: Is novel research worth doing? Evidence from peer review

#21

Post by api »

Frost1x wrote: post #18
PlasmonOwl wrote: post #3

Bullshit. Journals select for novelty, this is true. But this is novelty in a trifling fashion, like teaching a dog to roll over. The entire system, at least in the UK is set up to actively discourage risky research entirely. Doing something truly new is discouraged unless you are truly a senior or leader. By then, you've probably lost most of your imagination anyway.

My anecdata from the US agrees. The best part is when funding agencies require proposal reviewers to actually send their critiques back on research proposals. I've read through many reviewers replies to proposals designed for specifically set risky/novel research funding and the responses are sometimes so conservatively comical without a truly valid argument you can't help but laugh. I've also worked at research orga…

I've thought for a long time that this is why you see so much innovation during wartime. War overrides conservatism and gets risky things funded.

The same can apply to war-like scenarios. A lot of novel biological research got funded during COVID. We should expect to see a burst in advancement in biomedical technology as it bears fruit over the next 5-10 years.

The core problem is probably that conservatism makes the most sense at the individual level for most people. Even if you had a casino whose net payout was positive, it still may not make sense individually to gamble. If individual rewards are sporadic and concentrated on a power law distribution then statistically most players will lose.

An obvious solution is a less power-law type distribution, which tends toward some kind of socialism, but that has its own problems. Nobody's figured out yet how to provide a productive channel in a socialist system for humans' totally normal drives to compete and maximize individual outcome. Trying to suppress it doesn't work; it's like trying to totally suppress sexuality. Extreme socialist societies (e.g. Soviet Communism) often evolve into totalitarian mafia states because if there is no positive outlet for those urges they drive people into crime. Eventually you get a meritocracy concentrated in crime and the mob takes over everything.

Re: Is novel research worth doing? Evidence from peer review

#22

Post by auggierose »

aleks224 wrote: post #20
auggierose wrote: post #19

In the end, trying to find a perfect system for new and original research (I like new and original much better than "novel") is not possible, because new and original research does not follow any rules. All you can do is to build enough slack into the system so that people can do whatever the hell they are interested in, in addition to their measured duties. But then how do you admit people into this system? If it is…

I believe the culprit is the assumption the grants/tenure-track systems make about applicants/assistant professors. That assumption is negative, as if new hires will try _not to do research_. It's also about the number of grad school offerings, it just seems so huge at the moment, which again forces introducing such metrics on what is considered "success" in academia.

Of course researchers will want to do research. The question is, how do you select for those whose research will ever amount to something truly interesting, and avoid giving resources to those whose research will not?

You can try looking at the past, and derive criteria for it from that, via machine learning for example, but I would be hesitant to leave something like that up to a machine. Also times change, so criteria that worked in the past might not work now. Also, if you learn those criteria once, and then fix them, people will just game them.

Re: Is novel research worth doing? Evidence from peer review

#23

Post by a_bonobo »

PlasmonOwl wrote: post #3

Bullshit. Journals select for novelty, this is true. But this is novelty in a trifling fashion, like teaching a dog to roll over. The entire system, at least in the UK is set up to actively discourage risky research entirely. Doing something truly new is discouraged unless you are truly a senior or leader. By then, you've probably lost most of your imagination anyway.

>By then, you've probably lost most of your imagination anyway.

I've recently left academia after ~10 years to work on more blue-sky-stuff in industry, and this rings so true. It took me a few months to unlearn academia's focus on the minimal project we can do to get the next paper out (still unlearning!).

It's hard to get in, it's harder to get out.

Re: Is novel research worth doing? Evidence from peer review

#24

Post by teekert »

PlasmonOwl wrote: post #3

Bullshit. Journals select for novelty, this is true. But this is novelty in a trifling fashion, like teaching a dog to roll over. The entire system, at least in the UK is set up to actively discourage risky research entirely. Doing something truly new is discouraged unless you are truly a senior or leader. By then, you've probably lost most of your imagination anyway.

I heard established profs talk about how they ask for funding for something already done or almost successfully done. Then they do something new with the money.

Re: Is novel research worth doing? Evidence from peer review

#25

Post by Jensson »

auggierose wrote: post #22
aleks224 wrote: post #20

Earlier quoted context omitted.

I believe the culprit is the assumption the grants/tenure-track systems make about applicants/assistant professors. That assumption is negative, as if new hires will try _not to do research_. It's also about the number of grad school offerings, it just seems so huge at the moment, which again forces introducing such metrics on what is considered "success" in academia.

Of course researchers will want to do research. The question is, how do you select for those whose research will ever amount to something truly interesting, and avoid giving resources to those whose research will not? You can try looking at the past, and derive criteria for it from that, via machine learning for example, but I would be hesitant to leave something like that up to a machine. Also times change, so crite…

Current system is like measuring programmer productivity by lines of code so I think it is hard to do worse than what we currently do.

Re: Is novel research worth doing? Evidence from peer review

#26

Post by CrazyStat »

isaacfrond wrote: post #5
PlasmonOwl wrote: post #3

Bullshit. Journals select for novelty, this is true. But this is novelty in a trifling fashion, like teaching a dog to roll over. The entire system, at least in the UK is set up to actively discourage risky research entirely. Doing something truly new is discouraged unless you are truly a senior or leader. By then, you've probably lost most of your imagination anyway.

[citation needed]

I can confirm the GP matches my experience in academia (PhD, published a paper based on my dissertation, left academia shortly after).

There is immense pressure for early career researchers to work on low-hanging fruit that will give easy guaranteed publications: make a minor change to previous work and write a paper about it; write a paper about what other people wrote papers about (either a review or a meta-analysis); etc. etc. This is how you rack up publications for your tenure review. It's also how you rack up publications to get funding.

Once you have tenure you have a bit more freedom, but you still need to worry about funding (generally). And by that time you've been in the habit of grabbing the lowest-hanging fruit for 1-1.5 decades, assuming you were ~well trained~ in graduate school. After so long reinforcing that habit it becomes hard and scary to build a ladder and go after the better, higher fruit.

Re: Is novel research worth doing? Evidence from peer review

#27

Post by elcritch »

Frost1x wrote: post #18
PlasmonOwl wrote: post #3

Bullshit. Journals select for novelty, this is true. But this is novelty in a trifling fashion, like teaching a dog to roll over. The entire system, at least in the UK is set up to actively discourage risky research entirely. Doing something truly new is discouraged unless you are truly a senior or leader. By then, you've probably lost most of your imagination anyway.

My anecdata from the US agrees. The best part is when funding agencies require proposal reviewers to actually send their critiques back on research proposals. I've read through many reviewers replies to proposals designed for specifically set risky/novel research funding and the responses are sometimes so conservatively comical without a truly valid argument you can't help but laugh. I've also worked at research orga…

I have a hunch that this conservatism problem is why ADHD genes / traits remain so prevalent. The percentage of populations with it is around 5% and remarkably consistent across drastically different cultures.

The reason is that those with ADHD are often physiologically incapable of not pursuing an interest. Often those interest lie in novel areas. It’s unfortunate for those with the traits, but a net benefit for society.

So there’s a game theory equilibrium wherein successful societies will retain a certain percentage of those genes or become too conservative.

Re: Is novel research worth doing? Evidence from peer review

#28

Post by psychomugs »

I think often of this jab by Rodney Brooks on my home field of human-robot interaction [1]:

"A group of people who have never used a computer are recruited for an experiment, using variations of Unix commands to manipulate files from a command line interface. The users have never heard of computer files before and they don’t in any sense have the utility of files internalized.

The experiment is to determine whether something the experimenters have called utility versus expressiveness is more highly valued. One set of users is taught commands like “rm,” “mkdir,” and “cat,” etc., while another is taught “delete,” “new_directory,” and “type_out,” etc. They are asked to do some housekeeping tasks on a file system. Their speed and accuracy is measured, and they fill in a questionnaire about how hard and how intuitive the task was. The paper itself about these experiments touts “statistically significant results,” with lots of p-values, etc.

No HCI academics bother to invent windows, icons, a mouse, and so on. Such work is viewed as too informal and unscientific."

I just finished my PhD and these lines were in my head since they were published. Between an advisor who would only greenlight plans if they immediately led to a paper and an embarrassing amount of impractical "safe" research that only makes for good tweets or headlines, I've given up hope on anything meaningful coming out of the Ivory Towers. The robot revolution will not be peer reviewed.

[1] https://dl.acm.org/doi/10.1145/3209540

Re: Is novel research worth doing? Evidence from peer review

#29

Post by aleks224 »

auggierose wrote: post #22
aleks224 wrote: post #20

Earlier quoted context omitted.

I believe the culprit is the assumption the grants/tenure-track systems make about applicants/assistant professors. That assumption is negative, as if new hires will try _not to do research_. It's also about the number of grad school offerings, it just seems so huge at the moment, which again forces introducing such metrics on what is considered "success" in academia.

Of course researchers will want to do research. The question is, how do you select for those whose research will ever amount to something truly interesting, and avoid giving resources to those whose research will not? You can try looking at the past, and derive criteria for it from that, via machine learning for example, but I would be hesitant to leave something like that up to a machine. Also times change, so crite…

>The question is, how do you select for those whose research will ever amount to something truly interesting

You can't, as "truly interesting" is context dependent, changes over time and something that everyone deems as futile may become interesting - that's the point of research. You just increase the bar of entry to get people who work very hard and leave it to them to decide.

Re: Is novel research worth doing? Evidence from peer review

#30

Post by auggierose »

aleks224 wrote: post #29
auggierose wrote: post #22

Earlier quoted context omitted.

Of course researchers will want to do research. The question is, how do you select for those whose research will ever amount to something truly interesting, and avoid giving resources to those whose research will not? You can try looking at the past, and derive criteria for it from that, via machine learning for example, but I would be hesitant to leave something like that up to a machine. Also times change, so crite…

>The question is, how do you select for those whose research will ever amount to something truly interesting You can't, as "truly interesting" is context dependent, changes over time and something that everyone deems as futile may become interesting - that's the point of research. You just increase the bar of entry to get people who work very hard and leave it to them to decide.

Working very hard is not well-defined in this context. What does it even mean? You can work hard when you have a clear goal, let's say put those 10 barrels onto that truck over there. Or, let's write a new web browser within 2 years. Putting out 20 papers per year can also be considered working very hard.

You don't want people who work very hard. You want people who will EVENTUALLY put out new, original, and truly interesting research. HOW they do this is not up to you.

And there is a difference between truly interesting research, and just busy work research. It's not that easy to identify truly interesting research without the benefit of hindsight. It is somewhat easier to identify busy work research for an objective subject matter expert (but of course this is not 100% either, and personal preferences can definitely cloud the experts judgement).

Post reply on HN