Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Is novel research worth doing? Evidence from peer review

pnas.org

Post reply on HN
Print view View original

31–40 of 54 posts

Re: Is novel research worth doing? Evidence from peer review

#31

Post by einpoklum »

nerdponx wrote: post #15
rscho wrote: post #8

Earlier quoted context omitted.

As a clinical researcher, I wholeheartedly agree. Even worse, meta analysis is comparatively so well cited (even shitty ones) that doing anything else, such as original research, will be suboptimal since the reduced time investment allows you to publish nonstop. Truly, meta analysis has become a blood-ducking parasite of medical science. I know many young researchers who've specialized in meta analysis for that reaso…

Isn't there tremendous real value in meta-analysis? The problem seems to lie in how credit is allocated (via citations).

> Isn't there tremendous real value in meta-analysis?

There's also tremendous value in doing the dishes and mopping the floor. Without it we'd get diseases and live in filth. And when we do this, we should be respected and appreciated. But - not as brilliant trail-blazers.

Re: Is novel research worth doing? Evidence from peer review

#32

Post by asdff »

PlasmonOwl wrote: post #3

Bullshit. Journals select for novelty, this is true. But this is novelty in a trifling fashion, like teaching a dog to roll over. The entire system, at least in the UK is set up to actively discourage risky research entirely. Doing something truly new is discouraged unless you are truly a senior or leader. By then, you've probably lost most of your imagination anyway.

Certain fields are a lot easier to do novel work in. For a grad student in computer science/computational biology, you practically don't need any grant funding at all. You can download public data, work on the university hpc for basically free, publish whatever the hell you want, and pay your stipend with a TAship. Others, where you have to regularly buy reagents or maintain an animal model, you pretty much need grant support to buy these things.

Re: Is novel research worth doing? Evidence from peer review

#33

Post by guhidalg »

auggierose wrote: post #30
aleks224 wrote: post #29

Earlier quoted context omitted.

>The question is, how do you select for those whose research will ever amount to something truly interesting You can't, as "truly interesting" is context dependent, changes over time and something that everyone deems as futile may become interesting - that's the point of research. You just increase the bar of entry to get people who work very hard and leave it to them to decide.

Working very hard is not well-defined in this context. What does it even mean? You can work hard when you have a clear goal, let's say put those 10 barrels onto that truck over there. Or, let's write a new web browser within 2 years. Putting out 20 papers per year can also be considered working very hard. You don't want people who work very hard. You want people who will EVENTUALLY put out new, original, and truly in…

You need people who got excellent grades in their undergrad program, who have somehow demonstrated that they like the field they’re going into academia for (as in clubs, extracurriculars, and whatnot), and who peers recommend as being likely to do novel research. The last one requires, gasp, talking to the applicant and seeing if they’re full of shit.

Re: Is novel research worth doing? Evidence from peer review

#34

Post by godelski »

Yes and no, but first we need to define novelty.

I'm in ML and all the academic issues are on full display here (comes with any hyped research area). Reviewer 2 will always knock you on novelty no matter what you do. That's because all research is to some extent derivative. As it should be. We shouldn't be reinventing the wheel. But on the other hand, all research is incremental (especially in a publish or perish system. You can't publish breakthroughs frequently). Leaps and bounds don't happen often and they generally do either by accident (someone doing something dumb) or because someone was spending a lot of time on that area (likely not publishing in main track or at all). But novelty is also hard to measure post hoc. Most things are non-obvious a priori but are obvious post hoc. Reviewers go after these all the time and all it encourages is researchers to make their works convoluted. I've literally been told that papers I've written are too simple and straight forward. I've also seen those papers get better reviews after complexifying the text. I don't feel good about this btw.

But I want to give a perfect example of all this in ML. Diffusion models. The stochastic noise process has been around for decades. People have been applying them to generative models for awhile too. But they were basically unknown until they beat GANs at image generation and within a few short months they are a household name (even more than GANs). This looks like a bigger breakthrough than it is because they were flying under the radar for so long and really just one team spent a lot of time with them AND had access to A LOT of compute resources. But still, I think this should make you think about how novelty works and if you're reviewing, maybe don't squash papers because you think they are dead ends. Diffusion was a "dead end" until it wasn't.

Re: Is novel research worth doing? Evidence from peer review

#35

Post by sdenton4 »

jraph wrote: post #13
bjornsing wrote: post #12

> We find that higher novelty is consistently associated with higher acceptance; submissions in the top novelty quintile are 6.5 percentage points more likely than bottom quintile ones to get accepted. Ok, so “novel” research is 6.5% more likely to be accepted if submitted, but how likely is it to even be submitted? I would argue that doing “novel” research is much riskier at the earlier stages, and that you need to…

I've heard some researcher saying that a way to manage this is to dedicate a part of your time to risky research, and the rest to less risky research. So you still publish, but also get to explore "interesting" stuff (that also needs to be researched). You are covered by your less risky activity. So, a single number 20%-30% of the time, and a color 70%-80% of the time. And your risky activity could become your comfor…

Don't forget time for teaching, avoiding students (tried to say 'advising' but autocomplete had a better word choice), and grant applications...

Re: Is novel research worth doing? Evidence from peer review

#36

Post by SkyMarshal »

psychomugs wrote: post #28

I think often of this jab by Rodney Brooks on my home field of human-robot interaction [1]: "A group of people who have never used a computer are recruited for an experiment, using variations of Unix commands to manipulate files from a command line interface. The users have never heard of computer files before and they don’t in any sense have the utility of files internalized. The experiment is to determine whether s…

There's an interesting tension in academia between the need to publish regularly, the fact that deep research often needs longer work periods before achieving impactful results, and the fact that sometimes quantity of output produces better net results.

I wonder if this could be alleviated by incentivizing the publishing of papers on work with no statistically significant results, as has been discussed here and elsewhere. That might increase quantity of output, while building a useful database of what not to spend time on.

Re: Is novel research worth doing? Evidence from peer review

#37

Post by godelski »

ThereIsNoWorry wrote: post #17
PlasmonOwl wrote: post #3

Bullshit. Journals select for novelty, this is true. But this is novelty in a trifling fashion, like teaching a dog to roll over. The entire system, at least in the UK is set up to actively discourage risky research entirely. Doing something truly new is discouraged unless you are truly a senior or leader. By then, you've probably lost most of your imagination anyway.

That was my conclusion after publishing a paper during my BSc. Academia is full of petty politics. Professors that gatekeep whole fields on a "like"-basis, reviewers arguing with "not enough experiments done" and questioning uniqueness and novelty in ways that just don't make any sense (to me). It is a very bureaucratic process that cannot possibly be good for innovation and progress. Then you get incentives of field…

Dog whistles in academic reviewing; phrases that can be valid but are typically being used in an invalid way:

- Not novel: Novelty is ill-defined. Most journals and conferences have guidelines on this that come down to "will someone find it useful" which should make "not novel" a rare review, not the most common. Additionally, what's obvious post hoc is not obvious a priori (I actually saw an AC override reviewers because of this!). If something is well written it also comes across as easy and simple. Lack of novelty could be a true lack of novelty or just an indication of a well written paper.

- Incremental: All research is incremental. Research can be too incremental (not enough work done) but this is never expanded upon in those types of reviews. This kind of statement is meaningless in isolation. Incrementalism will always exist with a publish or perish paradigm. One cannot publish breakthroughs year over year. Those take significant periods of time or a lucky break. Neither is a common occurrence, by definition.

- Not enough experiments: "Money is all you need" has become a popular phrase. There may not be enough experiments and this can be true but perfection is the enemy of good. Especially in research, where the experiment space is non-exhaustive. Can a reviewer be satisfied in a finite amount of time and with finite resources?

I've started to change my mind about where blame falls for all this. It starts with the reviewers, but if this becomes the norm then the system is at fault. It has not removed the rotten apples and thus let the barrel spoil. All while we've had decades of discussion about this happening. I think a lot falls on the Area Chairs and Metareviewers, as they should be preventing these types of reviews to pass. But the whole incentive structure of publishing is wrong. There is high pressure to reject and zero pressure to let papers through. This especially hurts our junior researchers (grad students) as it becomes a lottery process for determining if they can graduate. After all, we can't have any wizards without any noobs.

Re: Is novel research worth doing? Evidence from peer review

#38

Post by rscho »

CrazyStat wrote: post #26
isaacfrond wrote: post #5

Earlier quoted context omitted.

[citation needed]

I can confirm the GP matches my experience in academia (PhD, published a paper based on my dissertation, left academia shortly after). There is immense pressure for early career researchers to work on low-hanging fruit that will give easy guaranteed publications: make a minor change to previous work and write a paper about it; write a paper about what other people wrote papers about (either a review or a meta-analysi…

Habit is not the problem. Once you get tenure, a good part of your job is to ensure that the younger generation in your lab/uni publishes well, i.e encourage them to pursue the next low-hanging fruit... So no, you do not in fact have more freedom.

Re: Is novel research worth doing? Evidence from peer review

#39

Post by CrazyStat »

rscho wrote: post #38
CrazyStat wrote: post #26

Earlier quoted context omitted.

I can confirm the GP matches my experience in academia (PhD, published a paper based on my dissertation, left academia shortly after). There is immense pressure for early career researchers to work on low-hanging fruit that will give easy guaranteed publications: make a minor change to previous work and write a paper about it; write a paper about what other people wrote papers about (either a review or a meta-analysi…

Habit is not the problem. Once you get tenure, a good part of your job is to ensure that the younger generation in your lab/uni publishes well, i.e encourage them to pursue the next low-hanging fruit... So no, you do not in fact have more freedom.

That's a fair point. I jumped ship before I got that far, obviously. From my perspective as a student it looked a lot like habit on the part of the senior/tenured people in the department, but I should not have been so quick to assert that.

Re: Is novel research worth doing? Evidence from peer review

#40

Post by auggierose »

guhidalg wrote: post #33
auggierose wrote: post #30

Earlier quoted context omitted.

Working very hard is not well-defined in this context. What does it even mean? You can work hard when you have a clear goal, let's say put those 10 barrels onto that truck over there. Or, let's write a new web browser within 2 years. Putting out 20 papers per year can also be considered working very hard. You don't want people who work very hard. You want people who will EVENTUALLY put out new, original, and truly in…

You need people who got excellent grades in their undergrad program, who have somehow demonstrated that they like the field they’re going into academia for (as in clubs, extracurriculars, and whatnot), and who peers recommend as being likely to do novel research. The last one requires, gasp, talking to the applicant and seeing if they’re full of shit.

That's already done now. People in Academia have excellent grades, and they like what they do. How would peers know about their ability to do novel research, not having done any themselves?
Post reply on HN