Live data from Hacker News

Queueing to publish in AI and CS

damaru2.github.io

31–40 of 59 posts

Re: Queueing to publish in AI and CS

#31

The real issue is careers valuing where papers are published over what they contribute, and incentives won’t change until hiring and funding reward depth over volume.

Depends on who is doing the "careers valuing" and how closely they're looking. At a coarse level, especially for jobs in industry, venue is a pretty simple (but obviously imperfect) indicator for quality. If you've managed to publish one or more papers at the most selective venues (esp. as main author), then I would assume there's a decent chance you are good at research, even if I don't know anything about the subfield you work on. As a further indicator, the number of citations is also a noisy but easy to check proxy for "impact".

But for academic or other high-level research jobs, whoever is doing the valuing is going to look at a lot more than just the venue.

Re: Queueing to publish in AI and CS

#32

The real issue is careers valuing where papers are published over what they contribute, and incentives won’t change until hiring and funding reward depth over volume.

It is difficult to judge the impact for most papers in the first few years. Additionally, impact is influenced by visibility, and when there’s a deluge of papers, guess what’s a really good way to make your paper stand out? Yup, publication in a good venue.

Re: Queueing to publish in AI and CS

#33

The real issue is careers valuing where papers are published over what they contribute, and incentives won’t change until hiring and funding reward depth over volume.

Depends on who is doing the "careers valuing" and how closely they're looking. At a coarse level, especially for jobs in industry, venue is a pretty simple (but obviously imperfect) indicator for quality. If you've managed to publish one or more papers at the most selective venues (esp. as main author), then I would assume there's a decent chance you are good at research, even if I don't know anything about the subfi…

> But for academic or other high-level research jobs, whoever is doing the valuing is going to look at a lot more than just the venue.

Depends on where. In some countries (e.g. mine, Spain), the notion that evaluation should be "objetive" leads to it degenerating into a pure bean-counting exercise: a first-quartile JCR indexed journal paper is worth 10 points, a top-tier (according to a specific ranking) conference paper is worth 8 points, etc. In some calls/contexts there is some leeway for evaluators to actually look at the content and e.g. subtract points for salami slicing or for publishing in journals that are known to be crap in spite of good quartile, but in others it's not even allowed to do that (you would face an appeal for not following the official scoring scale).

Re: Queueing to publish in AI and CS

#34

As pointed out in the comments, conferences have physical limits on the number of presenters (even in poster sessions), so the number of accepted papers at top conferences will likely stop growing at some point. (Perhaps it should have been capped already.) This will probably lead to new conferences appearing to satisfy the "demand", but more conferences of varying quality is probably better than reviewing at top con…

Creating a new, first-grade conference from scratch is hard. NeuriPS, CVPR and ICML are solid brands that took decades to build.

ICLR managed it quite fast.

Re: Queueing to publish in AI and CS

#35

As pointed out in the comments, conferences have physical limits on the number of presenters (even in poster sessions), so the number of accepted papers at top conferences will likely stop growing at some point. (Perhaps it should have been capped already.) This will probably lead to new conferences appearing to satisfy the "demand", but more conferences of varying quality is probably better than reviewing at top con…

Im on an academic foray to an R1 right now. One of the big items the PIs are drilling is the one paper per year mantra.

Many papers today are representing what would have happened via open source repos in yesteryear. Meaning that there is a lot of work which is useful to someone and having peer reviewed benchmarks etc. is useful to understand whether those people should care. The weakness is that some of this work is the equivalent of shovelware.

Re: Queueing to publish in AI and CS

#36

The real issue is careers valuing where papers are published over what they contribute, and incentives won’t change until hiring and funding reward depth over volume.

This has truth but is not the full story. It gets your foot in the door, but I think junior researcher overinflate the importance of numbers. At the end of the day when you interview, your future boss will ask about your research, and it doesn't matter all that much how many papers they are and how many are top published. Yes you should have some top conf papers, but one great Arxiv paper that is actually used by others can have more value than 4 meh papers that slipped through the reviews.

Remember that nobody is a passive actor in the system. Everyone sees the state of conferences and review randomness and the gaming of the system. Senior researchers are area chairs and program chairs. They are well aware and won't take paper counts as the only signal.

Again, papers are needed, but it's really not the only thing.

Re: Queueing to publish in AI and CS

#37

So research is also now a race to the bottom? Come join me at my private research coop, where we emphasize quality over quantity and don't give a shit about whats perceived as cool...

It's part of the wider credential inflation trend and increased grind-competition overall.

Re: Queueing to publish in AI and CS

#38

There's two main problems. First, the field is too saturated -- too many people publishing papers. Second, the reviewers are the same people that publish papers -- it's a zero sum game (regardless of acceptance criteria). Submission numbers this year have been absolutely crazy. I honestly don't think it can be solved.

The exponential scaling messes with the previous more honor-based gentleman-like reputation-based everyone-knows-everyone situation. That model has its own problems but different. Today it's more cutthroat, grind-competitive and nobody has proper incentives.

It's like working long years in a family-sized company vs job hopping between megacorps.

The actual science takes the backseat. Nobody has time to just think, you must pump out the next paper and somehow get it through peer review. As a reviewer, you don't get much out of reviewing. It used to be exciting to look at new developments from across the field in ones review stack. Today it's mostly filled with the nth resubmission of something by someone in anxious hurry to just produce something to tick a box. There is no cost to just submitting massive amounts of papers. Anyway, so it's not fun as a reviewer either, you get no reward for it and you take time away from your own research. So people now have to be forced to review in order to submit. These forced reviews do as good a job as you expect. The better case is if they are just disinterested. The worse is if they feel you are a dangerous competitor. Or they only try to assess whether you toiled "hard enough" to deserve the badge of a published paper. Intellectual curiosity etc. have taken the back seat. LLMs just make it all worse.

Nobody is truly incenticized to change this. It's a bit of a tragedy of the commons situation. Just extract as much as you can, and fight like in a war.

It's also like moving from a small village where everyone knows everyone to a big metropolis. People are all just in transit there, they want to take as much as possible before moving on. Wider impacts don't matter. Publish a bunch of stuff then get a well paying job. Who cares that these papers are not quite that scientifically valuable? Nobody reads it anyway. In 6 months it's obsolete either way. But in the meantime it increased the citation count of the PI, the department can put it into their annual report, use the numbers for rankings and for applying for state funding, the numbers look good when talking to the minister of education, it can be also pushed in press releases to do PR for the university which increases public reputation etc. The conferences rise on the impact tables because of the immense cross-citation numbers etc. The more papers, the more citations, the higher the impact factor. And this prestige then moves on to the editors, area chairs etc. and it looks good on a CV.

It mirrors a lot of other social developments where time horizons have shrunk, trust is lower, incentives are perverse and nobody quite likes how it is but nobody has unilateral power to change things in the face of institutional inertia.

Organizing activity at such scale is a hard problem. Especially because research is very decentralized by tradition. It's largely independent groups of 10-20 people centered around one main senior scientist. The network between these is informal. It's very different than megacorps. Megacorps can go sclerotic with admin bloat and paralyzed by middle manager layers. But in the distributed model, there is minimal coordination, it's an undifferentiated soup of these tiny groups, each holding on to their similar ideas and rushing to publish first.

Unfortunately, research is not like factory production, even if bureaucrats and bean counters would wish so. Simply throwing more people at it can make negative impact, analogous to the mythical man-month.

Re: Queueing to publish in AI and CS

#39
Honest question: why not charge a fee per submission or per review?

Or if the problem is bad papers, a fee that is returned unless it’s a universal strong reject.

Or if you don’t want to miss the best papers, a fee only for resubmitted papers?

Or a fee that is returned if your paper is strong accept?

Or a fee that is returned if your paper is accepted.

There’s some model that has to be fair (not a financial burden to those writing good papers) and will limit the rate of submissions.

Thoughts?

Re: Queueing to publish in AI and CS

#40

Honest question: why not charge a fee per submission or per review? Or if the problem is bad papers, a fee that is returned unless it’s a universal strong reject. Or if you don’t want to miss the best papers, a fee only for resubmitted papers? Or a fee that is returned if your paper is strong accept? Or a fee that is returned if your paper is accepted. There’s some model that has to be fair (not a financial burden to…

Fees mean very little.

Go for the jugular. Impact the career of people putting out substandard papers.

Come up with a score for "citation strength" or something.

Any given bad actor with too many substandard papers to his/her credit begins to negatively impact the "citation strength" of any paper on which they are a co-author. Maybe even negatively impacting the "citation strength" of papers that even cite papers authored or co-authored by the bad actor in question?

If, say, the major journals had a system like this in place, you'd see everyone perk up and get a whole lot less careless.

Post reply on HN