'Not of faculty quality': How Penn mistreated Katalin Karikó

When someone is a PhD, and you know this, and you choose to use ms/mr/mrs instead of dr (or omitting an honorific altogether, the most common, unobjectionable choice), it can easily be interpreted as condescension.

I knew there was trouble as soon as I saw "Nobel Prize [winner]" and "Adjunct professor" in the same sentence. What's it take to get tenure track these days? But she mentions the rate of publications. Kps noted that Peter Higgs said something similar as well. There's many others too! Turing prize winner Hinton had this to say about ML and I couldn't agree more

> One big challenge the community faces is that if you want to get a paper published in machine learning now it's got to have a table in it, with all these different data sets across the top, and all these different methods along the side, and your method has to look like the best one. If it doesn’t look like that, it’s hard to get published. I don't think that's encouraging people to think about radically new ideas.

> Now if you send in a paper that has a radically new idea, there's no chance in hell it will get accepted, because it's going to get some junior reviewer who doesn't understand it. Or it’s going to get a senior reviewer who's trying to review too many papers and doesn't understand it first time round and assumes it must be nonsense. Anything that makes the brain hurt is not going to get accepted. And I think that's really bad.

Or from Bengio

> In the rush preceding a conference deadline, many papers are produced, but there is not enough time to check things properly and the race to put out more papers (especially as first or equal-first author) is humanly crushing. On the other hand, I am convinced that some of the most important advances have come through a slower process, with the time to think deeply, to step back, and to verify things carefully. Pressure has a negative effect on the quality of the science we generate. I would like us to think about Slow Science (check their manifesto!).

> Students sometimes come to me two months before a deadline asking if I have ideas of something which could be achieved in two months.

I'm sure you can find one from LeCun too (drop it if you have it) and we have the 3 godfathers of ML. But as someone finishing my PhD, I'm utterly convinced that the whole process is psychotic and anti-scientific. I have written many rants on HN about this so what's another? Here's how I see it, and what I've been coining as Goodhart's Hell because the idea is more abstract that ML publishing or even academic publishing. There's just a huge fucking irony that this happens in ML.

It is Goodhart's Hell because everything in our world has become about easy to use metrics and bending over backwards to meet those metrics. There is not just a lack of concern about if the metric aligns with our intended goals, but an active readiness to brush off any concerns. We as a modern world just fucking embraced metric hacking as the actual goal. In ML we see this, as Hinton mentions, with benchmarkism with just trying to get top scores. But you need several (fwiw, I've held a top spot for over a year now on a popular generative dataset but the work remains unpublished because I don't have enough compute to tune other datasets. Reviewers just ask for more but not justify the ask by how another dataset says more). This is an insane world, especially as we've been degrading our statistical principles. The last 5+ years no one uses a validation set for classification but rather tunes their fucking hyperparameters on test set results. Generative models frequently measure metrics against the train set and don't have a test set! A true, honest to god, hold out set essentially doesn't exist (we might call it "zero shot", which is inaccurate, or "OOD"...). ML work has simply become a matter of compute. Like Higgs said, you need to publish fast, but these days top companies are asking for 5+ papers at a top conference for a newly minted PhD. I'm sorry, good work takes time. All this on top of several consistency experiments that demonstrate that reviewers are simply reject first ask questions later. Which why shouldn't they be? No one checks a reject and doing so increases the odds your work gets in since it's a zero sum game.

And in honesty, I don't see how conferences and journals are anything but fraud. Not in the sense that works in there are untrue (though a lot are and a lot more are junk. Regardless of field), but in the economic operation. The government and universities (double dipping on that gov money) pay for these to exist. Universities pay researchers to produce work. Researchers send to venues (journals/conferences). Researchers review other works submitted to the venue for no pay (so Uni pays). 80% of work gets rejected, and goes through the process again. And after all that, the only meaningful thing accomplished is that the university has a signal that the work that their researchers did is "good." Because the venue gets copyright ownership over the paper, which the university must now pay for to access (the "official" version, "preprints" are free). I'm sorry, but citation count is a bad metric but far more meaningful than venue publication and it's fucking free. Why don't we just fucking publish to OpenReview? The point of publishing is to communicate our work, nothing more nothing less. OR gives you hosting like arxiv but also comments and threads (and links to github). Do we need anything else? I mean no review can actually determine if a paper is valid or good work. But we forget that the world isn't binary, it's tertiary: True, False, Indeterminate (thanks Godel, Turing, and Young). In reviewing we do not have access to the "True" side, just as we don't have access to that in science in general. We do not know where the "True" direction points, but we know how to move away from the "False" and "Indeterminate" directions. That's why there's that famous substack named that way or Isaac Asimov's famous Relativity of Wrong paper[2]. We're not a religion here...

There is at least a few ways I know how to fight back. 1) Actually fucking review a work and do your god damn job. Your job isn't to be a filter, it is to earnestly read the work and to work with the authors to make it the best work it can be. Remember you're on the same side. 2) Simply don't review if you can't do #1. You're almost never required to and academic service isn't worth much, so why do it? 3) Flip the system on its head. Instead of concentrating on reasons to reject a paper (fucking easy shit right there), focus on reasons to accept a paper. Simply ask yourself "is there something __someone__ in the community would find useful here?" If yes, accept. Novelty doesn't exist in a world where we have 20k+ papers a year and produce works every few months. It's okay to move fast, but it's less novel and impactful, it's just closer to open science. Stop concentrating on benchmarks since if it's useful someone is going to tune the shit out of it anyways, benchmarks don't mean shit. These days benchmarks are better at showing overfitting than good results anyways (yes, your test loss can continue to decrease while you overfit).

[0] https://www.wired.com/story/googles-ai-guru-computers-think-...

[1] https://yoshuabengio.org/2020/02/26/time-to-rethink-the-publ...

[2] https://hermiene.net/essays-trans/relativity_of_wrong.html

I'm sick of lies and misrepresentations from people who clearly don't know what they're talking about, talking like schools are greedily taking away poor Petey PhD's hard-earned grant money.

First off: grants from most places factor in the administrative overhead. That is negotiated between the school and the grant org. For the NIH, it averages fifty percent. The school/university is very restricted in what they can bill a lab for; for example, I worked somewhere that we couldn't charge for storage because that would have violated NIH's rules on double-billing, because the storage cluster was paid for via administrative overhead.

Chances are when someone says "I got a $1M grant to study bubblegum's effects on the gall bladder", they actually got $1M plus another $500,000.

Second, that money isn't being greedily stolen. That overhead help pays for, directly or indirectly, things like (notice I said "like", because I am not an expert in the exact rules around what can and cannot be paid for via overhead):

* the building

* the real estate the building sits on

* the utilities to keep the building lit and comfortable (which in the case of life/bio/chemistry sciences can be an enormous challenge given how much airflow lab space needs, which is far greater than office airflow...and then there's biosafety / chemical hoods)

* security, both equipment and staff (which can be substantial if the university or school does biomedical research in any sensitive areas such as stem cells, animal research, infectious disease, etc). This includes monitoring for equipment failure (for example, sample storage systems often have dry contact alarm hookups so that if they fail, security or facilities finds out ASAP and can alert people)

* the utilities to power equipment, such as -80 freezers (just one of which can use more energy than a US household)...most of us would also go pale if we saw the power bill for some physics labs) and other "utilities" like vacuum, purified water, etc.

* construction, maintenance, cleaning...both staff and supplies

* grounds maintenance, everything from mowing the lawn to leaf and snow removal

* technology costs - telephone and networking infrastructure and staff, server admins for everything from websites to email to storage to computational clusters, desktop support staff

* business administration, which includes, but is a lot more than just, payroll/benefits/HR. Grant writing/administration is often its own entire department, because you need people who not only know how to submit the paperwork, but frankly, also follow faculty around badgering them to fix or submit paperwork on time - faculty are incredibly lazy about this.

* all the services the lab's grad students, staff, postdocs, and faculty use and don't think anything about, like shuttle busses, the library, and so on.

> I'm reading tea leaves here, but my guess as to why she was never hired is that she was deemed "unable to get grants".

Rest assured, this is exactly what happened. University administrators have no expertise, interest, or motivation to identify and invest in promising research direction - they outsource this task to funding agencies. The only signal universities are extremely skillful in reading is dollar amounts.

I do not necessarily criticize this setup. Think of a research university as a start-up accelerator of sorts. Its main task is to give resources to secure sources of funding, not provide funds themselves.

Next time I apply for a job, I'll write "Someday, I'm going to publish 3 papers that change how we understand physics" and I'm sure they'll hire me. Who wouldn't?