Behind the scenes: the struggle for each paper (2021)

This is an interesting note. We're talking about a student from one of the top CS schools (UIUC) and applying to another top school (UW). If you think about this a bit carefully, the paper being published did not change who he was or his capabilities, it was simply a difference in measured (distinct from measurable) signal.

It's incredible how many extremely noisy signals we use in academia but act as if we use a clear meritocracy. The review process is extremely noisy itself, with computer science in particular being generally more noisy given its preference of conferences over journals. I'm glad Jeff mentions people and opportunities, and it reminds me of the old saying about there being no self made man. But I think this is a very clear example of a instance where we need to think harder and more carefully. Counterfactually, it is almost certain that had that paper been rejected, but all else stays the same (i.e. getting into UW), his success story would also not change. Signals are definitely hard to measure and certainly schools are getting a lot of applicants, so I don't blame anyone for doing this, but I think it is incredibly important to remember these counterfactuals. To remember that metrics are guides and not causal variables themselves. Because there's a great irony in that metrics destroy meritocracies.

I have always thought this was a bit silly and that it creates really weird effects that also decrease readability. An interesting point is that reviewers are not required to read the appendix of works. So everything is required to be in the front matter. This is a bit silly when we do things like research graphics or do generative works and such. You want to include images and samples but then your space is eaten up. What if you want to discuss analysis on those images and explore some? You could easily do this on a blog but you're forced to throw this into the appendix. But then a reviewer can ask a question that's explained there and your work can still get rejected because it isn't in the front matter. Another weird incentive is that people end up padding works to fit page limits. This is because if you turn in a shorter paper reviewers will frequently reject your work the same way your boss might not think you're working if they don't see you at your desk.

We live in the 21st century and we still publish like it's the 15th. Computers gave us the ability to embed images, which is why there are so many more graphs and charts now, and it's not like more pages cost more. So just remove it. Some papers should be only a few pages and there's nothing wrong with that. Some papers should be far larger and there's nothing wrong with that. It's just weird to set these up considering they were likely created under other constraints but momentum continued and we back justify the continued decisions (there is something to be said about readability, but that can just be a reason to reject).

Side note: CS groups typically publish in conferences

edit: i'm from slovenia, univeristies here are "autonomous" (not a direct part of the government.. except for being government funded), and they decide all the internal rules themselves.

[1] https://distill.pub/2021/distill-hiatus/

I agree that you need to use metrics. But we need to be clear that metrics are not enough and very incomplete themselves. With something like admissions, I'm not sure there's anything except noisy signals and the strongest one by far is the interview.

> Since someone incapable of writing such a paper is probably unsuited for a PhD,

I very much disagree with this. The explicit purpose of schooling is to train people. Many undergrads are not going to have the opportunities to publish. It is not hard to train someone to write something publishable and this is not something I would be much concerned with myself given how much writing they're going to be doing over the next few years. The far more valuable skills are in being able to perform research which is quite ambiguous (there are at least 2 ways to read this sentence and both are correct: research type v measure). Your first 2 years of your PhD are almost exclusively training, with more class work and learning how to begin research. This isn't a job you're applying for, it is a training program.