Probably the only subfield of Computer Science, maybe even academic research in general, where citations number is the most poor metric of paper quality. The whole situation is more akin to mass media reporting the same breakthrough in cancer cure research (in mice).
The Most Cited AI Papers in 2022
11–20 of 42 posts
Re: The Most Cited AI Papers in 2022
#12Incredible how much benefit alphafold has brought. And all of that from a less than 100million parameters model. I might be dumb but could they scale it up and make an alphafold 3 with maybe like 10bln params? Would it be a lot better assuming the same training effort is put into it? If it does, can't biotech companies just go nuts and make a 100bln params internal model and have all the protein structures they want?
Has there been much research into the idea of distributing these large models across many heterogeneous machines?
I'm wondering if there could be a path towards a mix of alphafold and folding@home, with donated idle compute resources being used to train/run the models.
Designing for that sort of fragmentation could also make it easier to slowly run oversized models on local machines with swapped memory.
Re: The Most Cited AI Papers in 2022
#13These tweet counts are not accurate. I searched for one with 0 tweets and found plenty of tweets mentioning it.
Re: The Most Cited AI Papers in 2022
#14Did nothing of note happen outside of deep learning the whole year?
In AI? No, the other stuff all happened in the 70s
Re: The Most Cited AI Papers in 2022
#15Did nothing of note happen outside of deep learning the whole year?
Of course interesting things happen outside of mainstream deep learning. The problem is that sorting by citation count is a terrible way to look broadly at research, because mainstream deep learning is so hyped up that there are just way more people working in that area than in other areas, and with more people come more citations. It's a shame because I think that other areas offer much more interesting and technica…
Disagree, or can you name some recent AI developments outside of DL that are anywhere near as groundbreaking?
Re: The Most Cited AI Papers in 2022
#16Incredible how much benefit alphafold has brought. And all of that from a less than 100million parameters model. I might be dumb but could they scale it up and make an alphafold 3 with maybe like 10bln params? Would it be a lot better assuming the same training effort is put into it? If it does, can't biotech companies just go nuts and make a 100bln params internal model and have all the protein structures they want?
Has there been much research into the idea of distributing these large models across many heterogeneous machines? I'm wondering if there could be a path towards a mix of alphafold and folding@home, with donated idle compute resources being used to train/run the models. Designing for that sort of fragmentation could also make it easier to slowly run oversized models on local machines with swapped memory.
Using any current architecture it is infeasible to do backprop (training) due to the massive communication requirements. Inference is possible to do in sharded way but still not as practical as just loading the model weight that are needed on-demand from disk; still a distributed job queue being processed may be beneficial depending on the throughput/costs required by researchers.
Re: The Most Cited AI Papers in 2022
#17Did nothing of note happen outside of deep learning the whole year?
The most interesting AI thing happening outside deep learning is zero-knowledge techniques.
They are very nascent though and it isn't surprising they aren't yet highly cited since it's quite an achievement getting anything working at all still.
Re: The Most Cited AI Papers in 2022
#18>An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
>https://arxiv.org/abs/2010.11929
Very interesting. I didn't think of connecting these two dots. Let's see if it can be applied to other CV tasks, like object detection.
Re: The Most Cited AI Papers in 2022
#19I'm getting wildly different citation counts for some of the listed AI papers. For example, the paper "ColabFold: making protein folding accessible to all" is listed as having 1162 citations. I'm seeing that it was cited only by 899 publications on Scite: https://scite.ai/reports/colabfold-making-protein-folding-ac... I'm wondering if Google Scholar is overestimating or Scite is underestimating.
Semantic Scholar has 1,111. [0] I tend to trust Semantic more than GS. GS tends to overestimate. For example on GS I have 164 citations on one paper and semantic says 150. FWIW Scite says 49.[1] [0] https://www.semanticscholar.org/paper/ColabFold%3A-making-pr... [1] I'll note that this paper is an arxiv paper and has not been accepted at a conference but I'd also argue that conference acceptance means little in ML. I…
Please explain!
Re: The Most Cited AI Papers in 2022
#20Earlier quoted context omitted.
Semantic Scholar has 1,111. [0] I tend to trust Semantic more than GS. GS tends to overestimate. For example on GS I have 164 citations on one paper and semantic says 150. FWIW Scite says 49.[1] [0] https://www.semanticscholar.org/paper/ColabFold%3A-making-pr... [1] I'll note that this paper is an arxiv paper and has not been accepted at a conference but I'd also argue that conference acceptance means little in ML. I…
Please explain!
Facebook wav2vec2, for example, was also just uploaded to arxiv and then they formally published it months/years later after it was widely used.
The traditional publication flow just isn't useful if a runnable demo on HuggingFace explains your work way better than 3 pages of formulas.