No "Zero-Shot" Without Exponential Data

This deserves to be on the front page.

The authors ask whether image-to-text and text-to-image models (like CLIP and Stable Diffusion) are truly capable of zero-shot generalization.

To answer the question, the authors compile a list of 4000+ concepts (see paper for details on how they compile the list of concepts), and test how well 34 different models classify or generate those concepts at different scales of pretraining, from ~3M to ~400M samples.

They find that model performance on each concept scales linearly as the concept's frequency in pretraining data grows exponentially, i.e., the rarer the concept the less likely it is actually/properly learned -- which implies there is no "zero-shot" generalization.

The authors also release a long-tail test dataset that they cleverly name the "Let it Wag!" benchmark to allow other researchers to see for themselves how current models perform on the long tail of increasingly rare concepts.

Go read the whole thing, or at least the introduction. It's clear, concise, and well-written.

My long-standing observation has been that while nature may abhor a vacuum, she also really, really loves sigmoids.

That performance vs. training data is not linear, but logarithmic, doesn't exactly come as a surprise.

Can one upweight the known-rare concepts in the training set? But then which are rare and how should we identify the rare ones worth knowing?

ML is funny. It's predictably going to run into the Education field.

RAG seems to be all the rage. Not to mention the quest for the cooking up the correct cocktail of smaller MoE/ensemble models, and ... there's decades' worth of optimization work ahead (a few years of it seems to be already VC and edu grants funded), no?