Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

No "Zero-Shot" Without Exponential Data

arxiv.org

Post reply on HN
Print view View original

11–20 of 123 posts

Re: No "Zero-Shot" Without Exponential Data

#11

Post by treyd »

cs702 wrote: post #2

This deserves to be on the front page. The authors ask whether image-to-text and text-to-image models (like CLIP and Stable Diffusion) are truly capable of zero-shot generalization. To answer the question, the authors compile a list of 4000+ concepts (see paper for details on how they compile the list of concepts), and test how well 34 different models classify or generate those concepts at different scales of pretra…

But is this some abstract truth of statistics or is this just a property of how these types of models work?

Re: No "Zero-Shot" Without Exponential Data

#13

Post by kolinko »

cs702 wrote: post #2

This deserves to be on the front page. The authors ask whether image-to-text and text-to-image models (like CLIP and Stable Diffusion) are truly capable of zero-shot generalization. To answer the question, the authors compile a list of 4000+ concepts (see paper for details on how they compile the list of concepts), and test how well 34 different models classify or generate those concepts at different scales of pretra…

Worth noting that this paper is about CLIP only, which is way simpler than llm architectures. (if I’m not mistaken)

Still, interesting approach and kind of confirms the experience of most people where clip models can recognize known concepts but struggle with novel ones.

Re: No "Zero-Shot" Without Exponential Data

#14

Post by cs702 »

pas wrote: post #10
stephc_int13 wrote: post #5

Quite a few people saw this coming. It is still early to tell if we reached AI winter again or not, but at least we can see that news are slowing down.

RAG seems to be all the rage. Not to mention the quest for the cooking up the correct cocktail of smaller MoE/ensemble models, and ... there's decades' worth of optimization work ahead (a few years of it seems to be already VC and edu grants funded), no?

I think the grandparent comment is about AI research, driven by the quest for AGI.

Incremental improvement of proven approaches, driven by profit motive, will surely continue regardless of whether there is an AI winter or not.

Re: No "Zero-Shot" Without Exponential Data

#15

Post by crote »

This feels like the worst possible outcome of the current AI hype.

We've essentially been ripping off the entire internet and feeding it to the models already, spending many billions of dollars in the process. It's pretty much the largest possible dataset you can currently get, and due to the ever-increasing and now rapidly accelerated AI poisoning of the internet most likely the largest possible dataset which will ever exist.

All that and all we're getting out of it is not-entirely-useless but still quite crappy AI? We would've been better off if we had never done this.

Re: No "Zero-Shot" Without Exponential Data

#16

Post by some_random »

crote wrote: post #15

This feels like the worst possible outcome of the current AI hype. We've essentially been ripping off the entire internet and feeding it to the models already, spending many billions of dollars in the process. It's pretty much the largest possible dataset you can currently get, and due to the ever-increasing and now rapidly accelerated AI poisoning of the internet most likely the largest possible dataset which will e…

Quite crappy? It seems to me like the current SOTA is working plenty good enough for most use cases and like the highest impact (practical) way to improve right now is going to be advancements in domain knowledge acquisition and retention.

Re: No "Zero-Shot" Without Exponential Data

#17

Post by nkozyra »

pas wrote: post #10
stephc_int13 wrote: post #5

Quite a few people saw this coming. It is still early to tell if we reached AI winter again or not, but at least we can see that news are slowing down.

RAG seems to be all the rage. Not to mention the quest for the cooking up the correct cocktail of smaller MoE/ensemble models, and ... there's decades' worth of optimization work ahead (a few years of it seems to be already VC and edu grants funded), no?

RAG takes the current limit of LLM and focuses it on specific problems using custom data. It's not exactly magic, it just finds a way to produce something tangible and usable from an otherwise broadly focused model.

It's the rage because it's a way to practically _do work_ from LLMs that generally provide wow from conversationally accurate, often factually accurate responses.

Re: No "Zero-Shot" Without Exponential Data

#18

Post by nkozyra »

My biggest worry is the idea of generating data as training data. We're obviously already unwittingly doing this, but once someone decides to augment low-volume segments of the dataset with generative input, we're going to start getting some really crappy feedback loops.

Re: No "Zero-Shot" Without Exponential Data

#19

Post by greenavocado »

crote wrote: post #15

This feels like the worst possible outcome of the current AI hype. We've essentially been ripping off the entire internet and feeding it to the models already, spending many billions of dollars in the process. It's pretty much the largest possible dataset you can currently get, and due to the ever-increasing and now rapidly accelerated AI poisoning of the internet most likely the largest possible dataset which will e…

I'm getting a heck of a lot of useful work out of these so-called useless models

Re: No "Zero-Shot" Without Exponential Data

#20

Post by nopeNopeNooope »

nkozyra wrote: post #18

My biggest worry is the idea of generating data as training data. We're obviously already unwittingly doing this, but once someone decides to augment low-volume segments of the dataset with generative input, we're going to start getting some really crappy feedback loops.

[dead]
Post reply on HN