Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

No "Zero-Shot" Without Exponential Data

arxiv.org

Post reply on HN
Print view View original

71–80 of 123 posts

Re: No "Zero-Shot" Without Exponential Data

#71

Post by crote »

HarHarVeryFunny wrote: post #54
crote wrote: post #15

This feels like the worst possible outcome of the current AI hype. We've essentially been ripping off the entire internet and feeding it to the models already, spending many billions of dollars in the process. It's pretty much the largest possible dataset you can currently get, and due to the ever-increasing and now rapidly accelerated AI poisoning of the internet most likely the largest possible dataset which will e…

Even if LLMs (pre-trained transformers) turn out to be a dead end as far as AGI goes, there are productivity applications for them, and perhaps just as importantly interesting insights/confirmations about how the mind works and directions for future AGI research/architectures. The use cases for LLMs will no doubt grow as hallucinations are reduced, and they gain planning/reasoning ability over next couple of years. I…

Oh absolutely, LLMs are already causing a slaughter in applications where quality doesn't really matter to the company, like customer service. With minor improvements they're going to be a serious problem for any junior developer/lawyer/journalist/reviewer/artist/whatever, and if they ever fix the hallucinations issue it's game-changing.

On the other hand, it's still a big "if" whether a general hallucinations solution exists, and in the meantime we're paying a pretty high price for it, as the entire internet is being flooded with absolute garbage. We risk getting stuck in a situation where there is no way to get new senior people because nobody hired them as juniors because AI is cheaper and good enough, and senior people are getting less and less productive due to them being unable to use websites like StackOverflow as reference material. That's a pretty high price for a tiny gain, if you ask me.

Re: No "Zero-Shot" Without Exponential Data

#72

Post by ijustlovemath »

swatcoder wrote: post #34
loudmax wrote: post #26

Earlier quoted context omitted.

I think that depends what your expectations are, and what you mean by another AI winter. We're just scratching the surface of what's possible with the current state of the art. Even if there are no major advances or breakthroughs in the near future, LLMs and associated technologies are already useful in many use cases. Or close enough to useful where engineering rather than science will be sufficient to overcome many…

You're right, and I suspect the GP would agree with you about there being real engineering applications for LLMs, diffusion models, etc But I think the term "AI Winter" usually refers to the underlying research programme and the economics around it. Soaking up many billions of dollars of industry money and public grants on the pitch that AGI might be just around the corner, and then being unable to deliver on that pi…

The minor irony of this is that current efforts at AGI are focused on the data scaling laws, which we have showed no signs of slowing on, but if funding dries up these crazy expensive training runs wont be allowed anymore.

Re: No "Zero-Shot" Without Exponential Data

#73

Post by CuriouslyC »

stephc_int13 wrote: post #5

Quite a few people saw this coming. It is still early to tell if we reached AI winter again or not, but at least we can see that news are slowing down.

Definitely not an AI winter, since the tools are so useful. People who think AGI is right around the corner might be disappointed though, since we can't really even accurately define AGI.

Re: No "Zero-Shot" Without Exponential Data

#74

Post by a_wild_dandan »

Better title: Image classification models suck at identifying nouns that they've rarely seen.

Crucial context:

- They're only looking at image models -- not LLMs, etc

- Their models are tiny

- A "concept" here just means "a noun." The authors index images via these nouns.

- They didn't control for difficulty in visual representation/recognition of these exceptional infrequent, long-tail "concepts."

If I didn't know an object's label, I too would struggle to identify/draw it...

Re: No "Zero-Shot" Without Exponential Data

#75

Post by xboxnolifes »

loandbehold wrote: post #52
godelski wrote: post #50

I've always been rather upset that it's fairly common to train on things like LAION or COCO and then "zero shot" test on ImageNet. Zero shot doesn't mean a held out set, it means disjoint classes. You can't train on all the animals in the zoo with sentences and then be surprised your model knows zebras. You need to train on horses and test on zebras.

How would the model know what zebra was if it had never seen it? Same is true for humans.

When I was little, zebras were described to me as black and white stripped horses. Without even seeing one, I'm sure anything who has seen a horse could then merge those two concepts to create a close to accurate picture of what a zebra is.

If AI is supposed to resemble a human mind with ability to learn, then it must be able to learn from a blanker slate. You don't teach the human before it is born, and in this comparison an AI is born when you finish it's model and set its weights using the training set. If you test it with the training set, you aren't testing ability to comprehend, just regurgitate what it was born with

Re: No "Zero-Shot" Without Exponential Data

#76

Post by xboxnolifes »

dr_dshiv wrote: post #68
marcosdumay wrote: post #51

Earlier quoted context omitted.

"AI winter" was a political phenomenon that happened because the AI applications didn't hold water to the hype, and all of the gullible people that invested on them got burned out by the disparity. We are very clearly on that same path again. What leads to the conclusion that another winter is coming. But even the fact that people are talking about it is evidence it's not here yet, and as always with political phenom…

Except contemporary AI is useful on a day to day basis.

Being useful doesn't mean it met expectations.

Re: No "Zero-Shot" Without Exponential Data

#77

Post by xboxnolifes »

bilsbie wrote: post #21
nkozyra wrote: post #18

My biggest worry is the idea of generating data as training data. We're obviously already unwittingly doing this, but once someone decides to augment low-volume segments of the dataset with generative input, we're going to start getting some really crappy feedback loops.

Do you ever have novel ideas while walking or in the shower? Well, you’re learning from data you generate.

Even hear an idea from another person? You just trained one model with another.

Re: No "Zero-Shot" Without Exponential Data

#78

Post by ixaxaar »

hprotagonist wrote: post #6

My long-standing observation has been that while nature may abhor a vacuum, she also really, really loves sigmoids. That performance vs. training data is not linear, but logarithmic, doesn't exactly come as a surprise.

The physical world has limits, that's why sigmoids everywhere.

Re: No "Zero-Shot" Without Exponential Data

#79

Post by trgn »

stephc_int13 wrote: post #5

Quite a few people saw this coming. It is still early to tell if we reached AI winter again or not, but at least we can see that news are slowing down.

Commercial adoption has only started to ramp up. We are going to see a floodwave of LLM-powered bots/UX-wizards.

Even if academic progress stalls, we're going to be inundated for at least another few years.

Re: No "Zero-Shot" Without Exponential Data

#80

Post by CuriouslyC »

hprotagonist wrote: post #6

My long-standing observation has been that while nature may abhor a vacuum, she also really, really loves sigmoids. That performance vs. training data is not linear, but logarithmic, doesn't exactly come as a surprise.

The question that is unanswered, is the logarithmic performance improvement the result of better sampling of the underlying distribution over time, or related to just doing more training with slight variations to effectively regularize the model so it generalizes better? If it's the former, that indicates that we could achieve small models that are every bit as smart as large ones in limited domains, and if that's the case, it radically changes the landscape of what an optimal model architecture is.

I suspect from the success of Phi3 that it is in fact the former.

Post reply on HN