Live data from Hacker News

No "Zero-Shot" Without Exponential Data

arxiv.org

91–100 of 123 posts

Re: No "Zero-Shot" Without Exponential Data

#91

Earlier quoted context omitted.

When I was little, zebras were described to me as black and white stripped horses. Without even seeing one, I'm sure anything who has seen a horse could then merge those two concepts to create a close to accurate picture of what a zebra is. If AI is supposed to resemble a human mind with ability to learn, then it must be able to learn from a blanker slate. You don't teach the human before it is born, and in this comp…

You can look at a medieval bestiary to see how people thought animals might look based on descriptions alone. Like these lovely elephants https://britishlibrary.typepad.co.uk/digitisedmanuscripts/20...

Honestly, these are some of my favorite stories and I think more ML people need to learn more about mythology (I say this as a ML researcher btw). Because once you go down this path you start to understand how "Rino" == "Unicorn". You have to really think about how to explain things when you're working with a limited language. Yeah, we have the word "rino" now, but how would you describe one to someone who has no concept of this? Maybe a cow with one big horn? Is "like a big fat tough skin horse with a big horn coming out of its head" accurate? And then apply your classic game of telephone[0]. It is also how you get things like how in Chinese a giraffe is "long neck deer"[1] (that doesn't work for all things in Chinese and there's another game of telephone (lol, maybe I was too harsh on the British in [0]) and well... you can imagine things get warped like crazy).

There's so many rabbit holes to go down when trying to understand language, vision, reasoning, and all that stuff.

[0] (Jesus England... this is what you call this game?!) https://en.wikipedia.org/wiki/Chinese_whispers

[1] https://translate.google.com/?sl=en&tl=zh-CN&text=giraffe&op... ----> https://translate.google.com/?sl=zh-CN&tl=en&text=%E9%95%BF%...

Re: No "Zero-Shot" Without Exponential Data

#92
post #87
post #64

Earlier quoted context omitted.

This illustrates two ways of teaching I’ve experienced both, each at a different university In one, professors would teach one thing then ask very different (and much harder) questions on tests In the other, tests were more of a recap of the material up to that point I definitely learned a lot more in the second case and was a lot more motivated. It also required more effort from the professors The two methods also t…

> it’s about either being lucky or good at testing I think you are misunderstanding the experience. The first (harder questions) is testing your understanding of the material and problem. Can you applying the material to solve a novel problem? Do you understand the material not just the mechanics. Do you understand how it would interelate it with other problems? Do you understand the limitations? The second is just r…

Fwiw, I think both of you are on the same page.

And yes, to bring back to ML it is the difference of generalization and memorization (compression). I wrote a longer response to a different response to my initial comment to help clarify because I think this chain is a bit obtuse and aggressive for no reason :/ (I mean you can check the Wiki page to verify what I said)

Re: No "Zero-Shot" Without Exponential Data

#93

I've always been rather upset that it's fairly common to train on things like LAION or COCO and then "zero shot" test on ImageNet. Zero shot doesn't mean a held out set, it means disjoint classes. You can't train on all the animals in the zoo with sentences and then be surprised your model knows zebras. You need to train on horses and test on zebras.

Should we also try to teach children geometry and test them on calculus?

I think you're being overly obtuse, and I'd request you try to work in good faith. If you doubt what I claimed, you can quickly verify on the wiki page[0].

In essence you aren't wrong, but that's not what we'd typically do in a zero (or few) shot setting. We'd be focusing on things that are more similar. If you want to understand this a bit better in what we might do in a ML context I wrote more here[1].

And I like Nico's comment about how different professors test. Because it makes you think about what is actually being tested. Are you being tested on memorization or generalization? You can argue both these kinds of tests are testing "if you learned the material" but we'd understand that these two types of tests are fundamentally different and let's be real, are not reasonably fair to compare scores to. I'm sure many of us have experienced this where someone that gets a C in professor A's class likely learned more than than someone who got an A in professor B's class. The thing is that the nuance is incredibly important here to really understand. And you can trivialize anything, but be careful when doing so, you may overlook the most important things ;)

Now... you could make this argument about geometry -> calculus if we're not talking about the typical geometry (single) class most people will have taken in either middle school or high school. Because yes, at the end of the day there is a geometric interpretation and we have the Riemann sum. But we'd need to ensure those kids have the understanding of infinities (which aren't numbers btw). We'd have to be pretty careful about formulating this kind of test if we're going to take away any useful information from it. Though the naive version might give us clues about information leakage (in our case with children this might be "child who has a parent that's a mathematician" or something along those lines). It really all depends on what the question behind the test is. Scores only mean things when we have nuanced clear understandings of what we're measuring (so again, tread carefully because "here be dragons" and you're likely to get burned before, or even without, knowing it)

And truth be told, we actually do this a bit. There's a reason you take geometry before calculus. Because the skills build up. But you're right that they don't generalize.

[0] https://en.wikipedia.org/wiki/Zero-shot_learning

[1] https://news.ycombinator.com/item?id=40313501

Re: No "Zero-Shot" Without Exponential Data

#94

Earlier quoted context omitted.

It's not super surprising that LLMs perform well on standardized tests, given that they have a lot of standardized test related text in their training data. There are a lot of claims out there about the zero-shot ability of LLMs, and very little specific research to back it up. Until now that is.

This paper is about CLIP not LLMs and does not generalize to LLM architectures.

Why not?

Re: No "Zero-Shot" Without Exponential Data

#95
post #27
post #15

This feels like the worst possible outcome of the current AI hype. We've essentially been ripping off the entire internet and feeding it to the models already, spending many billions of dollars in the process. It's pretty much the largest possible dataset you can currently get, and due to the ever-increasing and now rapidly accelerated AI poisoning of the internet most likely the largest possible dataset which will e…

> not-entirely-useless but still quite crappy 15 months ago, general-purpose LLMs that have not been specifically trained on legal reasoning could score better than 90% of humans on the multistate bar exam, and these are humans who actually completed law school. General-purpose LLMs get similar results in medicine, and when the models are fine-tuned for medical diagnosis they're even better. And that was more than a…

> 15 months ago, general-purpose LLMs that have not been specifically trained on legal reasoning could score better than 90% of humans on the multistate bar exam

The claims of similar performance on coding problems were shown to be due to contamination of the training data on the tested problems. It did abysmal on problems made public after the model training cutoff.

I don’t think anyone has tested contamination for the MBE claims, but I would lean toward assuming the same issue exists for that assessment until proven otherwise.

Re: No "Zero-Shot" Without Exponential Data

#96
post #66
post #63

Earlier quoted context omitted.

When we learn the grammar of our language, the teacher does not stand in front of the class and proceed to say a large corpus of examples of ungrammatical sentences, only the correct ones are in the training set. When we learn to drive, we do not need to crash our car a thousand times in a row before we start to get it. When we play a new board game for the first time, we can do it fairly competently (though not as g…

please help yourself and do a quick Google search about "zero shot" and "few shot" learning.

You could explain them instead of being snarky.

https://xkcd.com/1053/

Re: No "Zero-Shot" Without Exponential Data

#97
post #15

This feels like the worst possible outcome of the current AI hype. We've essentially been ripping off the entire internet and feeding it to the models already, spending many billions of dollars in the process. It's pretty much the largest possible dataset you can currently get, and due to the ever-increasing and now rapidly accelerated AI poisoning of the internet most likely the largest possible dataset which will e…

The training sets used for current models, even the largest ones, is nowhere even close to "the entire Internet". Some napkin math based on known sizes of public datasets says it's less than 1%.

Re: No "Zero-Shot" Without Exponential Data

#98

Better title: Image classification models suck at identifying nouns that they've rarely seen. Crucial context: - They're only looking at image models -- not LLMs, etc - Their models are tiny - A "concept" here just means "a noun." The authors index images via these nouns. - They didn't control for difficulty in visual representation/recognition of these exceptional infrequent, long-tail "concepts." If I didn't know a…

[deleted]

Re: No "Zero-Shot" Without Exponential Data

#99
post #60

Earlier quoted context omitted.

For centuries we only taught children geometry and one of them invented calculus.

Now that’s setting a high bar. If AI could reliably invent calculus, then I’d be briefly impressed and then terrified.

One AI in a couple hundred years might be able to do it by luck?

Re: No "Zero-Shot" Without Exponential Data

#100
post #26

Earlier quoted context omitted.

I think that depends what your expectations are, and what you mean by another AI winter. We're just scratching the surface of what's possible with the current state of the art. Even if there are no major advances or breakthroughs in the near future, LLMs and associated technologies are already useful in many use cases. Or close enough to useful where engineering rather than science will be sufficient to overcome many…

The whole point of past AI winters is that genuinely useful technologies were preposterously overhyped as representing "human cognition," despite overwhelming evidence to the contrary. The bubble bursts - and the money evaporates - because expectations come crashing down, not because the usefulness was a mirage. Lisp was hardly the key to human symbolic thought like some hoped it would be, but it helped improve progr…

> But you would think after 70 years AI practitioners would learn some humility!

Why? The people doing AI now are not the same as those that did AI 60 years ago.

Post reply on HN