Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

No "Zero-Shot" Without Exponential Data

arxiv.org

Post reply on HN
Print view View original

111–120 of 123 posts

Re: No "Zero-Shot" Without Exponential Data

#111

Post by anon373839 »

ItsBob wrote: post #107

Not to derail this conversation but... When I'm explaining AI stuff to family, the example I use is classification and I specifically use cats and dogs. I use the analogy of how you teach a toddler that this is a cat and that is a dog. Essentially repetition. And at first they get them mixed up and the parent will say "no, that's a dog" when they think it's a cat and so on. But essentially, for a child learning the d…

Neural networks do not imitate the brain and “training” a network has nothing to do with learning, despite these terms serving double duty.

Machine learning models are just math functions fitted to some data. When we get predictions from them, we’re really just using a technique to interpolate between the data points. The denser a particular region has been sampled in the data, the better the predictions will be. (This is why GPT-anything will do a good job writing solutions to common leetcode problems, while struggling with a novel problem.)

Humans have a powerful abstraction ability far beyond any algorithm that has been developed. We can take in a few pieces of information describing a really unusual set of circumstances, run imaginary experiments and simulations on them, and make very granular and accurate predictions about their consequences. Nobody actually knows how.

Re: No "Zero-Shot" Without Exponential Data

#112

Post by pas »

PheonixPharts wrote: post #46
pas wrote: post #10

Earlier quoted context omitted.

RAG seems to be all the rage. Not to mention the quest for the cooking up the correct cocktail of smaller MoE/ensemble models, and ... there's decades' worth of optimization work ahead (a few years of it seems to be already VC and edu grants funded), no?

There's tons of optimization work left, but by it's nature optimization tends to push the limits of what we currently have, and rarely allows for substantial improvements. There are some major limitations to LLMs that aren't going to be "optimized" away. At the end of the day LLMs are Monte Carlo samplers over a latent, compressed representation existing human text data. Many of the tasks people hope LLMs will achiev…

> An LLM can only look at each token and can't, statistically speaking, think in terms of the entire query.

There's always some space in high-dimensions to put that extra context somewhere. So yes, only one token is predicted, but there's a lot of information squeezed into the "condition" (as in conditional probability, as mjburgess' comment shows)

> [hype instead of NLP]

yes, exactly. we'll see what remains after the bubble bursts.

Re: No "Zero-Shot" Without Exponential Data

#113

Post by pas »

cs702 wrote: post #14
pas wrote: post #10

Earlier quoted context omitted.

RAG seems to be all the rage. Not to mention the quest for the cooking up the correct cocktail of smaller MoE/ensemble models, and ... there's decades' worth of optimization work ahead (a few years of it seems to be already VC and edu grants funded), no?

I think the grandparent comment is about AI research, driven by the quest for AGI. Incremental improvement of proven approaches, driven by profit motive, will surely continue regardless of whether there is an AI winter or not.

Indeed, but there's probably a very real funding and attention (heh) issue, if there's ton of both then there will be progress. But. Usually the bigger the hype the more progress is expected so the faster the fund-tention will dissipate.

Re: No "Zero-Shot" Without Exponential Data

#114

Post by scotty79 »

ItsBob wrote: post #107

Not to derail this conversation but... When I'm explaining AI stuff to family, the example I use is classification and I specifically use cats and dogs. I use the analogy of how you teach a toddler that this is a cat and that is a dog. Essentially repetition. And at first they get them mixed up and the parent will say "no, that's a dog" when they think it's a cat and so on. But essentially, for a child learning the d…

Toddlers have a neural network architected and fine tuned for object recognition. They also have a benefit of learning thousands of hours of video from the real world, initially blurry, gradually getting sharper, to draw the model of physical reality before they are given first examples of "dog" or "cat" to classify. If you kept toddler immobile in the darkness then presented him with static 2d photos of a dog or a cat I think it wouldn't learn much faster than artificial neural net.

Re: No "Zero-Shot" Without Exponential Data

#115

Post by ranyume »

anon373839 wrote: post #111
ItsBob wrote: post #107

Not to derail this conversation but... When I'm explaining AI stuff to family, the example I use is classification and I specifically use cats and dogs. I use the analogy of how you teach a toddler that this is a cat and that is a dog. Essentially repetition. And at first they get them mixed up and the parent will say "no, that's a dog" when they think it's a cat and so on. But essentially, for a child learning the d…

Neural networks do not imitate the brain and “training” a network has nothing to do with learning, despite these terms serving double duty. Machine learning models are just math functions fitted to some data. When we get predictions from them, we’re really just using a technique to interpolate between the data points. The denser a particular region has been sampled in the data, the better the predictions will be. (Th…

> just math functions fitted to some data

I don't want to be pedantic. But I'd like to interject. We can simulate everything with math functions. If the algorithm isn't there yet it's because we're using the wrong functions.

Re: No "Zero-Shot" Without Exponential Data

#116

Post by satvikpendem »

anon373839 wrote: post #111
ItsBob wrote: post #107

Not to derail this conversation but... When I'm explaining AI stuff to family, the example I use is classification and I specifically use cats and dogs. I use the analogy of how you teach a toddler that this is a cat and that is a dog. Essentially repetition. And at first they get them mixed up and the parent will say "no, that's a dog" when they think it's a cat and so on. But essentially, for a child learning the d…

Neural networks do not imitate the brain and “training” a network has nothing to do with learning, despite these terms serving double duty. Machine learning models are just math functions fitted to some data. When we get predictions from them, we’re really just using a technique to interpolate between the data points. The denser a particular region has been sampled in the data, the better the predictions will be. (Th…

Are you a physicalist or a dualist, in the philosophy of mind senses of those words? I don't see how what we do is much different than computers as both use fundamental physical computation to achieve a result. It could simply be that our brains are much more complex in their computation than current computers, but they both compute nonetheless.

Re: No "Zero-Shot" Without Exponential Data

#117

Post by satvikpendem »

ItsBob wrote: post #107

Not to derail this conversation but... When I'm explaining AI stuff to family, the example I use is classification and I specifically use cats and dogs. I use the analogy of how you teach a toddler that this is a cat and that is a dog. Essentially repetition. And at first they get them mixed up and the parent will say "no, that's a dog" when they think it's a cat and so on. But essentially, for a child learning the d…

1. Children are not tabulae rasae, they have previous optimization of their brains via evolution such that they are much more predisposed to visual and other human tasks than computers. On the other hand, computers are tabulae rasae such that they are essentially blind and have no optimization until you train them.

2. Children see many more data points than computers, thousands of hours of lived experience from which they can learn, of not just the visual world but the auditory and other sensory worlds.

Essentially, you are comparing something that is ~80% trained and then training it on billions of data points to something that is 0% trained and then training it on still images of only two dimensions and asking why the former is much better.

Re: No "Zero-Shot" Without Exponential Data

#118

Post by pas »

Imnimo wrote: post #65
panarky wrote: post #27

Earlier quoted context omitted.

> not-entirely-useless but still quite crappy 15 months ago, general-purpose LLMs that have not been specifically trained on legal reasoning could score better than 90% of humans on the multistate bar exam, and these are humans who actually completed law school. General-purpose LLMs get similar results in medicine, and when the models are fine-tuned for medical diagnosis they're even better. And that was more than a…

I think this says more about the benchmark than the capabilities of the model. If it were the case that 90th percentile performance on the bar exam mean that a model was a 90th percentile lawyer, and we've had these models for 15 months (in fact longer), where are all the LLM lawyers? The lesson here is that a test designed for humans may not be equally representative of capabilities when given to an LLM.

nah, it's probably an okay-ish benchmark for humans, but it's just that, a filter to weed out those who can't learn hundreds of pages of legal trivia.

the model is great at this, because the training set is full of this stuff. the questions are static and simple. (the actual text of the questions change, of course, but the format and the answers are from a fixed set. and the LLM doesn't need to generate text basically, just one token. A B C or D ... that said I'm curious how it's administered to the LLMs and how much that influences their performance.)

Re: No "Zero-Shot" Without Exponential Data

#119

Post by pas »

crote wrote: post #55
panarky wrote: post #27

Earlier quoted context omitted.

> not-entirely-useless but still quite crappy 15 months ago, general-purpose LLMs that have not been specifically trained on legal reasoning could score better than 90% of humans on the multistate bar exam, and these are humans who actually completed law school. General-purpose LLMs get similar results in medicine, and when the models are fine-tuned for medical diagnosis they're even better. And that was more than a…

> Progress is still tracking the steep part of the S-curve, and there's no indication that they're near the top yet. If I understand it correctly, that seems to be exactly what this paper is suggesting. Scoring high on the bar exam is pretty trivial for AI - the data needed for that is fairly generic and widely available on the internet. It requires you to demonstrate a relatively basic understanding of the concepts…

what is novel?

what if you couple a random text generator with a picture generator, isn't each new picture novel?

if we could send all artists to work in the healthcare mines we might discover new treatment for various ailments. is that not novel? is that too indirect?

increasing "economic surplus with externalities factored in" is good for society even if it's not novel.

arguably philosophers come up with novel stuff all the time, and ...

> Current models simply aren't capable of that, and the results achieved here with multimodal models suggests that they never will.

can you elaborate on this a bit please?

Re: No "Zero-Shot" Without Exponential Data

#120

Post by anon373839 »

satvikpendem wrote: post #116
anon373839 wrote: post #111

Earlier quoted context omitted.

Neural networks do not imitate the brain and “training” a network has nothing to do with learning, despite these terms serving double duty. Machine learning models are just math functions fitted to some data. When we get predictions from them, we’re really just using a technique to interpolate between the data points. The denser a particular region has been sampled in the data, the better the predictions will be. (Th…

Are you a physicalist or a dualist, in the philosophy of mind senses of those words? I don't see how what we do is much different than computers as both use fundamental physical computation to achieve a result. It could simply be that our brains are much more complex in their computation than current computers, but they both compute nonetheless.

I have no idea! I don’t think these questions can be answered (despite modern culture expressing absolute faith in physicalism).

But back to the subject of deep learning, whatever brains do, I think it’s pretty clear that existing neural networks don’t approximate the biological process. They’re just too static.

Post reply on HN