Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Natural language benchmarks don’t measure AI models’ general knowledge well

venturebeat.com

Post reply on HN
Print view View original

31–40 of 71 posts

Re: Natural language benchmarks don’t measure AI models’ general knowledge well

#31

Post by mikkom »

ccozan wrote: post #30
mrfusion wrote: post #14

Earlier quoted context omitted.

So you’re saying ai needs to be grounded to the real world to be effective?

Yes. Specifically the way human brain interacts with environment, the same way any AGI will need to sense and react to an environment, the more real the better. An interesting example would be if the NN learns something from a book, also fed it with the all the critique on that book ( the environment reaction ). So it can possibly learn from a human what is good or bad in that book, etc. Then both AI and humans can s…

> An interesting example would be if the NN learns something from a book, also fed it with the all the critique on that book ( the environment reaction ).

But isn't that exactly what has been done with GPT-3? AFAIK it has been trained with both facts (wikipedia and such) and web-crawled text content that contains both content and critique of content (book reviews, movie reviews etc.)?

Re: Natural language benchmarks don’t measure AI models’ general knowledge well

#32

Post by ckcheng »

m0zg wrote: post #16
tsimionescu wrote: post #3

I'm always amazed that some people think performing statistical analysis (training neural networks) on text can lead to actual intelligence and knowledge about the world, if only we increase the number of model parameters by a few more orders of magnitude (GPT-3 and the likely strategy for GPT-4). Of course, all this without any model of how likely it is that the knowledge is embedded in the text, such as trying to t…

I don't see why you're "amazed". Neural networks are _provably_ an universal approximator. As such, they can approximate _any_ function to an arbitrary degree of precision. It just might require an impractically large/complex neural net to do so, and you might not be able to train it yet, but it's _provably_ doable in the limit, if you remove size/training constraints. I'm a practitioner in the field, and an AI skept…

For clarity, Universal approximation doesn't mean ANNs can approximate just absolutely any function...

Depending on which proof you're looking at, ANNs can provably approximate any continuous function, or any Lebesgue integrable function, or any real vector-valued continuous function on a compact subset of R^n space [1]. That's certainly a very large class of functions, but it's not just any function.

Also, those proofs only work if you allow ANNs with arbitrary number of layers or layer width. Perhaps the required ANN size to approximate the AI / AGI function is so huge that it's beyond the number of particles on Earth. At what point is it beyond impractical? What if the ANN size (number of layers or neurons) is beyond the number of particles in the galaxy? Maybe it's just the wrong approach.

I haven't studied those proofs in detail, but as existence proofs, they just show that some ANN exists that can approximate a given function (of a certain class). There's no guarantee that there's a general algorithm that can learn the required ANN, is there? Maybe you just have to be really lucky to randomly initialize onto the correct approximating ANN...

Also, I'm not aware of any proof that human or bio NNs implement a Lebesgue integrable function, or etc. It's entirely possible bio NNs uses some weird (quantum?) physical process, and implements a function outside the scope of any ANN universal approximation theorem.

I'm just a little more cautious around those universal approximation theorems. I'm more or less in agreement with your comment otherwise!

[1]: https://en.wikipedia.org/wiki/Universal_approximation_theore...

Re: Natural language benchmarks don’t measure AI models’ general knowledge well

#33

Post by tsimionescu »

m0zg wrote: post #16
tsimionescu wrote: post #3

I'm always amazed that some people think performing statistical analysis (training neural networks) on text can lead to actual intelligence and knowledge about the world, if only we increase the number of model parameters by a few more orders of magnitude (GPT-3 and the likely strategy for GPT-4). Of course, all this without any model of how likely it is that the knowledge is embedded in the text, such as trying to t…

I don't see why you're "amazed". Neural networks are _provably_ an universal approximator. As such, they can approximate _any_ function to an arbitrary degree of precision. It just might require an impractically large/complex neural net to do so, and you might not be able to train it yet, but it's _provably_ doable in the limit, if you remove size/training constraints. I'm a practitioner in the field, and an AI skept…

I'm not claiming that an NN can't approximate human or super human intelligence, I'm claiming that using human writing as the only training set is not going to get you there.

The text simply doesn't contain enough information to draw a model of the world. Human language only makes sense if you understand the world, not the other way around.

Essentially what we're doing is similar to a human trying to learn psychology by reading books in Mandarin on it, while knowing neither the writing system nor the language. Except that human languages share some universal grammar, and share more or less the same notions about the world.

Re: Natural language benchmarks don’t measure AI models’ general knowledge well

#34

Post by tsimionescu »

IfOnlyYouKnew wrote: post #22
tsimionescu wrote: post #3

I'm always amazed that some people think performing statistical analysis (training neural networks) on text can lead to actual intelligence and knowledge about the world, if only we increase the number of model parameters by a few more orders of magnitude (GPT-3 and the likely strategy for GPT-4). Of course, all this without any model of how likely it is that the knowledge is embedded in the text, such as trying to t…

It amazes me people think their brain works differently. It's pretty close to believing in body/mind dualism, the only thing in neuroscience more outdated than Freud. Your brain works within the same laws of physics as the outside world. We don't know how the brain works exactly. But once we understand it, it is unlikely to be qualitatively different from a neural network. On the other side of the equation, the emerg…

As another poster wrote, the main problem is not the neural network itself, it is the training set, and very likely the training algorithm as well.

My claim is that working knowledge of the world is simply not encoded in human text. All human text (and speech) presupposes a certain model of the world that is learned from experiencing the world, likely based on a model that itself is partially pre-trained by evolutionary processes.

Re: Natural language benchmarks don’t measure AI models’ general knowledge well

#35

Post by m0zg »

ckcheng wrote: post #32
m0zg wrote: post #16

Earlier quoted context omitted.

I don't see why you're "amazed". Neural networks are _provably_ an universal approximator. As such, they can approximate _any_ function to an arbitrary degree of precision. It just might require an impractically large/complex neural net to do so, and you might not be able to train it yet, but it's _provably_ doable in the limit, if you remove size/training constraints. I'm a practitioner in the field, and an AI skept…

For clarity, Universal approximation doesn't mean ANNs can approximate just absolutely any function... Depending on which proof you're looking at, ANNs can provably approximate any continuous function, or any Lebesgue integrable function, or any real vector-valued continuous function on a compact subset of R^n space [1]. That's certainly a very large class of functions, but it's not just any function. Also, those pro…

> Perhaps the required ANN size to approximate the AI / AGI function is so huge that it's beyond the number of particles on Earth

Perhaps there's a teapot orbiting Mars. It's unlikely tho. I think a better hypothesis would be that to model a brain (if we knew how) one would need the same order of magnitude of neurons and connections. This is completely unachievable now, and even if it were technically possible nobody has a faintest clue about how to train such a thing. We do not understand _many_ things about how the brain learns (no backprop!), how it retains and retrieves memories, etc, etc. The best studied system is vision, and computer vision does actually work pretty well by now, achieving superhuman results on some tasks. Thus far the most complex brain we've been able to accurately and completely model is not even a brain at all - it's just a handful of neurons of a worm. I took an online course in neuroscience a few years ago hoping to gain some insight for my ANN work. The situation with our understanding of even primitive brains of i.e. rodents is, well, primitive. We also don't really have math to accurately model more than a handful of real, biological spiking neurons - PDEs get way, way too complex for anybody to handle. But one could argue we don't have to: ANNs effectively operate in the frequency domain instead, where magnitude of activation approximates spike train frequency, and timing effects are ignored.

Don't get me wrong, I'm pretty sure if we do achieve real AGI, it likely won't be with anything even remotely like the current tech and science stack. Energy efficiency and compute density is just not sufficient. But I'm also pretty confident that even with the current stack we will be able to, eventually, approximate intelligence closely enough to suit many practical uses that seem completely out of reach today, including even some limited degree of logical inference, and dare I say, cognition.

If only people (including myself) weren't so chickenshit and actually seriously worked on (and funded) this stuff, that could even happen in my lifetime. As things stand right now, people are afraid to touch anything AGI related due mostly to fear of ridicule.

In my comment above I am merely referring to the conjecture that _if_ we had ANNs that are large enough, and _if_ we knew how to structure and train them, there doesn't seem to be anything that would _fundamentally_ prevent them from supporting AGI. Those are very, very massive "ifs" indeed, without even a faintest glimmer of light at the end of the tunnel.

Re: Natural language benchmarks don’t measure AI models’ general knowledge well

#36

Post by mjburgess »

Isinlor wrote: post #5
nextos wrote: post #2

Judea Pearl has been bringing up the lack of causal knowledge in ML very often. He has even posted lots of interesting comments in Andrew Gelman's blog, e.g.: https://statmodeling.stat.columbia.edu/2009/07/05/disputes_a... I tend to think that lots of solutions could come from topics like those discussed in this book, with a lot of further development: http://probmods.org/

Understanding of causality is very likely an emergent property. While extremely important, it's unlikely that we have some hard-coded low level architecture of causal inference in brains. It probably will just arise as a necessity of grounded understanding of the world.

I'm not exactly sure what you mean.

The conditions for causal inference being possible are pretty clear and have to do with the intentional modification of the local environment.

The intention to achieve some new environmental state, and your action to bring it about, is a dynamical activity that enables "deep" model building.

Causal inference is not going to be some "module" of the brain... it requires a body. When you place your hand on a hot surface, once, you immediately understand that it is hot. It does not require "induction" (as hume supposed). That is because our body identifies causes.

Its therefore pretty trivial to observe no NLP system understands language, or even can understand language, because it lacks this capacity to acquire language semantics via participation in environmental exploration. It has no body.

ie., you need to have experienced "on top", "green", etc. to know what "green leaves grow on top of trees" means. There is no meaning in the frequency co-incidence of symbols in text.

So no matter how much you are able to reproduce these patterns, they contain no content. The content is in the reader.

Re: Natural language benchmarks don’t measure AI models’ general knowledge well

#37

Post by tsimionescu »

sdenton4 wrote: post #19
mellosouls wrote: post #15

Earlier quoted context omitted.

It does lead to intelligence and knowledge of the world. It leads to somewhat similar understanding of the world as our understanding of 4D, quantum mechanics or environment close to a black hole. Pretty bad one [...] GPT-3 understanding of physical world is similarly bad. No, GPT-3 has no "understanding" whatsoever - we at least have a conscious engagement with the topics you cite and a genuine understanding that we…

Do we really understand? What's the mechanism for that? My thinking is that we humans have some ability to do things by rote until we have a good statistical understanding, at which point we use the statistical understanding mostly, and fall back to rote where we have to be careful. But, mostly, we're stats machines too.

We obviously have a working model of the world that we draw facts from, that is partly built-in (trained by evolutionary processes) and partly refined with personal learning.

By contrast, GPT-3 only has a knowledge of text. To the extent you could say that it thinks, it thinks in terms of textual symbols that it has seen before - that is its world.

It is theoretically possible that if we found a way to encode experiences of the world of the world in a dataset and used the same training mechanisms and models that we used for GPT-3, we would get an AI with a decent model of the world. But it is magical thinking to imagine that we can get there by training a model on raw text.

Re: Natural language benchmarks don’t measure AI models’ general knowledge well

#38

Post by mjburgess »

IfOnlyYouKnew wrote: post #22
tsimionescu wrote: post #3

I'm always amazed that some people think performing statistical analysis (training neural networks) on text can lead to actual intelligence and knowledge about the world, if only we increase the number of model parameters by a few more orders of magnitude (GPT-3 and the likely strategy for GPT-4). Of course, all this without any model of how likely it is that the knowledge is embedded in the text, such as trying to t…

It amazes me people think their brain works differently. It's pretty close to believing in body/mind dualism, the only thing in neuroscience more outdated than Freud. Your brain works within the same laws of physics as the outside world. We don't know how the brain works exactly. But once we understand it, it is unlikely to be qualitatively different from a neural network. On the other side of the equation, the emerg…

> it is unlikely to be qualitatively different from a neural network

I'm sorry but this shows a profound misunderstanding of what a NN is, and what the brain is.

There are no "neural network"s. The NN algorithm is a method for optimizing the parameters of a piece-wise linear regression model.

These regression models have no homology to any brain structure and the process of producing them ("training") has no neurological analogue either. They can be produced with a variety of algorithms.

Here's one: f(x) = max(0, 2max(0, 3x - 0.1) - 0.5)

The phrase "neural network" is, as peddled by the media and poorly informed lecturers, a lie. It is neither neural nor a network. It's just gradient descent with more parameters.

A model of the brain would model, at least: neuroplasticity, biochemical signalling, activation frequency, etc.

There is nothing about a peice-wise linear regression model which does any of that. No matter how many circles and lines you draw. (NB. essentially any mathematical function can be drawn as a "neural network". The "network" is just a way of diagramming function application & dot-products).

Aside from all that, the body-brain system of animals is a physical process whose properties are not abstract. The reason we are intelligent is because we have bodies capable of causal analysis; and that capability is biophysical.

To put it another way, no algorithm which runs on a digital computer will turn it into gold. Not even one called, "the midas algorithm".

Re: Natural language benchmarks don’t measure AI models’ general knowledge well

#39

Post by tsimionescu »

hackinthebochs wrote: post #9
tsimionescu wrote: post #3

I'm always amazed that some people think performing statistical analysis (training neural networks) on text can lead to actual intelligence and knowledge about the world, if only we increase the number of model parameters by a few more orders of magnitude (GPT-3 and the likely strategy for GPT-4). Of course, all this without any model of how likely it is that the knowledge is embedded in the text, such as trying to t…

It shouldn't be that hard to imagine. If the objective is to predict human text, at some point the best way to predict text is to capture the kinds of knowledge that humans know and use in text generation. Human knowledge is a strong prior for human text generation, and so it shouldn't be surprising that for a large model its parameters would end up in a portion of parameter space that encodes some human knowledge.

By this token, the best way to learn Mandarin would be to start looking at text in Mandarin and trying to assign it meaning. While it theoretically can work, it is vastly more difficult than trying to relate it to a previously learned model. In fact, it is so difficult that we have never successfully understood a lost human language this way (though with the caveat that we don't have the huge corpus of text that GPT-3 was trained on in any of the lost languages we tried it on).

Re: Natural language benchmarks don’t measure AI models’ general knowledge well

#40

Post by tsimionescu »

m0zg wrote: post #35
ckcheng wrote: post #32

Earlier quoted context omitted.

For clarity, Universal approximation doesn't mean ANNs can approximate just absolutely any function... Depending on which proof you're looking at, ANNs can provably approximate any continuous function, or any Lebesgue integrable function, or any real vector-valued continuous function on a compact subset of R^n space [1]. That's certainly a very large class of functions, but it's not just any function. Also, those pro…

> Perhaps the required ANN size to approximate the AI / AGI function is so huge that it's beyond the number of particles on Earth Perhaps there's a teapot orbiting Mars. It's unlikely tho. I think a better hypothesis would be that to model a brain (if we knew how) one would need the same order of magnitude of neurons and connections. This is completely unachievable now, and even if it were technically possible nobody…

I completely agree with your conclusions starting from these ifs!

However, I would add that another interesting area of study that is essentially not being explored at all is exactly to try to understand more about human cognition at the computational level (whereas neuroscience is working at the biological level). Basically, it seems we have abandoned any work in AI that would try to model human learning or human understanding of language in a human-understandable way, and instead we are aiming for reproducing human behaviors for engineering purposes.

Post reply on HN