Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Large Language Models Are Neurosymbolic Reasoners

arxiv.org

Post reply on HN
Print view View original

11–20 of 172 posts

Re: Large Language Models Are Neurosymbolic Reasoners

#11

Post by mjburgess »

bubblyworld wrote: post #7
clooper wrote: post #5

I was recently thinking how every neural network is equivalent to a lookup table where the input is all numbers up to what can be expressed within the context window and the output is the result of the arithmetic operations applied to that number. So every neural network is equivalent to T = {(i, f(i)) : i The main practical issue is the size of the table but I don't see any theoretical reasons why this is incorrect.…

This is an old argument against determinism - I think a serious challenge is that: 1. Modern physics suggests you can implement such a lookup table for any subset of our universe. 2. We are a subset of the universe. 3. Therefore we are representable by lookup tables too. ...so your argument appears to prove too much, namely that humans aren't thinking beings either. Which is fine, but personally I don't think that's…

We're not a lookup table of the things we're, eg., saying, or doing etc. Nor are we looking up, in this sense, when we act.

ie., when you compress text into an NN and use it to generate text, the generated text is just a synthesis of the compressed text.

Whereas when I type, I am not synthesising text. Rather I have the skill of typing, I have an interior subjectivity of thoughts, I have memories which arent text, and so on.

When my fingers move across the keyboard it isn't because they are looking up text.

Our causal properties (experiencing, thinking, seeing, feeling, remembering, moving, speaking, growing, digesting ...) are not each, "index on the total history of prior experience", "index on the total history of prior seeing". The world directly causes, eg., us to see -- seeing isnt a lookup table of prior seeings.

( Also, the whole of physics is formulated in terms that cannot be made into a lookup table; and there is no evidence, only insistence, of the converse. )

Re: Large Language Models Are Neurosymbolic Reasoners

#12

Post by josh-stylo »

clooper wrote: post #8
bubblyworld wrote: post #7

Earlier quoted context omitted.

This is an old argument against determinism - I think a serious challenge is that: 1. Modern physics suggests you can implement such a lookup table for any subset of our universe. 2. We are a subset of the universe. 3. Therefore we are representable by lookup tables too. ...so your argument appears to prove too much, namely that humans aren't thinking beings either. Which is fine, but personally I don't think that's…

How are people lookup tables? In the case of neural networks the representation of the table is obvious, it's just numbers. What would be the equivalent table for the liver? My argument isn't abstract. Neural networks really are just numerical functions which can be expanded into their equivalent graph representations.

Not sure what he's referring to in terms of modern physics saying we're just a lookup table but at the very least, you could say the same thing about the conversation that we're having now. You read words, those words map to meaning representation in our heads, we then generate a response.

Re: Large Language Models Are Neurosymbolic Reasoners

#13

Post by clooper »

Centigonal wrote: post #9
clooper wrote: post #5

I was recently thinking how every neural network is equivalent to a lookup table where the input is all numbers up to what can be expressed within the context window and the output is the result of the arithmetic operations applied to that number. So every neural network is equivalent to T = {(i, f(i)) : i The main practical issue is the size of the table but I don't see any theoretical reasons why this is incorrect.…

It sounds like you're asking whether the output of a neural network is a deterministic function of its input. For many LLMs, you can make that answer yes with the right combination of parameters (temperature = 0) and underlying compute (variance in floating point calculations can still introduce randomness in model outputs even when the model should theoretically return the same answer every time). There are some way…

People can not be reduced to lookup tables even in theory. No one even knows how a single cell does what it does let alone an entire organism like a person.

I'm not making an abstract claim about neural networks because all numerical algorithms like neural networks can be reduced to a lookup table given a large enough hard drive. This is not practical because the space required would exceed the number of atoms in the known universe but the argument is sound. The same isn't true for people unless a person is idealized and abstracted into a sequence of numbers. I'm not saying no one is allowed to think of people as some sequence of numbers but this is clearly an abstraction of what it means to be a person and in the case of the neural network there is no abstraction, it really is a numerical function which can be expanded into a large table which represents its graph.

Re: Large Language Models Are Neurosymbolic Reasoners

#14

Post by clooper »

josh-stylo wrote: post #12
clooper wrote: post #8

Earlier quoted context omitted.

How are people lookup tables? In the case of neural networks the representation of the table is obvious, it's just numbers. What would be the equivalent table for the liver? My argument isn't abstract. Neural networks really are just numerical functions which can be expanded into their equivalent graph representations.

Not sure what he's referring to in terms of modern physics saying we're just a lookup table but at the very least, you could say the same thing about the conversation that we're having now. You read words, those words map to meaning representation in our heads, we then generate a response.

Obviously if we are interacting over a digital medium then the responses will be encoded as numbers but there is no way to reduce an entire person to a lookup table. Measured output of human behavior can be expressed as lists of numbers but thinking is not the same as the list of numbers, unlike in the case of neural networks where the graph and the network are actually equivalent.

Re: Large Language Models Are Neurosymbolic Reasoners

#15

Post by Centigonal »

clooper wrote: post #13
Centigonal wrote: post #9

Earlier quoted context omitted.

It sounds like you're asking whether the output of a neural network is a deterministic function of its input. For many LLMs, you can make that answer yes with the right combination of parameters (temperature = 0) and underlying compute (variance in floating point calculations can still introduce randomness in model outputs even when the model should theoretically return the same answer every time). There are some way…

People can not be reduced to lookup tables even in theory. No one even knows how a single cell does what it does let alone an entire organism like a person. I'm not making an abstract claim about neural networks because all numerical algorithms like neural networks can be reduced to a lookup table given a large enough hard drive. This is not practical because the space required would exceed the number of atoms in the…

My "what if someone made a lookup table of everything I ever said in response to something else" hypothetical is pretty flimsy - I realized that right after writing it.

The point I wanted to make is that concepts of sentience, consciousness, reasoning, intelligence, etc. are very philosophically loaded ideas.

Responding to your comment, I don't think anyone credible is arguing that a human being is somehow the same as a neural network. I think the question at play here is "what constitutes reasoning?" - and more specifically "can a deterministic process reason?"

This is not a new debate at all - an abacus can tell us truths about the world, but we don't consider the abacus intelligent. Is GPT-4 somehow different, or is it a very large abacus?

Re: Large Language Models Are Neurosymbolic Reasoners

#16

Post by cryptoxchange »

Centigonal wrote: post #9
clooper wrote: post #5

I was recently thinking how every neural network is equivalent to a lookup table where the input is all numbers up to what can be expressed within the context window and the output is the result of the arithmetic operations applied to that number. So every neural network is equivalent to T = {(i, f(i)) : i The main practical issue is the size of the table but I don't see any theoretical reasons why this is incorrect.…

It sounds like you're asking whether the output of a neural network is a deterministic function of its input. For many LLMs, you can make that answer yes with the right combination of parameters (temperature = 0) and underlying compute (variance in floating point calculations can still introduce randomness in model outputs even when the model should theoretically return the same answer every time). There are some way…

To your last point, https://en.m.wikipedia.org/wiki/Problem_of_induction

A LUT version of you is inductive. Every observed input/output pair does not uniquely identify your current state. Much like a puddle left by a melted ice cube indicates its volume, but little to nothing of its shape.

Post LUT-you genesis, applying property based fuzz testing would quickly reveal that the LUT-you is one of an infinite number of LUT-yous that melts into the puddle of historical data, but not the LUT-you that is the original ice cube.

https://fsharpforfunandprofit.com/posts/property-based-testi...

Re: Large Language Models Are Neurosymbolic Reasoners

#17

Post by Legend2440 »

clooper wrote: post #13
Centigonal wrote: post #9

Earlier quoted context omitted.

It sounds like you're asking whether the output of a neural network is a deterministic function of its input. For many LLMs, you can make that answer yes with the right combination of parameters (temperature = 0) and underlying compute (variance in floating point calculations can still introduce randomness in model outputs even when the model should theoretically return the same answer every time). There are some way…

People can not be reduced to lookup tables even in theory. No one even knows how a single cell does what it does let alone an entire organism like a person. I'm not making an abstract claim about neural networks because all numerical algorithms like neural networks can be reduced to a lookup table given a large enough hard drive. This is not practical because the space required would exceed the number of atoms in the…

>People can not be reduced to lookup tables even in theory

Sure you can. Simply enumerate all of the physical states that the atoms in your body could be in. Any finite-sized object has a finite number of possible states, and so can be represented by a finite lookup table.

Your argument is so broad as to be meaningless.

Re: Large Language Models Are Neurosymbolic Reasoners

#18

Post by mewpmewp2 »

mjburgess wrote: post #11
bubblyworld wrote: post #7

Earlier quoted context omitted.

This is an old argument against determinism - I think a serious challenge is that: 1. Modern physics suggests you can implement such a lookup table for any subset of our universe. 2. We are a subset of the universe. 3. Therefore we are representable by lookup tables too. ...so your argument appears to prove too much, namely that humans aren't thinking beings either. Which is fine, but personally I don't think that's…

We're not a lookup table of the things we're, eg., saying, or doing etc. Nor are we looking up, in this sense, when we act. ie., when you compress text into an NN and use it to generate text, the generated text is just a synthesis of the compressed text . Whereas when I type, I am not synthesising text . Rather I have the skill of typing, I have an interior subjectivity of thoughts, I have memories which arent text,…

Typing is just a medium, it is irrelevant. Seeing and all the other senses that you mentioned are input within a context window.

Re: Large Language Models Are Neurosymbolic Reasoners

#19

Post by clooper »

Centigonal wrote: post #15
clooper wrote: post #13

Earlier quoted context omitted.

People can not be reduced to lookup tables even in theory. No one even knows how a single cell does what it does let alone an entire organism like a person. I'm not making an abstract claim about neural networks because all numerical algorithms like neural networks can be reduced to a lookup table given a large enough hard drive. This is not practical because the space required would exceed the number of atoms in the…

My "what if someone made a lookup table of everything I ever said in response to something else" hypothetical is pretty flimsy - I realized that right after writing it. The point I wanted to make is that concepts of sentience, consciousness, reasoning, intelligence, etc. are very philosophically loaded ideas. Responding to your comment, I don't think anyone credible is arguing that a human being is somehow the same a…

As a numerical function it can be implemented on an abacus so I don't think it's any different from a large enough abacus. It's practically not feasible but theoretically there is no idealization or abstraction happening when numerical calculations on a computer are transferred to an abacus.

Re: Large Language Models Are Neurosymbolic Reasoners

#20

Post by mewpmewp2 »

clooper wrote: post #14
josh-stylo wrote: post #12

Earlier quoted context omitted.

Not sure what he's referring to in terms of modern physics saying we're just a lookup table but at the very least, you could say the same thing about the conversation that we're having now. You read words, those words map to meaning representation in our heads, we then generate a response.

Obviously if we are interacting over a digital medium then the responses will be encoded as numbers but there is no way to reduce an entire person to a lookup table. Measured output of human behavior can be expressed as lists of numbers but thinking is not the same as the list of numbers, unlike in the case of neural networks where the graph and the network are actually equivalent.

You could represent all the input on different levels as numbers, e.g. all EM waves hitting our eyes, then all the physical output from our body also as numbers, and everything that causes this output from input within is what you would consider to be a lookup table.
Post reply on HN