Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Large Language Models Are Neurosymbolic Reasoners

arxiv.org

Post reply on HN
Print view View original

21–30 of 172 posts

Re: Large Language Models Are Neurosymbolic Reasoners

#21

Post by clooper »

Legend2440 wrote: post #17
clooper wrote: post #13

Earlier quoted context omitted.

People can not be reduced to lookup tables even in theory. No one even knows how a single cell does what it does let alone an entire organism like a person. I'm not making an abstract claim about neural networks because all numerical algorithms like neural networks can be reduced to a lookup table given a large enough hard drive. This is not practical because the space required would exceed the number of atoms in the…

>People can not be reduced to lookup tables even in theory Sure you can. Simply enumerate all of the physical states that the atoms in your body could be in. Any finite-sized object has a finite number of possible states, and so can be represented by a finite lookup table. Your argument is so broad as to be meaningless.

Then give some concrete numbers for the states of the atoms. My argument is not abstract, it is very concrete. Give me a neural network and I can generate the graph and prove the equivalence between the network and its graph representation as a table of tuples.

Re: Large Language Models Are Neurosymbolic Reasoners

#22

Post by wiz21c »

bubblyworld wrote: post #6

The authors get LLMs to perform pretty well in a variety of IF-style text based games. Which is pretty cool, these kinds of games are played and read in natural language, which makes them pretty hard to write AIs for normally. Something I'd love to see one day is modern AI applied to other kinds of text based games like nethack. Last I checked nobody had managed to solve the problem of nethack AI without using hard c…

Yes IF-style for sure but that's not Zork either. Cool work though, but having those solving Zork or Mystery House or whatever would be sooo cool !

Re: Large Language Models Are Neurosymbolic Reasoners

#23

Post by clooper »

mewpmewp2 wrote: post #20
clooper wrote: post #14

Earlier quoted context omitted.

Obviously if we are interacting over a digital medium then the responses will be encoded as numbers but there is no way to reduce an entire person to a lookup table. Measured output of human behavior can be expressed as lists of numbers but thinking is not the same as the list of numbers, unlike in the case of neural networks where the graph and the network are actually equivalent.

You could represent all the input on different levels as numbers, e.g. all EM waves hitting our eyes, then all the physical output from our body also as numbers, and everything that causes this output from input within is what you would consider to be a lookup table.

What are the dimension of the input and output spaces involved in this idealization? In the case of a neural network there is no idealization. The network is software, it's a number. It's inputs and outputs are all bounded and can be expressed as a table of bounded tuples.

Re: Large Language Models Are Neurosymbolic Reasoners

#24

Post by microtonal »

clooper wrote: post #5

I was recently thinking how every neural network is equivalent to a lookup table where the input is all numbers up to what can be expressed within the context window and the output is the result of the arithmetic operations applied to that number. So every neural network is equivalent to T = {(i, f(i)) : i The main practical issue is the size of the table but I don't see any theoretical reasons why this is incorrect.…

Suppose that we used embeddings as the input of the model rather than piece identifiers plus an embedding lookup table. This is possible with every transformer model and some libraries provide an API to do this. Moreover, we convert the parameters and ops to use arbitrary precision types. Then the network cannot be represented as a lookup table. Given that there is an infinite number of inputs, there is also an infinite number of outputs. But the arbitrary-precision network does not operate fundamentally different from the original network. It has the same parameters, ops, etc., yet you cannot store it as a (finite) lookup table.

Re: Large Language Models Are Neurosymbolic Reasoners

#25

Post by zeroonetwothree »

clooper wrote: post #21
Legend2440 wrote: post #17

Earlier quoted context omitted.

>People can not be reduced to lookup tables even in theory Sure you can. Simply enumerate all of the physical states that the atoms in your body could be in. Any finite-sized object has a finite number of possible states, and so can be represented by a finite lookup table. Your argument is so broad as to be meaningless.

Then give some concrete numbers for the states of the atoms. My argument is not abstract, it is very concrete. Give me a neural network and I can generate the graph and prove the equivalence between the network and its graph representation as a table of tuples.

You said "even in theory" which is obviously wrong, since the (local) universe is finite and deterministic, hence it is itself a giant lookup table.

Re: Large Language Models Are Neurosymbolic Reasoners

#26

Post by clooper »

microtonal wrote: post #24
clooper wrote: post #5

I was recently thinking how every neural network is equivalent to a lookup table where the input is all numbers up to what can be expressed within the context window and the output is the result of the arithmetic operations applied to that number. So every neural network is equivalent to T = {(i, f(i)) : i The main practical issue is the size of the table but I don't see any theoretical reasons why this is incorrect.…

Suppose that we used embeddings as the input of the model rather than piece identifiers plus an embedding lookup table. This is possible with every transformer model and some libraries provide an API to do this. Moreover, we convert the parameters and ops to use arbitrary precision types. Then the network cannot be represented as a lookup table. Given that there is an infinite number of inputs, there is also an infin…

Even if you increase the precision I can still generate a table T(P) for each fixed precision P. So the table is parametrized by P but it's still a table. The entire table T = colim T(P) is the colimit over all precision values but for every finite precision it is still a table.

Re: Large Language Models Are Neurosymbolic Reasoners

#27

Post by ccozan »

clooper wrote: post #23
mewpmewp2 wrote: post #20

Earlier quoted context omitted.

You could represent all the input on different levels as numbers, e.g. all EM waves hitting our eyes, then all the physical output from our body also as numbers, and everything that causes this output from input within is what you would consider to be a lookup table.

What are the dimension of the input and output spaces involved in this idealization? In the case of a neural network there is no idealization. The network is software, it's a number. It's inputs and outputs are all bounded and can be expressed as a table of bounded tuples.

I think the OP is right. All the input to a human brain can be expressed as numbers, at any given time a specific radiation, vibration, or chemical reaction is hitting our "sensors" and by the law of physics this is just numbers ( in terms of differentiation, brain does not know absolute values ).

Our output ( mechanical and vibrations ) is also fully quantifiable, thus numbers.

One giant lookup table.

Re: Large Language Models Are Neurosymbolic Reasoners

#28

Post by edflsafoiewq »

clooper wrote: post #5

I was recently thinking how every neural network is equivalent to a lookup table where the input is all numbers up to what can be expressed within the context window and the output is the result of the arithmetic operations applied to that number. So every neural network is equivalent to T = {(i, f(i)) : i The main practical issue is the size of the table but I don't see any theoretical reasons why this is incorrect.…

This reminds me of the classic problem in computation, where the simplest form of computation, the lookup table, input -> output, is limited to a finite domain. Turing modified the computation to have a finite internal state and infinite external environment (tape), so it becomes a transition function (state, stimulus) -> (new state, response), applied recursively in a feedback loop, allowing it to operate on infinite domains.

Famously a simple lookup table for the transition function then suffices to compute any computable function.

Re: Large Language Models Are Neurosymbolic Reasoners

#29

Post by clooper »

edflsafoiewq wrote: post #28
clooper wrote: post #5

I was recently thinking how every neural network is equivalent to a lookup table where the input is all numbers up to what can be expressed within the context window and the output is the result of the arithmetic operations applied to that number. So every neural network is equivalent to T = {(i, f(i)) : i The main practical issue is the size of the table but I don't see any theoretical reasons why this is incorrect.…

This reminds me of the classic problem in computation, where the simplest form of computation, the lookup table, input -> output, is limited to a finite domain. Turing modified the computation to have a finite internal state and infinite external environment (tape), so it becomes a transition function (state, stimulus) -> (new state, response), applied recursively in a feedback loop, allowing it to operate on infinit…

That's a good point.

Re: Large Language Models Are Neurosymbolic Reasoners

#30

Post by mewpmewp2 »

clooper wrote: post #23
mewpmewp2 wrote: post #20

Earlier quoted context omitted.

You could represent all the input on different levels as numbers, e.g. all EM waves hitting our eyes, then all the physical output from our body also as numbers, and everything that causes this output from input within is what you would consider to be a lookup table.

What are the dimension of the input and output spaces involved in this idealization? In the case of a neural network there is no idealization. The network is software, it's a number. It's inputs and outputs are all bounded and can be expressed as a table of bounded tuples.

You could pick a very large number depending on a reasonable processing capability a human has, which represents all the significant physical interactions on a human body over a certain amount of time. Then take the output over a certain amount of time, being all movements of the body.

If you wanted to focus on thoughts alone, you might want to skip few layers/systems, to give input directly to whatever causes thoughts to happen.

All particles and their interactions could also be represented as numbers. But it just depends on what level we do this, and at what level what kind of complex logic is required.

Post reply on HN