Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Large Language Models Are Neurosymbolic Reasoners

arxiv.org

Post reply on HN
Print view View original

31–40 of 172 posts

Re: Large Language Models Are Neurosymbolic Reasoners

#31

Post by clooper »

ccozan wrote: post #27
clooper wrote: post #23

Earlier quoted context omitted.

What are the dimension of the input and output spaces involved in this idealization? In the case of a neural network there is no idealization. The network is software, it's a number. It's inputs and outputs are all bounded and can be expressed as a table of bounded tuples.

I think the OP is right. All the input to a human brain can be expressed as numbers, at any given time a specific radiation, vibration, or chemical reaction is hitting our "sensors" and by the law of physics this is just numbers ( in terms of differentiation, brain does not know absolute values ). Our output ( mechanical and vibrations ) is also fully quantifiable, thus numbers. One giant lookup table.

Provide some concrete numbers for solar radiation then as a lookup table. You guys are confusing abstraction and idealization with what it means to be a thinking person. There is no such abstraction and idealization happening with software. The software is really just a number, there is no idealization or abstraction happening when I claim that GPT is a sequence of bits representing a numerical function.

Re: Large Language Models Are Neurosymbolic Reasoners

#32

Post by microtonal »

clooper wrote: post #26
microtonal wrote: post #24

Earlier quoted context omitted.

Suppose that we used embeddings as the input of the model rather than piece identifiers plus an embedding lookup table. This is possible with every transformer model and some libraries provide an API to do this. Moreover, we convert the parameters and ops to use arbitrary precision types. Then the network cannot be represented as a lookup table. Given that there is an infinite number of inputs, there is also an infin…

Even if you increase the precision I can still generate a table T(P) for each fixed precision P. So the table is parametrized by P but it's still a table. The entire table T = colim T(P) is the colimit over all precision values but for every finite precision it is still a table.

I did not say fixed precision. I said arbitrary precision, so P is infinite.

The only counter-argument is that even arbitrary precision is fixed-precision because computer memory is finite. But that's kind of a silly argument, because then you are arguing that computers can never reason, because they have finite memory, and moreover humans cannot reason either, because there is a finite number of brain cells.

Re: Large Language Models Are Neurosymbolic Reasoners

#33

Post by Legend2440 »

clooper wrote: post #5

I was recently thinking how every neural network is equivalent to a lookup table where the input is all numbers up to what can be expressed within the context window and the output is the result of the arithmetic operations applied to that number. So every neural network is equivalent to T = {(i, f(i)) : i The main practical issue is the size of the table but I don't see any theoretical reasons why this is incorrect.…

Here's a counterexample. Suppose I create a simple neural network that computes f(x) = x^2 + c (where x and c are complex numbers) and then I run it as an RNN. This RNN will compute the mandelbrot set, which can't be represented by a lookup table.

You can't even know if the RNN will halt for a given input. Neural networks are stronger than lookup tables, they are programs.

Re: Large Language Models Are Neurosymbolic Reasoners

#34

Post by clooper »

microtonal wrote: post #32
clooper wrote: post #26

Earlier quoted context omitted.

Even if you increase the precision I can still generate a table T(P) for each fixed precision P. So the table is parametrized by P but it's still a table. The entire table T = colim T(P) is the colimit over all precision values but for every finite precision it is still a table.

I did not say fixed precision. I said arbitrary precision, so P is infinite. The only counter-argument is that even arbitrary precision is fixed-precision because computer memory is finite. But that's kind of a silly argument, because then you are arguing that computers can never reason, because they have finite memory, and moreover humans cannot reason either, because there is a finite number of brain cells.

P obviously can't be infinite, even in theory, if you want the computation to terminate.

Re: Large Language Models Are Neurosymbolic Reasoners

#35

Post by clooper »

mewpmewp2 wrote: post #30
clooper wrote: post #23

Earlier quoted context omitted.

What are the dimension of the input and output spaces involved in this idealization? In the case of a neural network there is no idealization. The network is software, it's a number. It's inputs and outputs are all bounded and can be expressed as a table of bounded tuples.

You could pick a very large number depending on a reasonable processing capability a human has, which represents all the significant physical interactions on a human body over a certain amount of time. Then take the output over a certain amount of time, being all movements of the body. If you wanted to focus on thoughts alone, you might want to skip few layers/systems, to give input directly to whatever causes though…

Ok so give me some concrete number.

Re: Large Language Models Are Neurosymbolic Reasoners

#36

Post by microtonal »

clooper wrote: post #34
microtonal wrote: post #32

Earlier quoted context omitted.

I did not say fixed precision. I said arbitrary precision, so P is infinite. The only counter-argument is that even arbitrary precision is fixed-precision because computer memory is finite. But that's kind of a silly argument, because then you are arguing that computers can never reason, because they have finite memory, and moreover humans cannot reason either, because there is a finite number of brain cells.

P obviously can't be infinite, even in theory, if you want the computation to terminate.

Right, but then as others said, then you are also arguing that humans cannot reason, since the universe is a system with a finite number of particles. Or if we exclude external factors, because humans have a finite number of brain cells.

In the end it all depends on what your definition of reasoning is, which you did not provide.

Re: Large Language Models Are Neurosymbolic Reasoners

#37

Post by bubblyworld »

mjburgess wrote: post #11
bubblyworld wrote: post #7

Earlier quoted context omitted.

This is an old argument against determinism - I think a serious challenge is that: 1. Modern physics suggests you can implement such a lookup table for any subset of our universe. 2. We are a subset of the universe. 3. Therefore we are representable by lookup tables too. ...so your argument appears to prove too much, namely that humans aren't thinking beings either. Which is fine, but personally I don't think that's…

We're not a lookup table of the things we're, eg., saying, or doing etc. Nor are we looking up, in this sense, when we act. ie., when you compress text into an NN and use it to generate text, the generated text is just a synthesis of the compressed text . Whereas when I type, I am not synthesising text . Rather I have the skill of typing, I have an interior subjectivity of thoughts, I have memories which arent text,…

I strongly disagree with your last statement - physics explicitly _is_ formulated in terms that can be made into a lookup table (see phase spaces in classical mechanics, for instance).

My point is that there's a finite light cone of possible causal influences over you at any moment in time, and in principle you can break those down into state variables finely enough to predict future states of a person. This is isomorphic to a lookup table, albeit one we aren't able to construct right now.

Im not suggesting it's enough to consider just the person in this scenario - the causal factors are part of the lookup.

Re: Large Language Models Are Neurosymbolic Reasoners

#38

Post by cjfd »

Legend2440 wrote: post #33
clooper wrote: post #5

I was recently thinking how every neural network is equivalent to a lookup table where the input is all numbers up to what can be expressed within the context window and the output is the result of the arithmetic operations applied to that number. So every neural network is equivalent to T = {(i, f(i)) : i The main practical issue is the size of the table but I don't see any theoretical reasons why this is incorrect.…

Here's a counterexample. Suppose I create a simple neural network that computes f(x) = x^2 + c (where x and c are complex numbers) and then I run it as an RNN. This RNN will compute the mandelbrot set, which can't be represented by a lookup table. You can't even know if the RNN will halt for a given input. Neural networks are stronger than lookup tables, they are programs.

I am sorry to be this blunt but this is really utter and complete nonsense. The phrase that the mandelbrot set can't be represented in a lookup table is as such true but that is because nothing that you do with finite precision numbers can represent the mandelbrot set because it essentially is an inifinte object. The function f(x) = x^2 + c as an RNN can also not compute the mandelbrot set if the numbers it uses are of finite precision. That is exactly the same limitation that the lookup table also faces so there is no fundamental difference between the two.

Re: Large Language Models Are Neurosymbolic Reasoners

#40

Post by clooper »

microtonal wrote: post #36
clooper wrote: post #34

Earlier quoted context omitted.

P obviously can't be infinite, even in theory, if you want the computation to terminate.

Right, but then as others said, then you are also arguing that humans cannot reason, since the universe is a system with a finite number of particles. Or if we exclude external factors, because humans have a finite number of brain cells. In the end it all depends on what your definition of reasoning is, which you did not provide.

The bit precision of computation is always finite for halting computations and any finite computation can be turned into a lookup table which does no thinking or reasoning other than comparing two numbers and then extracting the value corresponding to the input key.

My argument carries through for any piece of software so if you think software can think and reason then you can remain unconvinced by my argument.

In any case, I have to drop out of this thread.

Post reply on HN