Large Language Models Are Neurosymbolic Reasoners
1–10 of 172 posts
Re: Large Language Models Are Neurosymbolic Reasoners
#2Yep, just along as it is fed biased data, it can still reason based on faulty info.
Re: Large Language Models Are Neurosymbolic Reasoners
#3I...
"It is a tale Told by an idiot, full of sound and fury Signifying nothing."
This is sort of water is wet kind of research. Im glad they did it but it's not exactly moving the ball down the field.
Re: Large Language Models Are Neurosymbolic Reasoners
#4Not really
Re: Large Language Models Are Neurosymbolic Reasoners
#5I was recently thinking how every neural network is equivalent to a lookup table where the input is all numbers up to what can be expressed within the context window and the output is the result of the arithmetic operations applied to that number. So every neural network is equivalent to T = {(i, f(i)) : i The main practical issue is the size of the table but I don't see any theoretical reasons why this is incorrect. The neural network is simply a compressed representation of the uncompressed lookup table. Given that the two representations are theoretically equivalent and a lookup table does not perform any reasoning we can conclude that no neural network is actually doing any thinking other than uncompressing the table and looking up the value corresponding to the input number.
Modern neural networks have some randomness but that doesn't change the table in any meaningful way because instead of the output being a number it becomes a distribution over some finite range which can again be turned into a table with some tuples.
Re: Large Language Models Are Neurosymbolic Reasoners
#6The authors get LLMs to perform pretty well in a variety of IF-style text based games. Which is pretty cool, these kinds of games are played and read in natural language, which makes them pretty hard to write AIs for normally.
Something I'd love to see one day is modern AI applied to other kinds of text based games like nethack. Last I checked nobody had managed to solve the problem of nethack AI without using hard coded heuristics and goals!
Re: Large Language Models Are Neurosymbolic Reasoners
#7I was recently thinking how every neural network is equivalent to a lookup table where the input is all numbers up to what can be expressed within the context window and the output is the result of the arithmetic operations applied to that number. So every neural network is equivalent to T = {(i, f(i)) : i The main practical issue is the size of the table but I don't see any theoretical reasons why this is incorrect.…
This is an old argument against determinism - I think a serious challenge is that:
1. Modern physics suggests you can implement such a lookup table for any subset of our universe.
2. We are a subset of the universe.
3. Therefore we are representable by lookup tables too.
...so your argument appears to prove too much, namely that humans aren't thinking beings either. Which is fine, but personally I don't think that's a useful definition of "thinking".
Re: Large Language Models Are Neurosymbolic Reasoners
#8I was recently thinking how every neural network is equivalent to a lookup table where the input is all numbers up to what can be expressed within the context window and the output is the result of the arithmetic operations applied to that number. So every neural network is equivalent to T = {(i, f(i)) : i The main practical issue is the size of the table but I don't see any theoretical reasons why this is incorrect.…
This is an old argument against determinism - I think a serious challenge is that: 1. Modern physics suggests you can implement such a lookup table for any subset of our universe. 2. We are a subset of the universe. 3. Therefore we are representable by lookup tables too. ...so your argument appears to prove too much, namely that humans aren't thinking beings either. Which is fine, but personally I don't think that's…
How are people lookup tables? In the case of neural networks the representation of the table is obvious, it's just numbers. What would be the equivalent table for the liver?
My argument isn't abstract. Neural networks really are just numerical functions which can be expanded into their equivalent graph representations.
Re: Large Language Models Are Neurosymbolic Reasoners
#9I was recently thinking how every neural network is equivalent to a lookup table where the input is all numbers up to what can be expressed within the context window and the output is the result of the arithmetic operations applied to that number. So every neural network is equivalent to T = {(i, f(i)) : i The main practical issue is the size of the table but I don't see any theoretical reasons why this is incorrect.…
It sounds like you're asking whether the output of a neural network is a deterministic function of its input. For many LLMs, you can make that answer yes with the right combination of parameters (temperature = 0) and underlying compute (variance in floating point calculations can still introduce randomness in model outputs even when the model should theoretically return the same answer every time).
There are some ways to introduce stochasticity:
1. Add randomness. The temperature or "creativity" hyperparameter in most LLMs does this, as do some decoders. The hardware these models run can also add randomness.
2. Add some concept of state. RNNs do this, some of the approaches which give the LLM a scratch pad or external memory do this, and continuous pre-training sort of does this.
How this affects people's perception of LLMs as thinking machines, I don't know. What if someone took every response I ever gave to every question that was ever asked of me in my life and made a Chinese Room[1] version of me? A lookup table that is functionally identical to my entire existence. In what contexts is the difference meaningful?
Re: Large Language Models Are Neurosymbolic Reasoners
#10[deleted]