Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Language Models Represent Space and Time

arxiv.org

Post reply on HN
Print view View original

91–100 of 192 posts

Re: Language Models Represent Space and Time

#91

Post by somewhereoutth »

ilaksh wrote: post #30

One thing that is clouding this discussion is that most people are mixing up a lot of different characteristics that animals like humans have as if they were all the same thing. So for many people they don't really distinguish between things like "reasoning", "self-aware", "conscious", "alive", "sentient", "intelligent", "has world model". They also don't distinguish between different types or varying levels of cogni…

If an LLM is trained on just text, then it's internal model can only be based on such text.

I think the 'conceit' of LLM as AI advocates is that this model is just as (or at least nearly as) our own internal model - that the 'simplest' model to match the text is indeed our 'real' model that matches reality as a whole. A bit of a stretch surely!

Re: Language Models Represent Space and Time

#92

Post by andrewguenther »

YeGoblynQueenne wrote: post #19
andrewguenther wrote: post #8

Earlier quoted context omitted.

I think LLMs are presenting some uncomfortable philosophical questions for people about how our own brains work and admitting that there is any kind of "intelligence" (even if very basic) in an LLM is an admission that our own brains may work in a similar manner.

For me, one of the most interesting things that have come out of LLMs is the confirmation that humans are very bad at reasoning and, consequently, its' a very bad idea to try and make machines that "think like humans", because that way we'll only make machines with none of the advantages of machines and all the disadvantages of computers. For instance -I'm not trying to be mean and I'm certainly not blaming you in pa…

No offense taken. I tried to be clear in my phrasing that I don't personally subscribe to the "LLMs are just like us" mentality. Just making an observation as to why people have such visceral reactions to any implication that they might be.

Re: Language Models Represent Space and Time

#93

Post by ImHereToVote »

nyrikki wrote: post #89
ImHereToVote wrote: post #79

Earlier quoted context omitted.

Great so biological neurons can perform feats of intuition, that simulated neurons simply can't. Even theoretically. Gotcha. Thank you very much for the explanation.

Not quite that simple, or well known. We know that the class of total Turing computable functions is not learnable in the limit. But as to what else is AI-complete is mostly open questions. But there is a difference. Obviously ML is far better than humans in some domains so it is not a simple dichotomy.

So what is it about a biological neuron that sidesteps the halting problem? Is it because there is a weight and bias set to it. Someone should tell OpenAI to add weights and biases to their LLMs. That will make them make have an emotional like intuition, will it not?

EDIT: Maybe if we train the ANNs to focus their training on attention based techniques. Then they will simply tire on halts and continue on other problems until the model has sufficiently grokked lesser problems to focus on the previous halts.

Re: Language Models Represent Space and Time

#94

Post by anigbrowl »

SiempreViernes wrote: post #24
anigbrowl wrote: post #9

Earlier quoted context omitted.

Indeed. I was offering examples of how you could bootstrap cognitive processes at the beginning of the year and people were just sticking their fingers in their ears. A lot of people exercise their intellectual, economic, or political freedoms by rejecting others' efforts to gain the same things.

Who is the subject in the last bit of your last sentence? More pointedly: are you claiming that OpenAI is doing LLM research to gain political freedoms?

No, the second sentence is a general observation on a widespread human behavior pattern. I don't feel like expanding it into an essay right now.

Re: Language Models Represent Space and Time

#95

Post by anigbrowl »

OneDonOne wrote: post #65
anigbrowl wrote: post #9

Earlier quoted context omitted.

Indeed. I was offering examples of how you could bootstrap cognitive processes at the beginning of the year and people were just sticking their fingers in their ears. A lot of people exercise their intellectual, economic, or political freedoms by rejecting others' efforts to gain the same things.

>A lot of people exercise their intellectual, economic, or political freedoms by rejecting others' efforts to gain the same things. Could you please explain what the above has to do with a Large Languange Model?

I'm talking about how many people react to any startling new information or idea. The advent of LLMs is just a relevant example.

Re: Language Models Represent Space and Time

#96

Post by fiso64 »

lsy wrote: post #90
nopinsight wrote: post #86

Earlier quoted context omitted.

Current LLMs clearly have much coarser models of the world than humans do. Their training data do not include as many modalities and training data with spatiotemporal dimensions are inadequate. Up and coming multimodal models are changing this.

"Multimodal" models only promise an additional layer of indirection and interpretation from the world through sensors into the same conceptual mechanism. An image is converted to tokens, a sensor is converted to coordinates, and it is all fed back into a correlating model. But this isn't adequate either. The famous dress illusion picture has RGB values closer to gold and white, but the actual physical dress was black…

Tokens are just another representation of the image, like 3 matrices or electrical impulses in our brain. They probably encode most of the relevant information, the question is whether the linear algebra at the core of a transformer is good enough to use this data as efficiently as we do. I don't think we can currently say if it is or isn't adequate.

Re: Language Models Represent Space and Time

#97

Post by nyrikki »

famouswaffles wrote: post #88
ImHereToVote wrote: post #79

Earlier quoted context omitted.

Great so biological neurons can perform feats of intuition, that simulated neurons simply can't. Even theoretically. Gotcha. Thank you very much for the explanation.

Yeah that's not the case lol. ann's can model any function even non linear ones.

Pairwise linear in respect to inputs, for feed forward networks like those that use attention is the restriction related to linear or linearizable.

Those words are important.

But we know that ML cannot model all turning machines in the limit. As to more reasonable models that gets more complex.

Rice's theorem in respect to functions doesn't fit with you claim:

For any non-trivial property of partial functions, no general and effective method can decide whether an algorithm computes a partial function with that property.

The generalization problem being the important part.

But if you want to claim that ANNs solve all of the undecidable problems too, write the paper and become famous.

Re: Language Models Represent Space and Time

#98

Post by OneDonOne »

anigbrowl wrote: post #95
OneDonOne wrote: post #65

Earlier quoted context omitted.

>A lot of people exercise their intellectual, economic, or political freedoms by rejecting others' efforts to gain the same things. Could you please explain what the above has to do with a Large Languange Model?

I'm talking about how many people react to any startling new information or idea. The advent of LLMs is just a relevant example.

No new ideas have been advanced in anything reported here. A state machine change state is all that is going on here.

What I'd like to know is why should anyone care about this preprint? Nothing has been stated on how this software will improve the life of the average human being.

LK-99 at least would have had lead.somewhere if it worked.

Re: Language Models Represent Space and Time

#99

Post by mjburgess »

somewhereoutth wrote: post #91
ilaksh wrote: post #30

One thing that is clouding this discussion is that most people are mixing up a lot of different characteristics that animals like humans have as if they were all the same thing. So for many people they don't really distinguish between things like "reasoning", "self-aware", "conscious", "alive", "sentient", "intelligent", "has world model". They also don't distinguish between different types or varying levels of cogni…

If an LLM is trained on just text, then it's internal model can only be based on such text. I think the 'conceit' of LLM as AI advocates is that this model is just as (or at least nearly as) our own internal model - that the 'simplest' model to match the text is indeed our 'real' model that matches reality as a whole. A bit of a stretch surely!

LLMs are a redemonstration of the lesson of sun worship: apes are easily fooled.

If the output is indistinguishably 'world modelling', so must its mechanism be.

There is no 'world' model in a hashmap from questions to answers about a world. There is only the condition that, given those questions, any old ape would accept those answers.

Science is required to determine whether a system has properties, such as a rational ecology; ie. a genuine mechanism of environmental coordination.

No engineer, no statistician, and certainly no VC HN Hype Man can determine-so. The nature of reality is no up for Big-O equivalences.

Systems have properties in virtue of their nature, not in virute of what I will pay $20/mo for

Re: Language Models Represent Space and Time

#100

Post by essive »

ilaksh wrote: post #30

One thing that is clouding this discussion is that most people are mixing up a lot of different characteristics that animals like humans have as if they were all the same thing. So for many people they don't really distinguish between things like "reasoning", "self-aware", "conscious", "alive", "sentient", "intelligent", "has world model". They also don't distinguish between different types or varying levels of cogni…

I think the term "world model" is a highly overloaded term. What does it really mean in your opinion?
Post reply on HN