Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Language Models Represent Space and Time

arxiv.org

Post reply on HN
Print view View original

181–190 of 192 posts

Re: Language Models Represent Space and Time

#181

Post by xcv123 »

YeGoblynQueenne wrote: post #180
xcv123 wrote: post #177

Earlier quoted context omitted.

> Text data doesn't have the information needed to learn a world model from it As I mentioned in another reply you have a philosophical disagreement and are missing the point. It provides "a" model of the world. Not necessarily the "one true model" that is your own personal life experience of the world since birth. There is enough information to build "a" world model in all the text ever written by humans about the w…

I don't do philosophical arguments. But it's obvious this conversation not productive so there's no reason for me to insist.

But you are essentially making a philosophical argument about what defines a "world model". It is a disagreement about a subjective interpretation of the term.

Re: Language Models Represent Space and Time

#182

Post by YeGoblynQueenne »

xcv123 wrote: post #178
YeGoblynQueenne wrote: post #172

Earlier quoted context omitted.

A HMM is one way to train a language model that optimises the conditional objective I note above. A transformer is another. Working with a HMM will give you an insight into how language modelling works, and it's something you can do easily and cheaply, unlike training a giant transformer architecture.

Those primitive obsolete models are not the subject of the article. I already have a degree in applied mathematics and computer science and know the basics of machine learning. My questions to you were rhetorical. I wasn't asking you for academic guidance.

I've already posted in another comment that I don't think there's a reason for me to insist, but I'd just like to point out that it's OK to not be an expert in everything but one should not have strong opinions on matters not of one's expertise. It's not a shame to not know everything, but it's unseemly to act like one does.

Re: Language Models Represent Space and Time

#183

Post by xcv123 »

YeGoblynQueenne wrote: post #180
xcv123 wrote: post #177

Earlier quoted context omitted.

> Text data doesn't have the information needed to learn a world model from it As I mentioned in another reply you have a philosophical disagreement and are missing the point. It provides "a" model of the world. Not necessarily the "one true model" that is your own personal life experience of the world since birth. There is enough information to build "a" world model in all the text ever written by humans about the w…

I don't do philosophical arguments. But it's obvious this conversation not productive so there's no reason for me to insist.

Here's a definition I found somewhere. Do you agree with it?

A "world model" refers to the internal representation of the external environment that an agent (e.g., a human, animal, or machine) maintains and uses to navigate and make decisions within that environment. In other words, it's a mental or computational model of the world as perceived and understood by the agent.

Obviously LLMs have a world model. It's not your favourite model or preferred model or the best model in town, but it is a model of the world whether you like it or not, whether you agree with it or not.

Re: Language Models Represent Space and Time

#184

Post by YeGoblynQueenne »

xcv123 wrote: post #181
YeGoblynQueenne wrote: post #180

Earlier quoted context omitted.

I don't do philosophical arguments. But it's obvious this conversation not productive so there's no reason for me to insist.

But you are essentially making a philosophical argument about what defines a "world model". It is a disagreement about a subjective interpretation of the term.

Be careful about jumping to conclusions just to convince yourself you're right and I'm wrong. I do not assume what you think I assume. But it's obvious you're not looking for a curious conversation and reaching any kind of common understanding.

Re: Language Models Represent Space and Time

#185

Post by xcv123 »

YeGoblynQueenne wrote: post #182
xcv123 wrote: post #178

Earlier quoted context omitted.

Those primitive obsolete models are not the subject of the article. I already have a degree in applied mathematics and computer science and know the basics of machine learning. My questions to you were rhetorical. I wasn't asking you for academic guidance.

I've already posted in another comment that I don't think there's a reason for me to insist, but I'd just like to point out that it's OK to not be an expert in everything but one should not have strong opinions on matters not of one's expertise. It's not a shame to not know everything, but it's unseemly to act like one does.

The exact same would apply to you. Projection. You are extremely arrogant with your opinions that no one else really agrees with and you talk down to people who disagree. Essentially you are insisting that your own personal definition of "world model" is true and the normal meaning of the term is not. Your argument is totally subjective and philosophical.

Re: Language Models Represent Space and Time

#186

Post by xcv123 »

YeGoblynQueenne wrote: post #184
xcv123 wrote: post #181

Earlier quoted context omitted.

But you are essentially making a philosophical argument about what defines a "world model". It is a disagreement about a subjective interpretation of the term.

Be careful about jumping to conclusions just to convince yourself you're right and I'm wrong. I do not assume what you think I assume. But it's obvious you're not looking for a curious conversation and reaching any kind of common understanding.

We are talking about the definition of "world model". If you disagree with it then your argument is not technical, it is philosophical in nature. This is like arguing with a religious fundamentalist.

Re: Language Models Represent Space and Time

#187

Post by fipar »

webdood90 wrote: post #67
fipar wrote: post #17

Earlier quoted context omitted.

My only experience so far is with chatgpt, and I am not an expert in AI, or even just in LLMs. With those disclaimers, my interactions inform my opinion that the LLM behind chatgpt has no internal world model. It shows no understanding of basic facts and makes very silly mistakes very easily. I have my bias, like anyone else, but in the case of AI in particular, I should say that I don't think there's anything especi…

I really don't understand replies like this. This is just the beginning. It is a tiny, tiny view into this new technology. Nobody with an ounce of intelligence looked at the first iPhone and thought "ah well it's just another phone". It's the potential of this new technology that is so impressive. How can you not be impressed by this? I am literally incapable of empathizing with your perspective. All I can think is t…

I'm not sure I follow how you get from me not being impressed by the LLMs I've interacted with (you may be incapable of empathizing with that perspective, but that doesn't make me less unimpressed at them) to some hypothetical people thinking the first iPhone was just another phone, and therefore not having an ounce of intelligence.

I'm not worried about the world passing me by, I think LLMs and similar technology will increase, not decrease, the need for people who can troubleshoot and fix live production software. I mean I do have my passive income if I'm forced into early retirement, but I'm not fearing that to happen soon.

Mind you, I did say I'd love to be in awe at an AI breakthrough; I'm not a luddite. I just don't see anything in LLMs to impress me, and on the other side, I do see a lot of hype behind it.

Re: Language Models Represent Space and Time

#188

Post by fipar »

corethree wrote: post #167
fipar wrote: post #39

Earlier quoted context omitted.

No, I've only used GPT-3 so far. Should I be excited about GPT-4? :) Examples of things I'd file as silly mistakes: - mentioning non-existing settings when asking it about a particular software. - responding to roughly "give me a way to list all rds clusters that are running a version that will reach EOL within the next 12 months" (which I know is almost impossible to do without scraping, as the EOLs are not returned…

Paraphrasing hofsadter? You realize Hofstadter is not delusional about LLMs at all. His view point is completely opposite of yours more logical and rational and he doesn't need to use chatGPT 4. Hofstadter is not a normal person because he is extremely unbiased. chatGPT basically got him to do a 180 on everything he thought and talked about in every book he has written. He flipped, you subscribe to his viewpoints, an…

You seem to be convinced I'm scared of something and in some sort of denial. However, I did say "I don't think there's anything especially magical or sacred about conscience and the human brain (or the brain of other animals, for that matter), and I'm sure it must be possible to arrive at different forms of intelligence starting with the "inanimate" building blocks of hardware and software". I just don't see that in LLMs, or at least not in the ChatGPT version I've interacted with.

I very much respect Hofstadter and the fact that he changed his opinion on GPT (I'm based on what you're saying, I have not read the links you provided yet). I'm not sure that means he changed his core beliefs though, even if he said so. At least, my understanding and interpretation of his work is that we're not the only substrate that can develop "strange loops", and that this is not limited only to organic substrates/living beings. He may have changed his view in GPT, but if he was true to his words, he must have always been open to software being able to develop a being.

I very much think AGI is possible, but:

- I don't think it will happen through LLMs, or at least not through LLMs alone, and

- I don't think it will be human-like. In fact, I expect it will feel very alien to us, and we'll feel very alien to them too. Mostly because we're not brains in jars. We eat, sleep, get laid, feel pain, etc., and all of that feeds back into the loop to make us what we are.

Re: Language Models Represent Space and Time

#189

Post by corethree »

fipar wrote: post #188
corethree wrote: post #167

Earlier quoted context omitted.

Paraphrasing hofsadter? You realize Hofstadter is not delusional about LLMs at all. His view point is completely opposite of yours more logical and rational and he doesn't need to use chatGPT 4. Hofstadter is not a normal person because he is extremely unbiased. chatGPT basically got him to do a 180 on everything he thought and talked about in every book he has written. He flipped, you subscribe to his viewpoints, an…

You seem to be convinced I'm scared of something and in some sort of denial. However, I did say "I don't think there's anything especially magical or sacred about conscience and the human brain (or the brain of other animals, for that matter), and I'm sure it must be possible to arrive at different forms of intelligence starting with the "inanimate" building blocks of hardware and software". I just don't see that in…

I don't think you're scared but I do think you're in denial.

Read what he says. He literally says it's depressing to watch all his core beliefs get dismantled. I'm not making this up. These words came out of his mouth. Follow the links. This is not some minor realization he went through. His entire world view was changed and he's the one claiming this... Not me.

His books which I read are all about recursion and strange loops and he's shocked that LLMs are simply feed forward networks with zero feedback. He claims it debunks everything he wrote.

He has the most to lose in disbelieving what's happening with LLMs. But he's to rational to do what you or most people do. He has to accept the truth and basically admit to being utterly wrong.

Once you read the links you will not agree with him. That's the most likely behavior for most people. But think on this. He has the most to lose and he thinks the books he wrote is wrong while you think the books he wrote is right.

Given the context the conditions are ripe for him to be the most biased and you to be less so, but he believes the opposite against his own self interest. From this you know he's not biased. Most likely you are.

It has nothing to do with fear or dislike of agi. It's just common human bias no one is sure where this lunacy comes from.

Re: Language Models Represent Space and Time

#190

Post by circuit10 »

YeGoblynQueenne wrote: post #171
circuit10 wrote: post #170

Earlier quoted context omitted.

> development of a world model, rather than, or in addition to, a model of the training corpus But the training corpus describes the world, so a model of the training corpus is a world model

A model of a training corpus is a model of a training corpus. Unless a system can read the corpus and understand what it says, it won't get any world model by modelling the corpus.

Then humans also have a model our training corpus, which is an interpretation of the world via our senses. If we don’t ever directly experience the true underlying reality, do we truly understand it? In a way no, but it’s close enough that we call it understanding
Post reply on HN