Much simpler models like word2vec also showed linear embeddings of geography in this way.
Language Models Represent Space and Time
11–20 of 192 posts
Re: Language Models Represent Space and Time
#12Space and time (the abstractions) are lingisuric concepts, so this is perhaps not too surprising, but nevertheless interesting.
Re: Language Models Represent Space and Time
#13I’m most surprised that the embedding is actually linear. I wonder if that makes extrapolation perform better
Re: Language Models Represent Space and Time
#14Space and time (the abstractions) are lingisuric concepts, so this is perhaps not too surprising, but nevertheless interesting.
The space and time mapping they pull out of the LLM here is non-linguistic.
Re: Language Models Represent Space and Time
#15No, no they don't. People looking at the output of LLMs project their own internal models of the world, time, and space onto the output and project that model as if it were the LLM's.
You’re responding to the headline. Go look at the paper
Re: Language Models Represent Space and Time
#16I think we have a new member to add to the group: Lies, damn lies, and statistics… and now neural activations. How exciting! This reminds me of all those fMRI studies that look at brain blood flow before the dead salmon experiment came out.
Haha, I have no idea how I was not aware of the dead salmon experiment prior. This article [1] has a nice overview for anybody else in my shoes. Thanks for the reference.
[1] - https://blogs.scientificamerican.com/scicurious-brain/ignobe...
Re: Language Models Represent Space and Time
#17A lot of HNers were so adamant that the LLMs understand absolutely nothing and that these models are just predicting the next most likely word. I think with this paper it becomes clear that the adamant denial was just human bias talking. LLMs crossed a certain line here. You would think people would be amazed or in awe or react in fear at technological break through a and they often are. The weird part for LLMs was t…
My only experience so far is with chatgpt, and I am not an expert in AI, or even just in LLMs.
With those disclaimers, my interactions inform my opinion that the LLM behind chatgpt has no internal world model. It shows no understanding of basic facts and makes very silly mistakes very easily. I have my bias, like anyone else, but in the case of AI in particular, I should say that I don't think there's anything especially magical or sacred about conscience and the human brain (or the brain of other animals, for that matter), and I'm sure it must be possible to arrive at different forms of intelligence starting with the "inanimate" building blocks of hardware and software, but what I've seen so far in LLMs doesn't make me support your statement that they've crossed a certain line. In fact, I'm very much let down by the experience and I'm afraid once the hype goes down, we may be in for another AI winter.
As I said at the start of the comment, I'm well aware I'm not a subject-matter expert. I'm open to being wrong. I just wanted to point out that not all of us unimpressed are in denial. I'd love to be in awe at an AI breakthrough, and I kind of feel I will be in awe sometime during my lifetime, but LLMs are not (yet?) that for me.
Edit: s/are not in denial/are in denial/
Re: Language Models Represent Space and Time
#18Much simpler models like word2vec also showed linear embeddings of geography in this way.
source?
Re: Language Models Represent Space and Time
#19A lot of HNers were so adamant that the LLMs understand absolutely nothing and that these models are just predicting the next most likely word. I think with this paper it becomes clear that the adamant denial was just human bias talking. LLMs crossed a certain line here. You would think people would be amazed or in awe or react in fear at technological break through a and they often are. The weird part for LLMs was t…
I think LLMs are presenting some uncomfortable philosophical questions for people about how our own brains work and admitting that there is any kind of "intelligence" (even if very basic) in an LLM is an admission that our own brains may work in a similar manner.
For me, one of the most interesting things that have come out of LLMs is the confirmation that humans are very bad at reasoning and, consequently, its' a very bad idea to try and make machines that "think like humans", because that way we'll only make machines with none of the advantages of machines and all the disadvantages of computers.
For instance -I'm not trying to be mean and I'm certainly not blaming you in particular, because I've seen this very often- but the reasoning that because LLMs can generate language, and humans can generate language not only LLMs are somehow like humans but also humans are like LLMs is not sound.
For example, walls have ears, cats have ears, therefore walls are like cats and cats are like walls. That doesn't work because walls' ears are not like cats' ears and even if they were, that still wouldn't make walls cats and cats walls, it would just make them both entities with ears.
Re: Language Models Represent Space and Time
#20A lot of HNers were so adamant that the LLMs understand absolutely nothing and that these models are just predicting the next most likely word. I think with this paper it becomes clear that the adamant denial was just human bias talking. LLMs crossed a certain line here. You would think people would be amazed or in awe or react in fear at technological break through a and they often are. The weird part for LLMs was t…
My only experience so far is with chatgpt, and I am not an expert in AI, or even just in LLMs. With those disclaimers, my interactions inform my opinion that the LLM behind chatgpt has no internal world model. It shows no understanding of basic facts and makes very silly mistakes very easily. I have my bias, like anyone else, but in the case of AI in particular, I should say that I don't think there's anything especi…
Have you used GPT-4?