All AI models might be the same • Hacker News Board

Re: All AI models might be the same

#51

Post by stillpointlab » Thu, Jul 17, 2025, 11:16 PM UTC

I have to be careful of confirmation bias when I read stuff like this because I have the intuition that we are uncovering a single intelligence with each of the different LLMs. I even feel, when switching between the big three (OpenAI, Google, Anthropic) that there is a lot of similarity in how they speak and think - but I am aware of my bias so I try not to let it cloud my judgement.

On the topic of compression, I am reminded of an anecdote about Heidegger. Apparently he had a bias towards German and Greek, claiming that these languages were the only suitable forms for philosophy. His claim was based on the "puns" in language, or homonyms. He had some intuition that deep truths about reality were hidden in these over-loaded words, and that the particular puns in German and Greek were essential to understand the most fundamental philosophical ideas. This feels similar to the idea of shared embeddings being a critical aspect of LLM emergent intelligence.

This "superposition" of meaning in representation space again aligns with my intuitions. I'm glad there are people seriously studying this.

Re: All AI models might be the same

#52

Post by godelski » Thu, Jul 17, 2025, 11:21 PM UTC

zos_kia wrote: post #47

cdrini wrote: post #43

Earlier quoted context omitted.

Hmm I don't think we'd need a rosetta stone. In the same way LLMs associate via purely contextual usage the meaning of words, two separate data sets of lion and English, encoded into the same vector space, might pick up patterns of contextual usage at a high enough level to allow for mapping between the two languages. For example, given thousands of English sentences with the word "sun", the vector embedding encodes…

Someone more knowledgeable might chime in, but I don't think two corpuses can be mapped to the same vector space. Wouldn't each vector space be derived from its corpus?

It depends how you define the vector space but I'm inclined to agree.

The reason I think this is from evidence in human language. Spend time with any translator and they'll tell you that some things just don't really translate. The main concepts might, but there's subtleties and nuances that really change the feel. You probably notice this with friends who have a different native language than you.

Even same language same language communication is noisy. You even misunderstand your friends and partners, right? The people who have the greatest chance of understanding you. It's because the words you say don't convey all the things in your head. It's heavily compressed. Then the listener has to decompress from those lossy words. I mean you can go to any Internet forum and see this in action. That there's more than one way to interpret anything. Seems most internet fights start this way. So it's good to remember that there isn't an objective communication. We improperly encode as well as improperly decode. It's on us to try to find out what the speaker means, which may be very different from the words they say (take any story or song to see the more extreme versions of this. This feature is heavily used in art)

Really, that comes down to the idea of universal language[0]. I'm not a linguist (I'm an AI researcher), but my understanding is most people don't believe it exists and I buy the arguments. Hard to decouple due to shared origins and experiences.

[0] https://en.wikipedia.org/wiki/Universal_language

Re: All AI models might be the same

#53

Post by groby_b » Thu, Jul 17, 2025, 11:23 PM UTC

coffeecoders wrote: post #16

I think “we might decode whale speech or ancient languages” is a huge stretch. Context is the most important part of what makes language useful. There is billions of human-written texts, grounded in shared experience that makes our AI good at language. We don't have that for a whale.

That's not necessarily what matters.

What matters is if there is a shared representation space across languages. If there is, you can then (theoretically, there might be a PhDs and a Nobel or two to be had :) separate underlying structure and the translation from underlying structure to language.

The latter - what they call the universal embedding inverter - is likely much more easily trainable. There's a good chance that certain structures are unique enough you can map them to underlying representation, and then lever that. But even if that's not viable, you can certainly run unsupervised training on raw material, and see if that same underlying "universal" structure pops out.

There's a lot of hope and conjecture in that last paragraph, but the whole point of the article is that maybe, just maybe, you don't need context to translate.

Re: All AI models might be the same

#54

Post by cdrini » Thu, Jul 17, 2025, 11:25 PM UTC

zos_kia wrote: post #47

cdrini wrote: post #43

Earlier quoted context omitted.

Hmm I don't think we'd need a rosetta stone. In the same way LLMs associate via purely contextual usage the meaning of words, two separate data sets of lion and English, encoded into the same vector space, might pick up patterns of contextual usage at a high enough level to allow for mapping between the two languages. For example, given thousands of English sentences with the word "sun", the vector embedding encodes…

Someone more knowledgeable might chime in, but I don't think two corpuses can be mapped to the same vector space. Wouldn't each vector space be derived from its corpus?

That's a very good point! I hadn't thought of that. And that makes sense, since the encoding of the word "sun" arises from its linguistic context, and there's no such shared context between the English word sun and any lion word in this imaginary multilingual corpus, so I don't think they'd go to the same point.

Apparently one thing you could do is train a word2vec on each corpus and then align them based on proximity/distances. Apparently this is called "unsupervised" alignment and there's a tool by Facebook called MUSE to do it. (TIL, Thanks ChatGPT!) https://github.com/facebookresearch/MUSE?tab=readme-ov-file

Although I wonder if there are better embedding approaches now as well. Word2Vec is what I've played around with from a few years ago, I'm sure it's ancient now!

Edit: that's what I get for posting before finishing the article! The whole point of their researh is to try to build such a mapping, ve2vec!

Re: All AI models might be the same

#55

Post by b112 » Thu, Jul 17, 2025, 11:33 PM UTC

stillpointlab wrote: post #51

I have to be careful of confirmation bias when I read stuff like this because I have the intuition that we are uncovering a single intelligence with each of the different LLMs. I even feel, when switching between the big three (OpenAI, Google, Anthropic) that there is a lot of similarity in how they speak and think - but I am aware of my bias so I try not to let it cloud my judgement. On the topic of compression, I a…

LLMs don't think, nor are they intelligent or exhibiting intelligence.

Language does have constraints, yet it evolves via its users to encompass new meanings.

Thus those constraints are artificial, unless you artificially enforce static language use. And of course, for an LLM to use those new concepts, it needs to be retokenized by being trained on new data.

For example, if we trained LLMs only on books, encyclopedias, newpapers, and personal letters from 1850, it would have zero capacity to speak comprehensibly or even seem cogent on much of the modern world.

And it would forever remain in that disconnected positon.

LLMs do not think, understand anything, nor learn. Should you wish to call tokenization, learning, then you'd better call a clock "learning" from the gears and cogs that enable its function.

LLMs do not think, learn, or exhibit intelligence. (I feel this is not said enough).

We will never, ever get AGI from an LLM. Ever.

I am sympathetic to the wonder of LLMs. To seeing them as such. But I see some art as wonderous too. Some machinery is beautiful in execution and to use.

But that doesn't change truths.

Re: All AI models might be the same

#56

Post by cdrini » Thu, Jul 17, 2025, 11:35 PM UTC

godelski wrote: post #52

zos_kia wrote: post #47

Earlier quoted context omitted.

Someone more knowledgeable might chime in, but I don't think two corpuses can be mapped to the same vector space. Wouldn't each vector space be derived from its corpus?

It depends how you define the vector space but I'm inclined to agree. The reason I think this is from evidence in human language. Spend time with any translator and they'll tell you that some things just don't really translate. The main concepts might, but there's subtleties and nuances that really change the feel. You probably notice this with friends who have a different native language than you. Even same language…

Hmm I don't think a universal language is implied by being able to translate without a rosetta stone. I agree, I don't think there is such a thing as a universal language, per se, but I do wonder if there is a notion of a universal language at a certain level of abstraction.

But I think those ambiguous cases can still be understood/defined. You can describe how this one word in lion doesn't neatly map to a single word in English, and is used like a few different ways. Some of which we might not have a word for in English, in which case we would likely adopt the lion word.

Although note I do think I was wrong about embedding a multilingual corpus into a single space. The example I was thinking of was word2vec, and that appears to only work with one language. Although I did find some papers showing that you can unsupervised align between the two spaces, but don't know how successful that is, or how that would treat these ambiguous cases.

Re: All AI models might be the same

#57

Post by b112 » Thu, Jul 17, 2025, 11:37 PM UTC

kindkang2024 wrote: post #46

coffeecoders wrote: post #16

I think “we might decode whale speech or ancient languages” is a huge stretch. Context is the most important part of what makes language useful. There is billions of human-written texts, grounded in shared experience that makes our AI good at language. We don't have that for a whale.

If we could help gorillas or elephants (both highly intelligent) learn to name things and use symbols — in a form they can comprehend and create to express their will — enabling them to pass down their experiences and wisdom across generations, I believe they could quietly be as smart as we are. Ps. I am excited about Google’s Gemma dolphin project ( https://blog.google/technology/ai/dolphingemma/ ), but I would pref…

I don't know why, but I just had a horrible vision of traveling 200 years hence, and elephants are now the ruling class.

I don't care that it might be better, and I shall hold you personally responsible should this come to pass.

Re: All AI models might be the same

#58

Post by benlivengood » Thu, Jul 17, 2025, 11:39 PM UTC

lsy wrote: post #49

The example given for inverting an embedding back to text doesn't help the idea that this effect is reflecting some "shared statistical model of reality": What would be the plausible whalesong mapping of "Mage (foaled April 18, 2020) is an American Thoroughbred racehorse who won the 2023 Kentucky Derby"? There isn't anything core to reality about Kentucky, its Derby, the Gregorian calendar, America, horse breeds, etc…

You also can't translate "Mage (foaled April 18, 2020) is an American Thoroughbred racehorse who won the 2023 Kentucky Derby" into Hellenistic Greek or some modern indigenous languages because there isn't enough shared context; you'd need to give humans speaking those languages a glossary for any of the translation to make sense, or allow them to interrogate an LLM to act as the glossary.

I'd say our current largest LLMs probably contain sufficient detail to explain a concept like a named race horse starting from QCD+gravity and ending up at cultural human events, given a foothold of some common ground to translate into a new unknown language. In a sense, that's what a model of reality is. I think it's possible because LLMs figure out translation between human languages by default with enough pretraining.

Re: All AI models might be the same

#59

Post by Quarrelsome » Thu, Jul 17, 2025, 11:43 PM UTC

empath75 wrote: post #9

This is kind of fascinating because I just tried to play mussolini or bread with chatgpt and it is absolutely _awful_ at it, even with reasoning models. It just assumes that your answers are going to be reasonably bread-like or reasonably mussolini-like, and doesn't think laterally at all. It just kept asking me about varieties of baked goods. edit: It did much better after I added some extra explanation -- that it c…

what surprised me when I asked if it knew what Mussolini or Bread was, it parroted this article. This article was posted in the last 24 hours and querying a search engine for "Mussolini or Bread" didn't yield it anywhere near a top result.

Re: All AI models might be the same

#60

Post by klank » Thu, Jul 17, 2025, 11:44 PM UTC

godelski wrote: post #50

klank wrote: post #23

Earlier quoted context omitted.

If a lion could speak, would we understand it?

I don't know about a Lion, but I think Wittgenstein could have benefited from having a pet. I train my cat and while I can't always understand her I think one of the most impressive features of the human mind is to be able to have such great understanding of others. We have theory of mind, joint attention, triadic awareness, and much more. My cat can understand me a bit but it's definitely asymmetric. It's definitely…

I think the response is generally you are communicating with your cat as an animal, as a mammal. Yes, communication is possible because we too are mammals, animals, etc.

But Lion is not just animal, it is not just mammal, it is something more. Something which I have no idea how we would communicate with.