Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

All AI models might be the same

blog.jxmo.io

Post reply on HN
Print view View original

41–50 of 157 posts

Re: All AI models might be the same

#41

Post by klank »

cdrini wrote: post #38
klank wrote: post #35

Earlier quoted context omitted.

That's fair. To me, the point of Wittgenstein's lion thought experiment though was not necessarily to say that _any_ communication would be impossible. But to understand what it truly meant to be a lion, not just what it meant to be an animal. But we have no shared lion experiences nor does a lion have human experiences. So would we be able to have a human to lion communication even if we could both speak human speec…

Hmm, I'm finding the premise a bit confusing, "understand what it truly meant to be a lion". I think that's quite different than having meaningful communication. One could make the same argument for "truly understanding" what it means to be someone else. My thinking is that if something is capable of human-style speech, then we'd be able to communicate with them. We'd be able to talk about our shared experiences of t…

> I think the fact that human speech is capable of abstract concepts, not just concrete concepts, means that shared experience isn't necessary to have meaningful communication?

I don't follow that line of reasoning. To me, in that example, you're still communicating with a human, who regardless of culture, or geographic location, still shares an immense amount of shared life experiences with you.

Or, they're not. For example, an intentionally extreme example, I bet we'd have a super hard time talking about homotopy type theory with a member of the amazon rain forest. Similarly, I'd bet they had their own abstract concepts that they would not be able to easily explain to us.

Re: All AI models might be the same

#42

Post by IAmNotACellist »

I agree LLMs are converging on a current representation of reality based on the collective works of humanity. What we need to do is provide AIs with realtime sensory input, simulated hormones each with their own half-lifes based on metabolic conditions and energy usage, a constant thinking loop, and discover a synthetic psilocybin that's capable of causing creative, cross-neural connections similar to human brains. We have the stoned ape theory, we need the stoned AI theory.

Re: All AI models might be the same

#43

Post by cdrini »

Isamu wrote: post #34
klank wrote: post #29

Earlier quoted context omitted.

I was referencing Wittgenstein's "If a lion could speak, we would not understand it." Wittgenstein believed (and I am strongly inclined to agree with him) that our ability to convey meaning through communication was intrinsically tied to (or, rather, sprang forth from) our physical, lived experiences. Thus, to your point, assuming communication, because "there's nothing really special about speech", does that mean we…

If we had a sufficiently large corpus of lion-speech we could build an LLM (Lion Language Model) that would “understand” as well as any model could. Which isn’t saying much, it still couldn’t explain Lion Language to us, it could just generate statistically plausible examples or recognize examples. To translate Lion speech you’d need to train a transformer on a parallel corpus of Lion to English, the existence of whi…

Hmm I don't think we'd need a rosetta stone. In the same way LLMs associate via purely contextual usage the meaning of words, two separate data sets of lion and English, encoded into the same vector space, might pick up patterns of contextual usage at a high enough level to allow for mapping between the two languages.

For example, given thousands of English sentences with the word "sun", the vector embedding encodes the meaning. Assuming the lion word for "sun" is used in much the same context (near lion words for "hot", "heat", etc), it would likely end up in a similar spot near the English word for sun. And because of our shared context living in earth/being animals, I reckon many words likely will be used in similar contexts.

That's my guess though, note I don't know a ton about the internals of LLMs.

Re: All AI models might be the same

#44

Post by samsartor »

yorwba wrote: post #7
TheSaifurRahman wrote: post #5

Has there been research on using this to make models smaller? If models converge on similar representations, we should be able to build more efficient architectures around those core features.

It's more likely that such an architecture would be bigger rather than smaller. https://arxiv.org/abs/2412.20292 demonstrated that score-matching diffusion models approximate a process that combines patches from different training images. To build a model that makes use of this fact, all you need to do is look up the right patch in the training data. Of course a model the size of its training data would typically be…

I have mixed feelings about this interpretation: that diffusion models approximately produce moseics from patches of training data. It does a good job helping people understand why diffusion models are able to work. I used it myself in talk almost 3 years ago! And it isn't a lie exactly, the linked paper is totally sound. It's just that it only works if you assume your model is an absolute optimal minimization of the loss (under some inductive biases). It isn't. No machine learning more complicated than OLS holds up to that standard.

_And that's the actual reason they work._ Undefit models don't just approximate, they interpolate, extrapolate, generalize a bit, and ideally smooth out the occasional total garbage mixed in with your data. In fact, diffusion models work so well because they can correct their own garbage! If extra fingers start to show up in step 5, then steps 6 and 7 still have a chance to reinterpret that as noise and correct back into distribution.

And then there's all the stuff you can do with diffusion models. In my research I hack into the model and use it to decompose images into the surface material properties and lighting! That doesn't make much sense as averaging of memorized patches.

Given all that, it is a very useful interpretation. But I wouldn't take it too literally.

Re: All AI models might be the same

#45

Post by buffet_overflow »

IAmNotACellist wrote: post #42

I agree LLMs are converging on a current representation of reality based on the collective works of humanity. What we need to do is provide AIs with realtime sensory input, simulated hormones each with their own half-lifes based on metabolic conditions and energy usage, a constant thinking loop, and discover a synthetic psilocybin that's capable of causing creative, cross-neural connections similar to human brains. W…

Or perhaps we make them attractions at a theme park, but let Anthony Hopkins have admin access to the source code. What could go wrong?

Re: All AI models might be the same

#46

Post by kindkang2024 »

coffeecoders wrote: post #16

I think “we might decode whale speech or ancient languages” is a huge stretch. Context is the most important part of what makes language useful. There is billions of human-written texts, grounded in shared experience that makes our AI good at language. We don't have that for a whale.

If we could help gorillas or elephants (both highly intelligent) learn to name things and use symbols — in a form they can comprehend and create to express their will — enabling them to pass down their experiences and wisdom across generations, I believe they could quietly be as smart as we are.

Ps. I am excited about Google’s Gemma dolphin project (https://blog.google/technology/ai/dolphingemma/), but I would prefer if they chose elephants instead of dolphins as the subject, since we live on land, not in water. This way, more emphasis could be placed on core research, and immediate communication feedback would be possible.

Re: All AI models might be the same

#47

Post by zos_kia »

cdrini wrote: post #43
Isamu wrote: post #34

Earlier quoted context omitted.

If we had a sufficiently large corpus of lion-speech we could build an LLM (Lion Language Model) that would “understand” as well as any model could. Which isn’t saying much, it still couldn’t explain Lion Language to us, it could just generate statistically plausible examples or recognize examples. To translate Lion speech you’d need to train a transformer on a parallel corpus of Lion to English, the existence of whi…

Hmm I don't think we'd need a rosetta stone. In the same way LLMs associate via purely contextual usage the meaning of words, two separate data sets of lion and English, encoded into the same vector space, might pick up patterns of contextual usage at a high enough level to allow for mapping between the two languages. For example, given thousands of English sentences with the word "sun", the vector embedding encodes…

Someone more knowledgeable might chime in, but I don't think two corpuses can be mapped to the same vector space. Wouldn't each vector space be derived from its corpus?

Re: All AI models might be the same

#48

Post by cdrini »

klank wrote: post #41
cdrini wrote: post #38

Earlier quoted context omitted.

Hmm, I'm finding the premise a bit confusing, "understand what it truly meant to be a lion". I think that's quite different than having meaningful communication. One could make the same argument for "truly understanding" what it means to be someone else. My thinking is that if something is capable of human-style speech, then we'd be able to communicate with them. We'd be able to talk about our shared experiences of t…

> I think the fact that human speech is capable of abstract concepts, not just concrete concepts, means that shared experience isn't necessary to have meaningful communication? I don't follow that line of reasoning. To me, in that example, you're still communicating with a human, who regardless of culture, or geographic location, still shares an immense amount of shared life experiences with you. Or, they're not. For…

I would say there's a difference between abstract and complex. A complex topic would take a lot of time to communicate mainly because you have to go through all the prerequisites. By abstract I mean something like "communicate" or "loss" or "zero"! The primitives of complex thought.

And if we're saying the lion can speak human, then I think it follows that they're capable of this abstract thought, which is what I think is making the premise confusing for me. Maybe if I change my thinking and let's just say the lion is speaking... But if they're speaking a "language" that's capable of communicating concrete and abstract concepts, then that's a human-style language! And because we share many concrete concepts in our shared life experience, I think we would be able to communicate concrete concepts, and then use those as proxies to communicate abstract concepts and hence all concepts?

Re: All AI models might be the same

#49

Post by lsy »

The example given for inverting an embedding back to text doesn't help the idea that this effect is reflecting some "shared statistical model of reality": What would be the plausible whalesong mapping of "Mage (foaled April 18, 2020) is an American Thoroughbred racehorse who won the 2023 Kentucky Derby"?

There isn't anything core to reality about Kentucky, its Derby, the Gregorian calendar, America, horse breeds, etc. These are all cultural inventions that happen to have particular importance in global human culture because of accidents of history, and are well-attested in training sets. At best we are seeing some statistical convergence on training sets because everyone is training on the same pile and scraping the barrel for any differences.

Re: All AI models might be the same

#50

Post by godelski »

klank wrote: post #23
coffeecoders wrote: post #16

I think “we might decode whale speech or ancient languages” is a huge stretch. Context is the most important part of what makes language useful. There is billions of human-written texts, grounded in shared experience that makes our AI good at language. We don't have that for a whale.

If a lion could speak, would we understand it?

I don't know about a Lion, but I think Wittgenstein could have benefited from having a pet.

I train my cat and while I can't always understand her I think one of the most impressive features of the human mind is to be able to have such great understanding of others. We have theory of mind, joint attention, triadic awareness, and much more. My cat can understand me a bit but it's definitely asymmetric.

It's definitely not easy to understand other animals. As Wittgenstein suggests, their minds are alien to us. But we seem to be able to adapt. I'm much better at understanding my cat than my girlfriend (all the local street cats love me, and I teach many of them tricks) but I'm also nothing compared to experts I've seen.

Honestly, I think everyone studying AI could benefit by spending some more time studying animal cognition. While not like computer minds these are testable "alien minds" and can help us better understand the general nature of intelligence

Post reply on HN