Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

All AI models might be the same

blog.jxmo.io

Post reply on HN
Print view View original

71–80 of 157 posts

Re: All AI models might be the same

#71

Post by hatthew »

Quarrelsome wrote: post #59
empath75 wrote: post #9

This is kind of fascinating because I just tried to play mussolini or bread with chatgpt and it is absolutely _awful_ at it, even with reasoning models. It just assumes that your answers are going to be reasonably bread-like or reasonably mussolini-like, and doesn't think laterally at all. It just kept asking me about varieties of baked goods. edit: It did much better after I added some extra explanation -- that it c…

what surprised me when I asked if it knew what Mussolini or Bread was, it parroted this article. This article was posted in the last 24 hours and querying a search engine for "Mussolini or Bread" didn't yield it anywhere near a top result.

searching for '"Mussolini or Bread"' (with quotes), or searching for 'Mussolini or Bread game' (without quotes) both return this article at the top.

Re: All AI models might be the same

#72

Post by b112 »

hatthew wrote: post #69
b112 wrote: post #65

Earlier quoted context omitted.

When an LLM can re-tokenize on the fly, due to newly learned data, let me know. It won't prove intelligence, but at least it won't be static like a book.

https://x.com/sukjun_hwang/status/1943703574908723674 > Tokenization has been the final barrier to truly end-to-end language models. > We developed the H-Net: a hierarchical network that replaces tokenization with a dynamic chunking process directly inside the model, automatically discovering and operating over meaningful units of data

Good to see. Will be better to see once the scope is clear.

But something such as this is required to move towards actual intelligence.

Re: All AI models might be the same

#73

Post by nitwit005 »

klank wrote: post #29
eddythompson80 wrote: post #26

Earlier quoted context omitted.

There is nothing really special about speech as a form of communication. All animals communicate with each other and with other animals. Informational density and, uhhhhh, cyclomatic complexity might be different between speech and a dance or a grunt or whatever.

I was referencing Wittgenstein's "If a lion could speak, we would not understand it." Wittgenstein believed (and I am strongly inclined to agree with him) that our ability to convey meaning through communication was intrinsically tied to (or, rather, sprang forth from) our physical, lived experiences. Thus, to your point, assuming communication, because "there's nothing really special about speech", does that mean we…

Observation has proven enough to understand the meaning of animal calls. People proved they correctly identified, for example, an distressed animal call for assistance, by playing it to their peers in the wild. They go look for the distressed animal. Other calls don't provoke the same reaction.

Re: All AI models might be the same

#74

Post by kindkang2024 »

OkayPhysicist wrote: post #61
kindkang2024 wrote: post #46

Earlier quoted context omitted.

If we could help gorillas or elephants (both highly intelligent) learn to name things and use symbols — in a form they can comprehend and create to express their will — enabling them to pass down their experiences and wisdom across generations, I believe they could quietly be as smart as we are. Ps. I am excited about Google’s Gemma dolphin project ( https://blog.google/technology/ai/dolphingemma/ ), but I would pref…

I think it would be extremely surprising for Universal Grammar to be proven false. We have not had any shortage of opportunity to teach animals structured language, and yet we have nothing to show for it. It seems pretty likely that a key factor in our fitness as a species was having the right hardware for grammar.

“Universal grammar” is a placeholder for the innate, domain-specific features of linguistic ability...

--— After looking up “Universal Grammar,” here are my thoughts: --—

Until we identify these features and prove that other species lack them, the concept of Universal Grammar remains uncertain. Only if we pinpoint what’s unique to humans and fail to help other species develop it can we confidently claim Universal Grammar is correct.

Personally, I think these innate features might not be entirely unique to humans. Even humans need sufficient social interaction and language exposure to truly behave like 'humans'. Feral children(https://en.wikipedia.org/wiki/Feral_child), for example, behave much like animals. I also imagine early humans had only a few words and symbols and acted similarly. However, if feral children or ancient humans were exposed to today’s social environment, they could likely learn our complex language quite quickly, because those words, symbols, and pronunciations are well-suited to humans for expressing our wills.

Perhaps, we humans are just lucky to have found effective ways to express ourselves —- symbols and words as we know them now -— and to pass them down through generations. This has allowed language and cognition to grow exponentially, giving us an edge over other species.

P.S. Our failure so far to teach animals structured language is exactly why I admire projects like Google’s Dolphin Gemma. To succeed, we need to tailor teaching methods to their species-specific traits—starting by understanding how they express their intentions/wills, and then hopefully helping enrich their naming systems and overall communication. We might even adapt this to help them learn from human contexts, much like large language models do.

Re: All AI models might be the same

#75

Post by Quarrelsome »

hatthew wrote: post #71
Quarrelsome wrote: post #59

Earlier quoted context omitted.

what surprised me when I asked if it knew what Mussolini or Bread was, it parroted this article. This article was posted in the last 24 hours and querying a search engine for "Mussolini or Bread" didn't yield it anywhere near a top result.

searching for '"Mussolini or Bread"' (with quotes), or searching for 'Mussolini or Bread game' (without quotes) both return this article at the top.

Idiot me forgot to try the quotes. Thank you.

Re: All AI models might be the same

#77

Post by jychang »

lsy wrote: post #49

The example given for inverting an embedding back to text doesn't help the idea that this effect is reflecting some "shared statistical model of reality": What would be the plausible whalesong mapping of "Mage (foaled April 18, 2020) is an American Thoroughbred racehorse who won the 2023 Kentucky Derby"? There isn't anything core to reality about Kentucky, its Derby, the Gregorian calendar, America, horse breeds, etc…

I fail to see how that matters. You're implying that all reality is cultural, but that seems irrelevant. The same thing would apply to scientific facts, but whales not having a word for science doesn't make it not real.

If we somehow discover LLMs right after Newton discovered the theory of gravity, and then a while later Einstein discovers General Relativity, then GR would not be in the training set of the neural net. That doesn't make GR any less of a description of reality! You also can't convert General Relativity into whalesong!

But you CAN explain General Relativity in English, or in Chinese to a person in china. So the fact that we can create a mapping from the concept of General Relativity in the neural network of the brain of a human in the USA using english, to someone in china using chinese, to a ML model, is what makes it a shared statistical model of reality.

You also can't convert General Relativity to the language of "infant babble", does it make general relativity any less real?

Re: All AI models might be the same

#78

Post by trhway »

coffeecoders wrote: post #16

I think “we might decode whale speech or ancient languages” is a huge stretch. Context is the most important part of what makes language useful. There is billions of human-written texts, grounded in shared experience that makes our AI good at language. We don't have that for a whale.

>There is billions of human-written texts, grounded in shared experience that makes our AI good at language. We don't have that for a whale.

the world around us is a very large part of that shared experience. It is shared among humans, shared among whales, and shared among whales and humans as well.

Re: All AI models might be the same

#79

Post by photonthug »

benlivengood wrote: post #58
lsy wrote: post #49

The example given for inverting an embedding back to text doesn't help the idea that this effect is reflecting some "shared statistical model of reality": What would be the plausible whalesong mapping of "Mage (foaled April 18, 2020) is an American Thoroughbred racehorse who won the 2023 Kentucky Derby"? There isn't anything core to reality about Kentucky, its Derby, the Gregorian calendar, America, horse breeds, etc…

You also can't translate "Mage (foaled April 18, 2020) is an American Thoroughbred racehorse who won the 2023 Kentucky Derby" into Hellenistic Greek or some modern indigenous languages because there isn't enough shared context; you'd need to give humans speaking those languages a glossary for any of the translation to make sense, or allow them to interrogate an LLM to act as the glossary. I'd say our current largest…

> You also can't translate .. into Hellenistic Greek or some modern indigenous languages because there isn't enough shared context; you'd need to give humans speaking those languages a glossary for any of the translation to make sense

What? By substitution, this means you can translate it. As long as we're assuming a large enough basis of concept vectors of course it works.

> I'd say our current largest LLMs probably contain sufficient detail to explain a concept like a named race horse starting from QCD+gravity and ending up at cultural human events

What? I'm curious how you'd propose to move from gravity to culture. This is like TFAs assertion that the M+B game might be as expressive as 20 questions / universal. M or B is just (bad,sentient) or (good,object). Sure, entangling a few concepts is slightly more expressive than playing with a completely flattened world of 1/0 due to some increase in dimensionality. But trying to pinpoint anything like (neutral,concept) fails because the basis set isn't fundamentally large enough. Any explanation of how this could work will have to cheat, like when TFA smuggles new information in by communicating details about distance-from-basis. For example to really get to the word or concept of "neutral" from inferred good/bad dichotomy of bread/mussolini, you would have to answer "Hmmmmmm, closer to bread I guess" in one iteration and then "Umm.. closer to Mussolini I guess" when asked again, and then have the interrogator notice the uncertainty/hesitation/contradiction and then infer neutrality. This is just the simple case.. physics to culture seems much harder

Re: All AI models might be the same

#80

Post by chamomeal »

IAmNotACellist wrote: post #42

I agree LLMs are converging on a current representation of reality based on the collective works of humanity. What we need to do is provide AIs with realtime sensory input, simulated hormones each with their own half-lifes based on metabolic conditions and energy usage, a constant thinking loop, and discover a synthetic psilocybin that's capable of causing creative, cross-neural connections similar to human brains. W…

I’m so incredibly tired of reading about AI but if I saw “we gave an AI shrooms” I’d be all over it
Post reply on HN