Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

All AI models might be the same

blog.jxmo.io

Post reply on HN
Print view View original

141–150 of 157 posts

Re: All AI models might be the same

#141

Post by cwmoore »

benlivengood wrote: post #58
lsy wrote: post #49

The example given for inverting an embedding back to text doesn't help the idea that this effect is reflecting some "shared statistical model of reality": What would be the plausible whalesong mapping of "Mage (foaled April 18, 2020) is an American Thoroughbred racehorse who won the 2023 Kentucky Derby"? There isn't anything core to reality about Kentucky, its Derby, the Gregorian calendar, America, horse breeds, etc…

You also can't translate "Mage (foaled April 18, 2020) is an American Thoroughbred racehorse who won the 2023 Kentucky Derby" into Hellenistic Greek or some modern indigenous languages because there isn't enough shared context; you'd need to give humans speaking those languages a glossary for any of the translation to make sense, or allow them to interrogate an LLM to act as the glossary. I'd say our current largest…

Man"splain a concept like...given a foothold of some common ground to translate into a new unknown language. In a sense, that's what a model of reality is" not. It is a story with motivated characters.

Re: All AI models might be the same

#142

Post by cwmoore »

waldrews wrote: post #93
benlivengood wrote: post #58

Earlier quoted context omitted.

You also can't translate "Mage (foaled April 18, 2020) is an American Thoroughbred racehorse who won the 2023 Kentucky Derby" into Hellenistic Greek or some modern indigenous languages because there isn't enough shared context; you'd need to give humans speaking those languages a glossary for any of the translation to make sense, or allow them to interrogate an LLM to act as the glossary. I'd say our current largest…

Your point holds, but the example of Hellenistic Greek seems ill-chosen to make that case - they had horse races and calendars, and mythological mage equivalents that would be reasonable to name a horse after, so the only thing left to map is 'American' as a geographic proper name and 'an important race in America' - which is about as translatable as it gets. Maybe if we pick one of the many cultures that never had h…

The LLM almost caught a 42 slork babelfish.

Re: All AI models might be the same

#143

Post by RedNifre »

klank wrote: post #29
eddythompson80 wrote: post #26

Earlier quoted context omitted.

There is nothing really special about speech as a form of communication. All animals communicate with each other and with other animals. Informational density and, uhhhhh, cyclomatic complexity might be different between speech and a dance or a grunt or whatever.

I was referencing Wittgenstein's "If a lion could speak, we would not understand it." Wittgenstein believed (and I am strongly inclined to agree with him) that our ability to convey meaning through communication was intrinsically tied to (or, rather, sprang forth from) our physical, lived experiences. Thus, to your point, assuming communication, because "there's nothing really special about speech", does that mean we…

That seems a bit extreme, given that a lion also has a mammal brain. I'd expect it to also think in terms of distinct entities that can move around in the environment and possibly talk about things like "hunger" and "prey".

I'd expect incomprehensible language from something that is wildly different from us, e.g. sentient space crystals that eat radiation.

On the other hand, we still haven't figured out dolphin language (the most interesting guess was that they shout 3D images at each other).

Re: All AI models might be the same

#144

Post by otabdeveloper4 »

OkayPhysicist wrote: post #61
kindkang2024 wrote: post #46

Earlier quoted context omitted.

If we could help gorillas or elephants (both highly intelligent) learn to name things and use symbols — in a form they can comprehend and create to express their will — enabling them to pass down their experiences and wisdom across generations, I believe they could quietly be as smart as we are. Ps. I am excited about Google’s Gemma dolphin project ( https://blog.google/technology/ai/dolphingemma/ ), but I would pref…

I think it would be extremely surprising for Universal Grammar to be proven false. We have not had any shortage of opportunity to teach animals structured language, and yet we have nothing to show for it. It seems pretty likely that a key factor in our fitness as a species was having the right hardware for grammar.

> having the right hardware for grammar

Do we have hardware for playing Super Mario or dancing to pop music? Animals can't do that either.

Re: All AI models might be the same

#145

Post by hatthew »

b112 wrote: post #115
hatthew wrote: post #67

Earlier quoted context omitted.

Do you think an average human from 1850 transported to the modern world would fare better than an LLM?

This isn't a serious question, is it? A human would immediately start learning, and understanding the differences. Forming new memories. People learn every, single day. An LLM would never have a single new memory, nor understand a single change had happened. Without being retokenized via new data it trains on, nothing of its symbolic world view would even change in the tiniest bit. Telling it so in a context window i…

For long term memory, LLMs can learn new information by training on it. It would need a lot of training data, but it will get there eventually.

For short term memory, they can store information in their context window. Yes the context window is finite, but they can still be very large, and there exist techniques to fit more information within that window.

For medium term memory, there's not one single solution, but anecdotally it seems like solutions exist. My workplace has a chatbot that can ingest documents and save slack conversations, and then use the knowledge from those to answer questions indefinitely into the future. I haven't looked into how it works, but empirically it functions well across thousands of people for more than a year.

Between a human and an LLM transported 200 years into the future, a human will certainly have better critical thinking skills, but I'd bet an LLM system designed for the task will probably be able to more quickly learn to have coherent and symmetric conversations about the world with modern humans.

Overall, there's a lot that's unclear, and I think I'm pretty pessimistic about progress towards AGI, but I don't think we can deny that LLMs have something that is, at minimum, the best substitute for human intelligence that exists.

Re: All AI models might be the same

#146

Post by AIPedant »

GuB-42 wrote: post #82
godelski wrote: post #50

Earlier quoted context omitted.

I don't know about a Lion, but I think Wittgenstein could have benefited from having a pet. I train my cat and while I can't always understand her I think one of the most impressive features of the human mind is to be able to have such great understanding of others. We have theory of mind, joint attention, triadic awareness, and much more. My cat can understand me a bit but it's definitely asymmetric. It's definitely…

Cats are domestic animals, and dogs are even more. You probably didn't adapt to understanding cats as much as cats have adapted over millennia to be understood by humans. Working with and being understood by the dominant specie that is humans is a big evolutionary advantage. Understanding a wild animal like a lion is a different story. There is a reason why most specialists will say that keeping wild animals as pets…

FWIW I don't think this is quite true for domestic cats - cross-species collaboration is very common in carnivores. A famous example is the coyote and badger, which will team up to hunt prairie dogs: the badger chases them out of the burrows, the coyote catches them outside, and they split the meat 70-30 in acknowledgement that the coyote is larger and needs more food. Recently the same has been observed with ocelots and possums. It seems like most mammals are able to at least somewhat understand the mood and intentions of other mammals - note that we share facial expressions and body language.

OTOH feral cats are known for being highly social compared to other cats, forming large semi-collaborative colonies. And adult cats have much more difficulty socializing to humans than adult dogs, even if they don't have trauma/etc. I suspect the real story of cat domestication goes both ways: an unusually gregarious subspecies of African wildcat started forming colonies near human settlements and forming cross-carnivore collaborations with the humans who lived there. This was also true for dogs - it likely started with unusually peaceful Siberian wolves - but I believe cats were more "accidental." Humans have been deliberately creating dog breeds since antiquity, but with a tiny number of exceptions cat breeds are modern. I doubt ancient humans ever "bred" cats like they did dogs, it seems closer to natural selection.

Re: All AI models might be the same

#147

Post by throwpoaster »

Rover222 wrote: post #121
throwpoaster wrote: post #119

I asked Grok, o3-pro, and Claude a question about piezoelectric effects. They all got it "right", but Claude called out a second order effect that arose from the use case that the other two missed. I get it, they might all be exploring the same space, but Claude went an extra, important, hop.

I'm just curious if that was Grok 4 or 3

3! So it’s not a fair comparison. The other two were both paid.

I am very curious to run real world engineering problems through SuperHeavy.

Re: All AI models might be the same

#148

Post by godelski »

AIPedant wrote: post #146
GuB-42 wrote: post #82

Earlier quoted context omitted.

Cats are domestic animals, and dogs are even more. You probably didn't adapt to understanding cats as much as cats have adapted over millennia to be understood by humans. Working with and being understood by the dominant specie that is humans is a big evolutionary advantage. Understanding a wild animal like a lion is a different story. There is a reason why most specialists will say that keeping wild animals as pets…

FWIW I don't think this is quite true for domestic cats - cross-species collaboration is very common in carnivores. A famous example is the coyote and badger, which will team up to hunt prairie dogs: the badger chases them out of the burrows, the coyote catches them outside, and they split the meat 70-30 in acknowledgement that the coyote is larger and needs more food. Recently the same has been observed with ocelots…

Fwiw, a lot of cat breeds are very old. Especially the ones coming out of northeast Africa (like Egypt).

But yes, both accidental domestication happens as well as non human cross species collaboration. Another famous example is with the cleaner fish and sharks. Animals also frequently collaborate with plants. Ants even have farms, both fungi and other insects

Re: All AI models might be the same

#149

Post by kindkang2024 »

OkayPhysicist wrote: post #123
kindkang2024 wrote: post #74

Earlier quoted context omitted.

“Universal grammar” is a placeholder for the innate, domain-specific features of linguistic ability... --— After looking up “Universal Grammar,” here are my thoughts: --— Until we identify these features and prove that other species lack them, the concept of Universal Grammar remains uncertain. Only if we pinpoint what’s unique to humans and fail to help other species develop it can we confidently claim Universal Gra…

Other animals have vocabulary. Anybody with a pet can easily understand that lots of animals can associate words with objects and actions, and Koko and Alex and all the other examples of talking animals demonstrate that animals can communicate their own wants with words. The thing that makes humans nearly unique in the animal kingdom is structured language. I don't need separate words for "leopard in the bush", "snak…

> The thing that makes humans nearly unique in the animal kingdom is structured language.

But what is “structured language”? I would argue it’s just another layer of naming—naming applied to already named names.

If animals have the ability to map leopard, snake, bush, tree in reality to names in their naming system, then why can’t they apply that ability again—create a new name like ‘leopard-bush’ or ‘snake-leopard’, simply by combining the existing solid names in their system? (Similarly, words like “in” could also be added.)

You might say they can’t—because they lack grammar rules. And I’d argue the issue runs even deeper: they don’t have a valid naming system to begin with, nor the ecological fitness or incentive needed to develop further naming.

If they can name ‘leopard’ within a valid system, that name should be reusable. They should be able to work on those names—as objects themselves. The process is the same. It requires the same abstraction: treat a thing as a nameable unit, then name it again. And that already starts to look like compositionality—or grammar.

The real difference is that humans ended up with a robust naming system. We got lucky—we gradually invented stable symbols that let us name on top of previous names. We could write them down, save them, pass them on—and then build new names on top of that naming recursively, iteratively, and divisively.

And that’s likely where the fitness gain came from. It’s a lucky, genuine self-bootstrap story.

Re: All AI models might be the same

#150

Post by CamperBob2 »

godelski wrote: post #50
klank wrote: post #23

Earlier quoted context omitted.

If a lion could speak, would we understand it?

I don't know about a Lion, but I think Wittgenstein could have benefited from having a pet. I train my cat and while I can't always understand her I think one of the most impressive features of the human mind is to be able to have such great understanding of others. We have theory of mind, joint attention, triadic awareness, and much more. My cat can understand me a bit but it's definitely asymmetric. It's definitely…

I'm much better at understanding my cat than my girlfriend

Huh. Apparently attention isn't all we need in order to parse that sentence.

Post reply on HN