Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

All AI models might be the same

blog.jxmo.io

Post reply on HN
Print view View original

91–100 of 157 posts

Re: All AI models might be the same

#91

Post by edwardbernays »

benlivengood wrote: post #58
lsy wrote: post #49

The example given for inverting an embedding back to text doesn't help the idea that this effect is reflecting some "shared statistical model of reality": What would be the plausible whalesong mapping of "Mage (foaled April 18, 2020) is an American Thoroughbred racehorse who won the 2023 Kentucky Derby"? There isn't anything core to reality about Kentucky, its Derby, the Gregorian calendar, America, horse breeds, etc…

You also can't translate "Mage (foaled April 18, 2020) is an American Thoroughbred racehorse who won the 2023 Kentucky Derby" into Hellenistic Greek or some modern indigenous languages because there isn't enough shared context; you'd need to give humans speaking those languages a glossary for any of the translation to make sense, or allow them to interrogate an LLM to act as the glossary. I'd say our current largest…

Why QCD? Quantum chromodynamics, the quantized theory of the nuclear strong force? There is also QED, quantum electrodynamics, which is the quantized field theory for electrodynamics, and then also QFD (quantum flavordynamics) for the weak force. Do you seriously mean to imply that the quantum field theory corresponding to ONLY the strong force, plus gravity, explains every emergent phenomena from there to culture? Fully half of the fundamental forces we account for, in two disparate theoretical frameworks?

Re: All AI models might be the same

#92

Post by godelski »

klank wrote: post #60
godelski wrote: post #50

Earlier quoted context omitted.

I don't know about a Lion, but I think Wittgenstein could have benefited from having a pet. I train my cat and while I can't always understand her I think one of the most impressive features of the human mind is to be able to have such great understanding of others. We have theory of mind, joint attention, triadic awareness, and much more. My cat can understand me a bit but it's definitely asymmetric. It's definitely…

I think the response is generally you are communicating with your cat as an animal, as a mammal. Yes, communication is possible because we too are mammals, animals, etc. But Lion is not just animal, it is not just mammal, it is something more. Something which I have no idea how we would communicate with.

  > But Lion is not just animal, it is not just mammal, it is something more.
Are you saying "lion" is a stand-in for "an arbitrary creature"? If so, yes, that is how I understand Wittgenstein and it doesn't change my comment.

Re: All AI models might be the same

#93

Post by waldrews »

benlivengood wrote: post #58
lsy wrote: post #49

The example given for inverting an embedding back to text doesn't help the idea that this effect is reflecting some "shared statistical model of reality": What would be the plausible whalesong mapping of "Mage (foaled April 18, 2020) is an American Thoroughbred racehorse who won the 2023 Kentucky Derby"? There isn't anything core to reality about Kentucky, its Derby, the Gregorian calendar, America, horse breeds, etc…

You also can't translate "Mage (foaled April 18, 2020) is an American Thoroughbred racehorse who won the 2023 Kentucky Derby" into Hellenistic Greek or some modern indigenous languages because there isn't enough shared context; you'd need to give humans speaking those languages a glossary for any of the translation to make sense, or allow them to interrogate an LLM to act as the glossary. I'd say our current largest…

Your point holds, but the example of Hellenistic Greek seems ill-chosen to make that case - they had horse races and calendars, and mythological mage equivalents that would be reasonable to name a horse after, so the only thing left to map is 'American' as a geographic proper name and 'an important race in America' - which is about as translatable as it gets. Maybe if we pick one of the many cultures that never had horses, their translation would have to throw in so much context that the corresponding text is structurally different?

Re: All AI models might be the same

#94

Post by stevage »

I tried playing Mussolini or Bread with ChatGPT, but it didn't go very well. It seemed to have trouble grasping the rules, and kept getting overly specific when we were miles from the right concept.

Re: All AI models might be the same

#95

Post by kindkang2024 »

ordu wrote: post #85
OkayPhysicist wrote: post #61

Earlier quoted context omitted.

I think it would be extremely surprising for Universal Grammar to be proven false. We have not had any shortage of opportunity to teach animals structured language, and yet we have nothing to show for it. It seems pretty likely that a key factor in our fitness as a species was having the right hardware for grammar.

> I think it would be extremely surprising for Universal Grammar to be proven false. All you need to make it unsurprising you need to find an alternative hypothesis, that can explain observed facts. How about this one. Humans got the ability to abstract deeper than others. While some animals could deal with concepts closely associated with real phenomena, and therefore they can use words to name things or maybe even…

> While some animals could deal with concepts closely associated with real phenomena, and therefore they can use words to name things or maybe even actions, they cannot go further and use abstractions over abstractions. As the result they cannot deal with a recursive grammar.

I prefer to think that animals simply haven’t been lucky enough to invent a suitable naming system that could serve as the foundation of their civilization. If they were fortunate, they might bootstrap their own form of civilization through recursive or iterative divisive naming-—naming the act of naming itself. Naming is the foundation, and everything else naturally follows from there.

As Laozi said over 2,500 years ago in the Dao De Jing:

  “The Tao that can be spoken is not the eternal Tao.
   The name that can be named is not the eternal name.
   The nameless is the origin of Heaven and Earth;
   The named is the mother of all things.”
(Just days ago, AnthropicAI even mentioned Laozi in a tweet: https://x.com/AnthropicAI/status/1925926102725202163 )

P.S. By the way, the entire Dao De Jing advocates thinking beyond symbols, transcending naming—not being constrained by it—and thus connect the reality and following the Tao, which embodies the will of love, frugality, and humility. Yet, few truly understand it fairly, as Laozi emphasized in his book. Perhaps we humans can transcend naming altogether eventually, with advancements like Musk’s brain-communication chip, we might soon discover how human wills are encoded and potentially move beyond the limitations of naming.

Re: All AI models might be the same

#96

Post by klank »

godelski wrote: post #92
klank wrote: post #60

Earlier quoted context omitted.

I think the response is generally you are communicating with your cat as an animal, as a mammal. Yes, communication is possible because we too are mammals, animals, etc. But Lion is not just animal, it is not just mammal, it is something more. Something which I have no idea how we would communicate with.

> But Lion is not just animal, it is not just mammal, it is something more. Are you saying "lion" is a stand-in for "an arbitrary creature"? If so, yes, that is how I understand Wittgenstein and it doesn't change my comment.

No. I'm saying the areas you point out that you feel you could communicate, are ones in which you share a lived experience with the lion. You are both animals, you are both mammals. You get cold, hungry, thirsty, etc.

But lions, and us, are not just animals + mammals. Being a lion or a human means more. Ultimately, there is a uniquely human or lion element. Wittgenstein is saying we cannot communicate this.

Re: All AI models might be the same

#97

Post by klank »

nitwit005 wrote: post #73
klank wrote: post #29

Earlier quoted context omitted.

I was referencing Wittgenstein's "If a lion could speak, we would not understand it." Wittgenstein believed (and I am strongly inclined to agree with him) that our ability to convey meaning through communication was intrinsically tied to (or, rather, sprang forth from) our physical, lived experiences. Thus, to your point, assuming communication, because "there's nothing really special about speech", does that mean we…

Observation has proven enough to understand the meaning of animal calls. People proved they correctly identified, for example, an distressed animal call for assistance, by playing it to their peers in the wild. They go look for the distressed animal. Other calls don't provoke the same reaction.

Analogies are always possible. I believe in the philosophical context though, understanding the meaning of something is not possible through analogy alone.

Reminds me of the quote:

“But people have an unfortunate habit of assuming they understand the reality just because they understood the analogy. You dumb down brain surgery enough for a preschooler to think he understands it, the little tyke’s liable to grab a microwave scalpel and start cutting when no one’s looking.”

― Peter Watts, Echopraxia

Re: All AI models might be the same

#98

Post by yorwba »

samsartor wrote: post #44
yorwba wrote: post #7

Earlier quoted context omitted.

It's more likely that such an architecture would be bigger rather than smaller. https://arxiv.org/abs/2412.20292 demonstrated that score-matching diffusion models approximate a process that combines patches from different training images. To build a model that makes use of this fact, all you need to do is look up the right patch in the training data. Of course a model the size of its training data would typically be…

I have mixed feelings about this interpretation: that diffusion models approximately produce moseics from patches of training data. It does a good job helping people understand why diffusion models are able to work. I used it myself in talk almost 3 years ago! And it isn't a lie exactly, the linked paper is totally sound. It's just that it only works if you assume your model is an absolute optimal minimization of the…

> I used it myself in talk almost 3 years ago! And it isn't a lie exactly, the linked paper is totally sound.

The paper was published in December last year and addresses your concerns head-on. For example, from the introduction:

"if the network can learn this ideal score function exactly, then they will implement a perfect reversal of the forward process. This, in turn, will only be able to turn Gaussian noise into memorized training examples. Thus, any originality in the outputs of diffusion models must lie in their failure to achieve the very objective they are trained on: learning the ideal score function. But how can they fail in intelligent ways that lead to many sensible new examples far from the training set?"

Their answers to these questions are very good and also cover things like correcting the output of previous steps. But the proof is in the pudding: the outputs of their alternative procedure match the models they're explaining very well.

I encourage you to read it; maybe you'll even find a new way to decompose images into surface material properties and lighting as a result.

Re: All AI models might be the same

#99

Post by spiritplumber »

OkayPhysicist wrote: post #61
kindkang2024 wrote: post #46

Earlier quoted context omitted.

If we could help gorillas or elephants (both highly intelligent) learn to name things and use symbols — in a form they can comprehend and create to express their will — enabling them to pass down their experiences and wisdom across generations, I believe they could quietly be as smart as we are. Ps. I am excited about Google’s Gemma dolphin project ( https://blog.google/technology/ai/dolphingemma/ ), but I would pref…

I think it would be extremely surprising for Universal Grammar to be proven false. We have not had any shortage of opportunity to teach animals structured language, and yet we have nothing to show for it. It seems pretty likely that a key factor in our fitness as a species was having the right hardware for grammar.

Darmok and Jalad at Tenegra.

Re: All AI models might be the same

#100

Post by sigmoid10 »

zer00eyz wrote: post #86
jychang wrote: post #77

Earlier quoted context omitted.

I fail to see how that matters. You're implying that all reality is cultural, but that seems irrelevant. The same thing would apply to scientific facts, but whales not having a word for science doesn't make it not real. If we somehow discover LLMs right after Newton discovered the theory of gravity, and then a while later Einstein discovers General Relativity, then GR would not be in the training set of the neural ne…

> You're implying that all reality is cultural, Let's look at two examples of cultural reality: Fan death in South Korea. Where people believe that a fan running while you sleep can kill you. The book "Pure, White and Deadly". Where we discredited the author and his findings and spent decades blaming fat, while packing on the pounds with high fructose corn syrup. An LLM isn't going to find some intrinsic truth, that…

You are describing the state of LLMs from 2 years ago. Which basically means they were just pre-trained on the internet and then fine tuned to follow a particular instruction format. Current models still use this as a first step, but are then trained a lot using reinforcement learning, which has given them much better skills at reasoning and logic than human tainted data ever could. See how Grok 4 for example still eagerly dismisses all those right wing hoaxes, despite being massively tuned to favour right wingers by its creators carefully selecting pre-training data.
Post reply on HN