Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

All AI models might be the same

blog.jxmo.io

Post reply on HN
Print view View original

101–110 of 157 posts

Re: All AI models might be the same

#101

Post by [deleted] »

jychang wrote: post #77
lsy wrote: post #49

The example given for inverting an embedding back to text doesn't help the idea that this effect is reflecting some "shared statistical model of reality": What would be the plausible whalesong mapping of "Mage (foaled April 18, 2020) is an American Thoroughbred racehorse who won the 2023 Kentucky Derby"? There isn't anything core to reality about Kentucky, its Derby, the Gregorian calendar, America, horse breeds, etc…

I fail to see how that matters. You're implying that all reality is cultural, but that seems irrelevant. The same thing would apply to scientific facts, but whales not having a word for science doesn't make it not real. If we somehow discover LLMs right after Newton discovered the theory of gravity, and then a while later Einstein discovers General Relativity, then GR would not be in the training set of the neural ne…

[deleted]

Re: All AI models might be the same

#105

Post by johnisgood »

klank wrote: post #29
eddythompson80 wrote: post #26

Earlier quoted context omitted.

There is nothing really special about speech as a form of communication. All animals communicate with each other and with other animals. Informational density and, uhhhhh, cyclomatic complexity might be different between speech and a dance or a grunt or whatever.

I was referencing Wittgenstein's "If a lion could speak, we would not understand it." Wittgenstein believed (and I am strongly inclined to agree with him) that our ability to convey meaning through communication was intrinsically tied to (or, rather, sprang forth from) our physical, lived experiences. Thus, to your point, assuming communication, because "there's nothing really special about speech", does that mean we…

You might find https://www.lojban.org/files/why-lojban/whylojb.txt interesting. It is not really about your quote of Wittgenstein, but there is:

> In broad terms, the Hypothesis claims that the limits of the language one speaks are the limits of the world one inhabits (also in Wittgenstein), that the grammatical categories of that language define the ontological categories of the word, and that combinatory potentials of that language delimit the complexity of that world (this may be Jim Brown's addition to the complex Hypothesis.) The test then is to see what changes happen in these areas when a person learns a language with a new structure, are they broadened in ways that correspond to the ways the structure of the new language differs from that of the old?

Re: All AI models might be the same

#106

Post by johnisgood »

UltraSane wrote: post #32
klank wrote: post #23

Earlier quoted context omitted.

If a lion could speak, would we understand it?

We should understand common concepts like hungry, tired, horny, pain, etc.

Not just the concepts themselves, but their behavioral representation of it.

Re: All AI models might be the same

#107

Post by samsartor »

yorwba wrote: post #98
samsartor wrote: post #44

Earlier quoted context omitted.

I have mixed feelings about this interpretation: that diffusion models approximately produce moseics from patches of training data. It does a good job helping people understand why diffusion models are able to work. I used it myself in talk almost 3 years ago! And it isn't a lie exactly, the linked paper is totally sound. It's just that it only works if you assume your model is an absolute optimal minimization of the…

> I used it myself in talk almost 3 years ago! And it isn't a lie exactly, the linked paper is totally sound. The paper was published in December last year and addresses your concerns head-on. For example, from the introduction: "if the network can learn this ideal score function exactly, then they will implement a perfect reversal of the forward process. This, in turn, will only be able to turn Gaussian noise into m…

I did read it, all the way through! It's really good. The part you are quoting is setting up the ELS, which does not memorize entire images due to the inductive biases of a CNN (translation symmetry, limited receptive field). But the equivalence to a patch moseic is still due to the assumption that the loss is perfectly minimized under those restrictions.

And I was impressed by the close fit to real CNNs/ResNets and even to UNets. But what that shows is that the real models are heavily overfit. The datasets they are using for evaluation here are _tiny_.

Edit: oh the talk is here btw, if anyone is curious https://youtu.be/c-eIa8QuB24

Re: All AI models might be the same

#108

Post by lloeki »

tgsovlerkhgsel wrote: post #28

I've noticed that many of the large, separately developed AIs often answer with remarkably similar wording to the same question.

I'm not exactly surprised that "separately developed" AI that hinge on essentially the same core techniques and are fed largely identical corpus often answer with remarkably similar wording to the same question.

Re: All AI models might be the same

#109

Post by tempfile »

I know it's only one tiny part, but I can't resist:

> Is it closer to Mussolini or David Beckham? Uhh, I guess Mussolini. (Ok, they’re definitely thinking of a person.)

This deduction is absurd! The only information you have at all is that it's more like a person than it is like bread (which could be almost anything).

Re: All AI models might be the same

#110

Post by vintagedave »

dr_dshiv wrote: post #14

What about the platonic bits? Any other articles that give more details there?

It refers to Plato's Theory of Forms: https://en.wikipedia.org/wiki/Theory_of_forms

Consider how every chair you've seen is different. Yet they are all chairs. What is 'chair-ness'? The Forms theory is that there is a single ideal chair, and this truly exists - not materially, but non-materially as a concept or way of understanding chairs, but real. That is, the 'essence' of a chair exists. All chairs are imperfect representations of the essence of a chair, being perfect chairs or representing the essence of a chair to a greater or lesser degree.

You might find Neal Stephenson's bool Anathem a wonderful read. It's one of my favorites. I want to tell you why without spoilers but I can't (it won't be what you expect) -- but if you read it perhaps you'll find it interesting re this topic.

Post reply on HN