Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

All AI models might be the same

blog.jxmo.io

Post reply on HN
Print view View original

81–90 of 157 posts

Re: All AI models might be the same

#81

Post by never_inline »

streptomycin wrote: post #18

Is it closer to Mussolini or bread? Mussolini. Is it closer to Mussolini or David Beckham? Uhh, I guess Mussolini. (Ok, they’re definitely thinking of a person.) That reasoning doesn't follow. Many things besides people would have the same answers, for instance any animal that seems more like Mussolini than Beckham.

My thought exactly. My word would be "Gun" or "Artillery". easily defeats his logic.

But that should also explain us why pure embeddings search is not sufficient for RAG.

Re: All AI models might be the same

#82

Post by GuB-42 »

godelski wrote: post #50
klank wrote: post #23

Earlier quoted context omitted.

If a lion could speak, would we understand it?

I don't know about a Lion, but I think Wittgenstein could have benefited from having a pet. I train my cat and while I can't always understand her I think one of the most impressive features of the human mind is to be able to have such great understanding of others. We have theory of mind, joint attention, triadic awareness, and much more. My cat can understand me a bit but it's definitely asymmetric. It's definitely…

Cats are domestic animals, and dogs are even more.

You probably didn't adapt to understanding cats as much as cats have adapted over millennia to be understood by humans. Working with and being understood by the dominant specie that is humans is a big evolutionary advantage.

Understanding a wild animal like a lion is a different story. There is a reason why most specialists will say that keeping wild animals as pets is a bad idea, they tend to be unpredictable, which, in other words, mean we don't understand them.

Re: All AI models might be the same

#85

Post by ordu »

OkayPhysicist wrote: post #61
kindkang2024 wrote: post #46

Earlier quoted context omitted.

If we could help gorillas or elephants (both highly intelligent) learn to name things and use symbols — in a form they can comprehend and create to express their will — enabling them to pass down their experiences and wisdom across generations, I believe they could quietly be as smart as we are. Ps. I am excited about Google’s Gemma dolphin project ( https://blog.google/technology/ai/dolphingemma/ ), but I would pref…

I think it would be extremely surprising for Universal Grammar to be proven false. We have not had any shortage of opportunity to teach animals structured language, and yet we have nothing to show for it. It seems pretty likely that a key factor in our fitness as a species was having the right hardware for grammar.

> I think it would be extremely surprising for Universal Grammar to be proven false.

All you need to make it unsurprising you need to find an alternative hypothesis, that can explain observed facts.

How about this one. Humans got the ability to abstract deeper than others. While some animals could deal with concepts closely associated with real phenomena, and therefore they can use words to name things or maybe even actions, they cannot go further and use abstractions over abstractions. As the result they cannot deal with a recursive grammar.

Or another hypothesis. Humans got an ability to reflect on themselves, so they came to concepts like "motive (of action)", it made it possible for them to talk about actions, inventing words-categories, that categorize actions by supposed motive of action. But it is not just that, people started to feel that simple words do not describe the reality good enough, because a) actions have more than one motive, b) they started to see actions everywhere (the stone is lying flat? it is the action of the stone, that strictly speaking cannot act.). All this led them to invent complex grammars to describe the world where they live. Animals from other hand can't get it, because their world is much simplier. They don't see the stone "lying": it is a stone, it cannot do anything.

These hypotheses don't seem explaining everything, but I have just invented them. I didn't really tried to explain everything. But thinking about it, I'm in favor of the second one: all people are schizophrenics if you compare them with animals. They believe in things that are not exist, they see inanimate objects as animate, they are talking with themselves (inner dialog). It just happens that the particular kind of schizophrenia most people are struggling with is a condition that allows them to not tear apart all the connections with the Reality. It is more like borderline schizophrenia.

Re: All AI models might be the same

#86

Post by zer00eyz »

jychang wrote: post #77
lsy wrote: post #49

The example given for inverting an embedding back to text doesn't help the idea that this effect is reflecting some "shared statistical model of reality": What would be the plausible whalesong mapping of "Mage (foaled April 18, 2020) is an American Thoroughbred racehorse who won the 2023 Kentucky Derby"? There isn't anything core to reality about Kentucky, its Derby, the Gregorian calendar, America, horse breeds, etc…

I fail to see how that matters. You're implying that all reality is cultural, but that seems irrelevant. The same thing would apply to scientific facts, but whales not having a word for science doesn't make it not real. If we somehow discover LLMs right after Newton discovered the theory of gravity, and then a while later Einstein discovers General Relativity, then GR would not be in the training set of the neural ne…

> You're implying that all reality is cultural,

Let's look at two examples of cultural reality:

Fan death in South Korea. Where people believe that a fan running while you sleep can kill you.

The book "Pure, White and Deadly". Where we discredited the author and his findings and spent decades blaming fat, while packing on the pounds with high fructose corn syrup.

An LLM isn't going to find some intrinsic truth, that we are ignoring, in its data set. An LLM isn't going to find issues in the reproducibility / replication crisis. I have not seen one discredit a scientific paper with its own findings.

To be clear LLM's can be amazing tools, but garbage in garbage out still applies.

Re: All AI models might be the same

#87

Post by godelski »

cdrini wrote: post #56
godelski wrote: post #52

Earlier quoted context omitted.

It depends how you define the vector space but I'm inclined to agree. The reason I think this is from evidence in human language. Spend time with any translator and they'll tell you that some things just don't really translate. The main concepts might, but there's subtleties and nuances that really change the feel. You probably notice this with friends who have a different native language than you. Even same language…

Hmm I don't think a universal language is implied by being able to translate without a rosetta stone. I agree, I don't think there is such a thing as a universal language, per se, but I do wonder if there is a notion of a universal language at a certain level of abstraction. But I think those ambiguous cases can still be understood/defined. You can describe how this one word in lion doesn't neatly map to a single wor…

  > I don't think a universal language is implied by being able to translate without a rosetta stone.
Depends what you mean. If you want a 1-to-1 translation then your languages need to be isomorphic. For lossy translation you still need some intersection within the embedding space. The intersection will determine how good you can translate. It isn't unreasonable to assume that there are some universal traits here as any being lives in this universe and we're all subject to these experiences at some level, right? But that could result in some very lossy translations that are effectively impossible to translate, right?

Another way you can think about it, though, is that language might not be dependent on experience. If it is completely divorced, we may be able to understand anyone regardless of experience. If it is mixed, then results can be mixed.

  > The example I was thinking of was word2vec
Be careful with this. If you haven't actually gone deep into the math (more than 3Blue1Brown) you'll find some serious limitations to this. Play around with it and you'll experience these too. Distances in high dimensions are not well defined. There also aren't smooth embeddings here. You have a lot of similar problems to embedding methods like t-SNE. Certainly has uses but it is far too easy to draw the wrong conclusions from them. Unfortunately, both of these are often spoken about incorrectly (think as incorrect as most peoples understandings of things like Schrodinger's Cat or the Double Slit experiment, or really most of QM. There's some elements of truth but it's communicated through a game of telephone).

Re: All AI models might be the same

#88

Post by godelski »

GuB-42 wrote: post #82
godelski wrote: post #50

Earlier quoted context omitted.

I don't know about a Lion, but I think Wittgenstein could have benefited from having a pet. I train my cat and while I can't always understand her I think one of the most impressive features of the human mind is to be able to have such great understanding of others. We have theory of mind, joint attention, triadic awareness, and much more. My cat can understand me a bit but it's definitely asymmetric. It's definitely…

Cats are domestic animals, and dogs are even more. You probably didn't adapt to understanding cats as much as cats have adapted over millennia to be understood by humans. Working with and being understood by the dominant specie that is humans is a big evolutionary advantage. Understanding a wild animal like a lion is a different story. There is a reason why most specialists will say that keeping wild animals as pets…

I agree with the logic, but I don't think it doesn't properly accounts for people who work very closely with wild animals.

You or I, yeah, probably not going to understand a lion pretty well. But someone who works at the zoo? A lion tamer? Someone studying lion cognition? Hell, people have figured out how to train hippos so that they can clean their teeth[0], and these are one of, if not the, most aggressive animals in the world. Humans have gotten impressively good at communicating with many different animals and training them. There's plenty of Steve Irwin types who have strong understandings about many creatures who would be quite alien to the rest of us. Which, that requires at least one side to have a strong understanding of the other's desires and how they perceive the world. But me? I have no doubt that hippo would murder me.

My point isn't so much about would we understand the lion, but rather could we. Wittgenstein implied we wouldn't be able to. I'm pointing to evidence that we, to at least some degree, can. How much we ultimately will be able to, is still unknown. But I certainly don't think it is an impossible task.

[0] https://www.youtube.com/watch?v=dMoWvh2AtKk

Re: All AI models might be the same

#89

Post by edwardbernays »

photonthug wrote: post #79
benlivengood wrote: post #58

Earlier quoted context omitted.

You also can't translate "Mage (foaled April 18, 2020) is an American Thoroughbred racehorse who won the 2023 Kentucky Derby" into Hellenistic Greek or some modern indigenous languages because there isn't enough shared context; you'd need to give humans speaking those languages a glossary for any of the translation to make sense, or allow them to interrogate an LLM to act as the glossary. I'd say our current largest…

> You also can't translate .. into Hellenistic Greek or some modern indigenous languages because there isn't enough shared context; you'd need to give humans speaking those languages a glossary for any of the translation to make sense What? By substitution, this means you can translate it. As long as we're assuming a large enough basis of concept vectors of course it works. > I'd say our current largest LLMs probably…

I completely believe that physics to culture is intractable given our current corpus, to the degree it's probably a nonsense claim. There are so many emergent phenomena that introduce confounding effects at every level of abstraction.

Also, why QCD? Quantum chromodynamics, the quantized theory of the nuclear strong force? There is also QED, quantum electrodynamics, which is the quantized field theory for electrodynamics, and then also QFD (quantum flavordynamics) for the weak force. Does OP seriously mean to imply that the quantum field theory corresponding to ONLY the strong force, plus gravity, explains every emergent phenomena from there to culture? Fully half of the fundamental forces we account for, in two disparate theoretical frameworks?

OP's comment is not serious speculation.

Re: All AI models might be the same

#90

Post by ars »

kindkang2024 wrote: post #46
coffeecoders wrote: post #16

I think “we might decode whale speech or ancient languages” is a huge stretch. Context is the most important part of what makes language useful. There is billions of human-written texts, grounded in shared experience that makes our AI good at language. We don't have that for a whale.

If we could help gorillas or elephants (both highly intelligent) learn to name things and use symbols — in a form they can comprehend and create to express their will — enabling them to pass down their experiences and wisdom across generations, I believe they could quietly be as smart as we are. Ps. I am excited about Google’s Gemma dolphin project ( https://blog.google/technology/ai/dolphingemma/ ), but I would pref…

They tried that with Koko, it didn't work. It did not make the gorilla any smarter, it was an animal, and remained an animal.

If you read the criticism about the Koko project, it's that Patterson prompted her to make certain signs. I watched some clips of the signs, and it's very obvious that that is exactly what she did.

Animals can only communicate in a limited fashion. They do not have some "hidden" intelligence that you are trying to find.

Post reply on HN