Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

All AI models might be the same

blog.jxmo.io

Post reply on HN
Print view View original

61–70 of 157 posts

Re: All AI models might be the same

#61

Post by OkayPhysicist »

kindkang2024 wrote: post #46
coffeecoders wrote: post #16

I think “we might decode whale speech or ancient languages” is a huge stretch. Context is the most important part of what makes language useful. There is billions of human-written texts, grounded in shared experience that makes our AI good at language. We don't have that for a whale.

If we could help gorillas or elephants (both highly intelligent) learn to name things and use symbols — in a form they can comprehend and create to express their will — enabling them to pass down their experiences and wisdom across generations, I believe they could quietly be as smart as we are. Ps. I am excited about Google’s Gemma dolphin project ( https://blog.google/technology/ai/dolphingemma/ ), but I would pref…

I think it would be extremely surprising for Universal Grammar to be proven false. We have not had any shortage of opportunity to teach animals structured language, and yet we have nothing to show for it. It seems pretty likely that a key factor in our fitness as a species was having the right hardware for grammar.

Re: All AI models might be the same

#62

Post by kevin42 »

b112 wrote: post #55
stillpointlab wrote: post #51

I have to be careful of confirmation bias when I read stuff like this because I have the intuition that we are uncovering a single intelligence with each of the different LLMs. I even feel, when switching between the big three (OpenAI, Google, Anthropic) that there is a lot of similarity in how they speak and think - but I am aware of my bias so I try not to let it cloud my judgement. On the topic of compression, I a…

LLMs don't think, nor are they intelligent or exhibiting intelligence. Language does have constraints, yet it evolves via its users to encompass new meanings. Thus those constraints are artificial, unless you artificially enforce static language use. And of course, for an LLM to use those new concepts, it needs to be retokenized by being trained on new data. For example, if we trained LLMs only on books, encyclopedia…

You made a lot of bold assertions there. It's as if you have a complete and definitive theory of human intelligence to compare it against. Which if true, would be incredible, because there isn't a scientifically accepted theory, nor is there consensus from a philosophical standpoint.

I can't say that you are wrong, you might be right, especially about AGI. And I think it's unlikely that LLMs are the direct path to AGI. But, just looking at how human brains work, it seems unlikely that we would be intelligent either if we used your same reductionist logic.

An individual neuron doesn't "think" or "understand" anything. It's a biological cell that simply fires an electrochemical signal when its input threshold is met. It has no understanding of language or context. By your logic, since the fundamental components are just simple signal processors, the brain cannot possibly learn or be intelligent. Yet, from the complex interaction of ~86 billion of these simple biochemical machines, the emergent properties of thought, understanding, and consciousness arise.

Dismissing an LLM's capabilities because its underlying operations are basically just math operating on tokenized data is like dismissing human consciousness because it's "just electrochemistry" in a network of cells. Both arguments mistake the low-level mechanism for the high-level emergent phenomenon.

Re: All AI models might be the same

#63

Post by SJC_Hacker »

klank wrote: post #29
eddythompson80 wrote: post #26

Earlier quoted context omitted.

There is nothing really special about speech as a form of communication. All animals communicate with each other and with other animals. Informational density and, uhhhhh, cyclomatic complexity might be different between speech and a dance or a grunt or whatever.

I was referencing Wittgenstein's "If a lion could speak, we would not understand it." Wittgenstein believed (and I am strongly inclined to agree with him) that our ability to convey meaning through communication was intrinsically tied to (or, rather, sprang forth from) our physical, lived experiences. Thus, to your point, assuming communication, because "there's nothing really special about speech", does that mean we…

Correlate the vocalizations with the subsequent behavior. I believe this has been done for some species in certain situations.

Its also pretty much how humans acquire language. No one is born knowing English or Spanish or Mandarin.

Re: All AI models might be the same

#64

Post by megaloblasto »

Really cool idea that I hadn't considered yet. If true, seems like a big plus for open source and not having a few companies controll all the models. If they all converge to the same intelligence, one open source model would make all proprietary models obsolete.

Re: All AI models might be the same

#65

Post by b112 »

kevin42 wrote: post #62
b112 wrote: post #55

Earlier quoted context omitted.

LLMs don't think, nor are they intelligent or exhibiting intelligence. Language does have constraints, yet it evolves via its users to encompass new meanings. Thus those constraints are artificial, unless you artificially enforce static language use. And of course, for an LLM to use those new concepts, it needs to be retokenized by being trained on new data. For example, if we trained LLMs only on books, encyclopedia…

You made a lot of bold assertions there. It's as if you have a complete and definitive theory of human intelligence to compare it against. Which if true, would be incredible, because there isn't a scientifically accepted theory, nor is there consensus from a philosophical standpoint. I can't say that you are wrong, you might be right, especially about AGI. And I think it's unlikely that LLMs are the direct path to AG…

When an LLM can re-tokenize on the fly, due to newly learned data, let me know.

It won't prove intelligence, but at least it won't be static like a book.

Re: All AI models might be the same

#66

Post by kindkang2024 »

b112 wrote: post #55
stillpointlab wrote: post #51

I have to be careful of confirmation bias when I read stuff like this because I have the intuition that we are uncovering a single intelligence with each of the different LLMs. I even feel, when switching between the big three (OpenAI, Google, Anthropic) that there is a lot of similarity in how they speak and think - but I am aware of my bias so I try not to let it cloud my judgement. On the topic of compression, I a…

LLMs don't think, nor are they intelligent or exhibiting intelligence. Language does have constraints, yet it evolves via its users to encompass new meanings. Thus those constraints are artificial, unless you artificially enforce static language use. And of course, for an LLM to use those new concepts, it needs to be retokenized by being trained on new data. For example, if we trained LLMs only on books, encyclopedia…

LLMs don't think, nor are they intelligent or exhibiting intelligence. ....

---

My 2$: If you replace 'LLMs' with 'Humans,' most of your statements above still make sense — which suggests AGI might be possible.

Re: All AI models might be the same

#67

Post by hatthew »

b112 wrote: post #55
stillpointlab wrote: post #51

I have to be careful of confirmation bias when I read stuff like this because I have the intuition that we are uncovering a single intelligence with each of the different LLMs. I even feel, when switching between the big three (OpenAI, Google, Anthropic) that there is a lot of similarity in how they speak and think - but I am aware of my bias so I try not to let it cloud my judgement. On the topic of compression, I a…

LLMs don't think, nor are they intelligent or exhibiting intelligence. Language does have constraints, yet it evolves via its users to encompass new meanings. Thus those constraints are artificial, unless you artificially enforce static language use. And of course, for an LLM to use those new concepts, it needs to be retokenized by being trained on new data. For example, if we trained LLMs only on books, encyclopedia…

Do you think an average human from 1850 transported to the modern world would fare better than an LLM?

Re: All AI models might be the same

#68

Post by stillpointlab »

b112 wrote: post #55
stillpointlab wrote: post #51

I have to be careful of confirmation bias when I read stuff like this because I have the intuition that we are uncovering a single intelligence with each of the different LLMs. I even feel, when switching between the big three (OpenAI, Google, Anthropic) that there is a lot of similarity in how they speak and think - but I am aware of my bias so I try not to let it cloud my judgement. On the topic of compression, I a…

LLMs don't think, nor are they intelligent or exhibiting intelligence. Language does have constraints, yet it evolves via its users to encompass new meanings. Thus those constraints are artificial, unless you artificially enforce static language use. And of course, for an LLM to use those new concepts, it needs to be retokenized by being trained on new data. For example, if we trained LLMs only on books, encyclopedia…

I disagree with you but I'll only repeat and expand on my final statement: I'm glad there are people seriously studying this. If you are right (which I doubt) I hope it comes to light with evidence. And if you are wrong, I hope you have the grace to admit it.

Re: All AI models might be the same

#69

Post by hatthew »

b112 wrote: post #65
kevin42 wrote: post #62

Earlier quoted context omitted.

You made a lot of bold assertions there. It's as if you have a complete and definitive theory of human intelligence to compare it against. Which if true, would be incredible, because there isn't a scientifically accepted theory, nor is there consensus from a philosophical standpoint. I can't say that you are wrong, you might be right, especially about AGI. And I think it's unlikely that LLMs are the direct path to AG…

When an LLM can re-tokenize on the fly, due to newly learned data, let me know. It won't prove intelligence, but at least it won't be static like a book.

https://x.com/sukjun_hwang/status/1943703574908723674

> Tokenization has been the final barrier to truly end-to-end language models.

> We developed the H-Net: a hierarchical network that replaces tokenization with a dynamic chunking process directly inside the model, automatically discovering and operating over meaningful units of data

Re: All AI models might be the same

#70

Post by SJC_Hacker »

jxmorris12 wrote: post #3
tyronehed wrote: post #2

Especially if they are all me-too copies of a Transformer. When we arrive at AGI, you can be certain it will not contain a Transformer.

I don't think architecture matters. It seems to be more a function of the data somehow. I once saw a LessWrong post claiming that the Platonic Representation Hypothesis doesn't hold when you only embed random noise, as opposed to natural images: http://lesswrong.com/posts/Su2pg7iwBM55yjQdt/exploring-the-p...

Its a bit like saying algorithms don't matter for solving computational problems. Two different algorithms might produce equivalent results but if you have to wait years for an answer, when seconds matter, the slow algorithm isn't helpful.

I believe the current approach of using mostly a feed-forward in the inference stage, with well-filtered training data and backpropagation for discrete "training cycles" has limitations. I know this has been tried in the past, but something modelling how animal brains actually function, with continuous feedback, no explicit "training" (we're always being trained), might be the key.

Unfortunately our knowledge of "whats really going on" in the brains is still limited, investigative methods are crude as the brain is difficult to image at the resolution we need, and in real time. Last I checked no one's quite figured out how memory works, for example. Whether its "stored in the network" somehow through feedback (like a SR-latch or flip-flop in electronics) or whether there's some underlying chemical process within the neuron itself (we know that chemicals definitely regulate brain function, don't know how much it goes the other way and it can be used to encode state)

Post reply on HN