851–860 of 1001 posts
Earlier quoted context omitted.
What was the last example where humans succeeded at a hard problem like that? Space flight?
Even if it's not some staggering triumph of human achievement, I'd argue that Ozempic (etc.) is similar. A magic weight loss drug has always captured the public's imagination, and it feels like I've been hearing about new weight loss drug studies in the news for my entire life that never went anywhere.
Earlier quoted context omitted.
Its very interesting to see how many people struggle to understand this.
We are paying the price now for not teaching language philosophy as a core educational requirement. Most people have had no exposure to even the most basic ideas of language philosophy. The idea all these people go to school for years and don't even have to take a 1 semester class on the main philosophical ideas of the 20th century is insane.
One must imagine Sisyphus happy and Chomsky incoherent with rage.
Earlier quoted context omitted.
This world model talk is interesting, and Yann Lecunn has broached on the same topic, but the fact is there are video diffusion models that are quite good at representing the "video world" and even counterfactually and temporally coherently generating a representation of that "world" under different perturbations. In fact you can go to a SOTA LLM today, and it will do quite well at predicting the outcomes of basic co…
Photons hit a human eye and then the human came up with language to describe that and then encoded the language into the LLM. The LLM can capture some of this relationship, but the LLM is not sensing actual photons, nor experiencing actual light cone stimulation, nor generating thoughts. Its "world model" is several degrees removed from the real world. So whatever fragment of a model it gains through learning to comp…
Map vs Territory is a common analogy. Maps describe territories but in an abstract and lossy manner.
But, most of us dont construct grounded concepts in our understanding. We carry a muddle of ungrounded ideas - some told to us by others, and some we intuit directly. There is a long tradition of attempting to think clearly all the way from Socrates, Descartes, Feynman etc.. where an attempt is made to ground the ideas we have. Try explaining your ideas to others, and soon, you will hit the illusion of explanatory depth.
LLM is a map and is a useful tool, but it doesnt interact with the territory, and it does not have skin in the game, and as a result, it cant carve new categories in a learning process that we have as humans.
Earlier quoted context omitted.
I mean, I think the reason I would say the night sky is “beautiful” is because the meaning of the word for me is constructed from the experiences I’ve had in which I’ve heard other people use the word. So I’d agree that the night sky is “beautiful”, but not because I somehow have access to a deeper meaning of the word or the sky than an LLM does. As someone who (long ago) studied philosophy of mind and (Chomskian) li…
The more I learn about AI, biology and the brain, the more it seems to me that the difference between life and machines is just complexity. People are just really really complex machines. However there are clearly qualitative differences between the human mind and any machines we know of yet, and those qualitative differences are emergent properties, in the same way that a rabbit is qualitatively different than a sto…
A “machine” is precisely what eliminates complexity by design. "People are complex machines" already has no meaning and then adding just and really doesn't make the statement more meaningful it makes it even more confused and meaningless.
The older I get the more obvious it becomes the idea of a "thinking machine" is a meaningless absurdity.
What we really think we want is a type of synthetic biological thinking organism that somehow still inherits the useful properties of a machine. If we say it that way though the absurdity is obvious and no one alive reading this will ever witness anything like that. Then we wouldn't be able to pretend we live at some special time in history that gets to see the birth of this new organism.
He also mentions that Waymo is switching to a vision-only approach, like Tesla has been doing for a couple years already.
Earlier quoted context omitted.
While you are right about the broader (and sort of ill defined) chase toward 'AGI' - another way to look at it is the self driving car - they got there eventually.And, if you work on applications using LLMs you can pretty easily see that Karpathy's sentiment is likely correct. You see it because you do it. Even simple applications are shaped like this, albeit each 9 takes less time than self driving cars for a simple…
> another way to look at it is the self driving car - they got there eventually Current self driving cars only work in American roads. Maybe Canada too, not sure how their roads are. Come to Europe/anywhere else and every other road would be intractable. Much tighter lanes, many turns you have a little mirror to see who's coming on the other side, single car at a time lanes that you need to "understand" who goes firs…
>What takes the long amount of time and the way to think about it is that it’s a march of nines. Every single nine is a constant amount of work. Every single nine is the same amount of work. When you get a demo and something works 90% of the time, that’s just the first nine. Then you need the second nine, a third nine, a fourth nine, a fifth nine. While I was at Tesla for five years or so, we went through maybe three…
The interview which I've watched recently with Rich Sutton left me with the impression that AGI is not just a matter of adding more 9s. The interviewer had an idea that he took for granted: that to understand language you have to have a model of the world. LLMs seem to udnerstand language therefore they've trained a model of the world. Sutton rejected the premise immediately. He might be right in being skeptical here…
Earlier quoted context omitted.
While you are right about the broader (and sort of ill defined) chase toward 'AGI' - another way to look at it is the self driving car - they got there eventually.And, if you work on applications using LLMs you can pretty easily see that Karpathy's sentiment is likely correct. You see it because you do it. Even simple applications are shaped like this, albeit each 9 takes less time than self driving cars for a simple…
> another way to look at it is the self driving car - they got there eventually. No they did not. Elon has been saying Tesla will get there “next year” since 2015. He is still saying that, and despite changing definitions, we still are not there.
Earlier quoted context omitted.
There is some evidence from Anthropic that LLMs do model the world. This paper[0] tracing their "thought" is fascinating. Basically an LLM translating across languages will "light up" (to use a rough fMRI equivalent) for the same concepts (e.g. bigness) across languages. It does have clusters of parameters that correlate with concepts, not just randomly "after X word tends to have Y word." Otherwise you would expect…
How large is a lion? Learning the size of objects using pure text analysis requires significant gymnastics. Vision demonstrates physical size more easily. Multimodal learning is important. Full stop. Purely textual learning is not sample efficient for world modeling and the optimization can get stuck in local optima that are easily escaped through multimodal evidence. ("How large are lions? inducing distributions ove…
Ask a blind person that question - they can answer it.
Too many people think you need to "see" as in human sight to understand things like this. You obviously don't. The massive training data these models ingest is more than sufficient to answer this question - and not just by looking up "dimensions of a lion" in the high-dimensional space.
The patterns in that space are what generates the concept of what a lion is. You don't need to physically see a lion to know those things.
Earlier quoted context omitted.
While you are right about the broader (and sort of ill defined) chase toward 'AGI' - another way to look at it is the self driving car - they got there eventually.And, if you work on applications using LLMs you can pretty easily see that Karpathy's sentiment is likely correct. You see it because you do it. Even simple applications are shaped like this, albeit each 9 takes less time than self driving cars for a simple…
> another way to look at it is the self driving car - they got there eventually Current self driving cars only work in American roads. Maybe Canada too, not sure how their roads are. Come to Europe/anywhere else and every other road would be intractable. Much tighter lanes, many turns you have a little mirror to see who's coming on the other side, single car at a time lanes that you need to "understand" who goes firs…