Andrej Karpathy – It will take a decade to work through the issues with agents • Hacker News Board

Re: Andrej Karpathy – It will take a decade to work through the issues with agents

#871

Post by ninetyninenine » Sat, Oct 18, 2025, 5:25 PM UTC

latexr wrote: post #758

amelius wrote: post #743

Earlier quoted context omitted.

Humans evolved to think the night sky is beautiful. That's also training. If humans were zapped by lightning every time they went outside at night, they would not think that a night sky is beautiful.

Being struck by lighting may affect your desire to go outside, but it has zero correlation with the sky’s beauty. Outer space is beautiful, poison dart frogs are beautiful, lava is beautiful. All of them can kill or maim you if you don’t wear protection, but that doesn’t take away from their beauty. Conversely, boring safe things aren’t automatically beautiful. I see no reasonable reason to believe that finding beaut…

Do you think a fat pig is beautiful? Like a hairy fat pig that snorts and rolls in the mud… is this animal so beautiful to you that you would want to make love to this animal?

Of course not! Because pigs are intrinsically and universally ugly and sex with a pig is universally disgusting.

But you realize that horny male pigs think this is beautiful right? Horny pigs want to fuck other pigs because horny pigs think fat sweaty female hogs are beautiful.

Beauty is arbitrary. It is not intrinsic. Even among life forms and among humans we all have different opinions on what is beautiful. I guarantee you there are people who think the night sky is ugly af.

Attributes like beauty are not such profound categories that separate an LLM from humanity. These are arbitrary classifications and even though you can’t fully articulate the “experience” you have of “beauty” the LLM can’t fully articulate its “experience” either. You think it’s impossible for the LLM to experience what you experience… but you really have no evidence for this because you have no idea what the LLM experiences internally.

Just like you can’t articulate what the LLM experiences neither can the LLM. These are both black box processes that can’t be described but neither is very profound given the fact that we all have completely different opinions on what is beautiful.

Re: Andrej Karpathy – It will take a decade to work through the issues with agents

#872

Post by tim333 » Sat, Oct 18, 2025, 5:36 PM UTC

jb1991 wrote: post #812

tim333 wrote: post #679

Earlier quoted context omitted.

I guess so. Is there reason to think an appropriate algorithm and scale can't do that?

Yes, perhaps an "appropriate algorithm" could, but it is my opinion that we have not found that algorithm. LLMs are cool but I think they are very primitive compared to human intelligence and we aren't even close to getting AGI via that route.

I agree with you that we are not there yet, algorithm wise.

Re: Andrej Karpathy – It will take a decade to work through the issues with agents

#873

Post by whatevertrevor » Sat, Oct 18, 2025, 5:39 PM UTC

arthurofbabylon wrote: post #693

whatevertrevor wrote: post #653

Earlier quoted context omitted.

> In the natural world, agency is a consequence of death: by dying, the feedback loop closes in a powerful way. I don't follow. If we, in some distant future, find a way to make humans functionally immortal, does that magically remove our agency? Or do we not have agency to begin with? If your position on the "free will" question is that it doesn't exist, then sure I get it. But that seems incompatible with the death…

When I think of the term "agency" I think of a feedback loop whereby an actor is aware of their effect and adjusts behavior to achieve desired effects. To be a useful agent, one must operate in a closed feedback loop; an open loop does not yield results. Consider the distinction between probabilistic and deterministic reasoning. When you are dealing with a probabilistic method (eg, LLMs, most of the human experience)…

I see your perspective about the inevitability of death causing a forcing-function directedness for agents, but that's a much much weaker claim than (emphasis mine):

> In the natural world, agency is a consequence of death: by dying, the feedback loop closes in a powerful way.

My original question was why could agency not exist without death, not why it was hampered without it. For clarity, I'm coming at from an analytic philosophy angle, not its more rhetorical counterpart that I struggle to wrap my head around.

I don't really view death or evolution as a necessity for agency. Nebulous AGI predictions aside: if a self-aware, conscious and intelligent being, capable of affecting consequential changes to its environment, becomes functionally immortal, it doesn't somehow lose its agency. I'd actually go further and say losing the forcing function of inevitable death is the biggest freedom a species can aim for. Without it, our agency is limited to solving problems of survival, in one form or another.

The existence of death is ultimately arbitrary and random, as random as our existence in the first place. The "direction" we get for evolution as a result of it, is another random function on top, also taking: the random circumstances the soup of organic molecules live in, as another parameter. Only once this random inevitability is conquered can we truly shape our lives and environments in ways that are a true reflection of who we are. Only then are we genuinely free. And "agency" without freedom is impotent at best.

(Addendum: I know positing "Immortality is good actually" can cause negative associations with "billionaires who want to cryopreserve themselves". This association has melded with the general romanticization of death in various philosophical and religious beliefs that has existed since millennia, further empowering the distaste against trying to reverse aging and eventually remove death as moral goals. While I personally have no plans (or means) to cryopreserve myself when I get old, I do believe it's a goal worth fighting for. One of the more important ones, alongside ensuring we have a planet to live on in the interim)

Re: Andrej Karpathy – It will take a decade to work through the issues with agents

#874

Post by ninetyninenine » Sat, Oct 18, 2025, 5:40 PM UTC

danans wrote: post #870

ninetyninenine wrote: post #861

Earlier quoted context omitted.

Except Sutton has no idea or even a clue about the internal model of a squirrel. He just uses it as a symbol for utterly stupid but still smarter than an LLM. It’s semantic manipulation in attempt to prove his point but he proves nothing. We have no idea how much of the world a squirrel understands. We understand LLMs more than squirrels. Arguably we don’t know if LLMs are more intelligent than squirrels. > Finally h…

> We have no idea how much of the world I squirrel understands. We understand LLMs more than squirrels Based on our understanding of biology and evolution we know that a squirrel brain works more similarly to the way we humans do vs an LLM. To the extent we understand LLMs, it's because they are strictly less complex than both ours and squirrels' brains, not because they are better model for our intelligence. They ar…

> Based on our understanding of biology and evolution we know that a squirrel understands its world more similarly to the way we do than an LLM.

Bro. Evolution is random walk. That means most of the changes are random and arbitrary based on whatever allows the squirrel to survive.

We know squirrels and humans diverged from a common ancestor but we do not know how much has changed since the common ancestor and we do not know what changed and we do not know the baseline for what this common ancestor is.

Additionally we don’t even understand the current baseline. We have no idea how brains work. if we did we would be able to build a human brain but as of right now LLMs are the closest model we have ever created to something that simulates or is remotely similar to the brain.

So your fuzzy qualitative statement of we understand evolution and biology is baseless. We don’t understand shit.

> We also see that a squirrel, like us, is capable of continuous learning driven by its own goals, all on an energy budget many orders of magnitude lower. That last part is a strong empirical indication that suggests that LLMs are a dead end for AGI.

So an LLM cant continuously learn? You realize that LLMs are deployed agentically all the time now so they both continuously learn and follow goals? Right? You’re aware of this i hope.

The energy efficiency is a byproduct of hardware. The theory of LLMs and machine learning is independent from the flawed silicon technology that is causing the energy efficiencies. Like how a computer can be made mechanical an LLM can be as well. The LLM is independent of the actual implementation and energy inefficiencies. This is not at all a strong empirical indication that LLMs are a dead end. It’s a strong indication that your thinking is illogical and flawed.

> Also remember that Sutton is still of an AI maximalist. He isn't saying that AGI isn't possible, just that LLMs can't get us there.

He can’t say any of this because he doesn’t actually know. None of us know for sure. We literally don’t know why LLMs work. The fact that training transformers on massive amounts of data produced this level of intelligence was a total surprise for all the experts and we still have no idea why this stuff works. His statements are too overarching and glossing over a lot of things we don’t actually know.

Yann lecuun for example called LLMs stochastic parrots. We now know this is largely incorrect. The reason Yan can be so wrong is because nobody actually knows shit.

Re: Andrej Karpathy – It will take a decade to work through the issues with agents

#875

Post by strangescript » Sat, Oct 18, 2025, 5:46 PM UTC

I love Karpathy, but he is wrong here. In a few short years we went from chat bots being toys and video creation predicted to be impossible in the near term to agents writing working apps and high def video that occasionally is indistinguishable from real life.

The rate depth, breadth and frequency of releases has only increased, not decreased. Meanwhile, everyone is waiting on bated breath for Gemini 3 to drop. A decade for reliable agents is not only comical, but willful cognitive dissonance at this point.

Re: Andrej Karpathy – It will take a decade to work through the issues with agents

#876

Post by heyjamesknight » Sat, Oct 18, 2025, 5:47 PM UTC

del82 wrote: post #569

ziofill wrote: post #545

Earlier quoted context omitted.

I agree with this. A metaphor I like is that the reason why humans say the night sky is beautiful is because they see that it is, whereas an LLM says it because it’s been said enough times in its training data.

I mean, I think the reason I would say the night sky is “beautiful” is because the meaning of the word for me is constructed from the experiences I’ve had in which I’ve heard other people use the word. So I’d agree that the night sky is “beautiful”, but not because I somehow have access to a deeper meaning of the word or the sky than an LLM does. As someone who (long ago) studied philosophy of mind and (Chomskian) li…

You don’t have a deeper “meaning of the word,” you have an actual experience of beauty. Three word is just a label for the thing you, me, and other humans have experienced.

The machine has no experience.

Re: Andrej Karpathy – It will take a decade to work through the issues with agents

#877

Post by LarsDu88 » Sat, Oct 18, 2025, 5:50 PM UTC

dreambuffer wrote: post #509

LarsDu88 wrote: post #443

Earlier quoted context omitted.

This world model talk is interesting, and Yann Lecunn has broached on the same topic, but the fact is there are video diffusion models that are quite good at representing the "video world" and even counterfactually and temporally coherently generating a representation of that "world" under different perturbations. In fact you can go to a SOTA LLM today, and it will do quite well at predicting the outcomes of basic co…

Photons hit a human eye and then the human came up with language to describe that and then encoded the language into the LLM. The LLM can capture some of this relationship, but the LLM is not sensing actual photons, nor experiencing actual light cone stimulation, nor generating thoughts. Its "world model" is several degrees removed from the real world. So whatever fragment of a model it gains through learning to comp…

The workings of a human eye versus a webcam is mostly an implementation detail IMO and has nothing important to say about what underlies "intelligence" or "world models"

It's like saying a component video out cable for the SNES is intrinsically different from an HDMI for putting an image on a screen. They are different, yes, but the outcome we care about is the same.

As for causality, go and give a frontier level LLM a simple counterfactual scenario. I think 4/5 will be able to answer correctly or reasonably for most basic cases. I even tried this exercise on some examples from Judea Pearl's 2018 book, "The Book of Why". The fact that current LLMs can tackle this sort of stuff is strongly indicative of there being a decent world model locked inside many of these language models.

Re: Andrej Karpathy – It will take a decade to work through the issues with agents

#878

Post by somenameforme » Sat, Oct 18, 2025, 5:51 PM UTC

modeless wrote: post #646

somenameforme wrote: post #130

Earlier quoted context omitted.

I lost all respect for him after reading about his views on medical immortality. His argument is that over time human life expectancy has been constantly increasing * and he calculated that based on some arbitrary rate of acceleration, that science would be expanding human life expectancy by more than a year, per year - medical immortality in other words, and all expected to happen just prior to the time he's reachin…

This is true, and I tend to believe that indefinite human lifespan extension will come too late for anyone who is already an adult today including myself. But I do think that it will come, mostly as a consequence of advanced AI accelerating medical research. It may be wishful thinking to believe that it will happen within our lifetimes, but that doesn't mean it won't ever happen.

While it'd be absurd to say it's impossible, the one thing I'd observe is that it's almost certain that a precursor to anything like this would be achieving something comparable in a simpler species. And that would likely come long before we might be able to see something similar in humans. For instance the fruit fly has been studied and experimented on extensively, particularly for aging, for over a century now.

But the results remain modest. The biggest breakthrough was in the 80s when somebody was able to roughly double their life expectancy from 2 months to 4 through artificial selection. But the context there is that fruit flies are a textbook 'quantity over quality' species, meaning that survival is not generally selected for, whereas humans are an equally textbook 'quality over quantity' species meaning that survival is one of the key things we select for. In other words, there was likely a lot more genetic low hanging fruit for survivability with fruit flies than there is for humans.

So I don't know. We need some serious acceleration and I'm not seeing much of anything when looked at with a critical eye.

Re: Andrej Karpathy – It will take a decade to work through the issues with agents

#879

Post by heyjamesknight » Sat, Oct 18, 2025, 5:55 PM UTC

ninetyninenine wrote: post #867

ziofill wrote: post #545

Earlier quoted context omitted.

I agree with this. A metaphor I like is that the reason why humans say the night sky is beautiful is because they see that it is, whereas an LLM says it because it’s been said enough times in its training data.

Guys you realize that you can go to ChatGPT right now and it can generate an actual picture of the night sky because it has seen thousands of pictures and drawings of the actual night sky right? Your logic is flawed because your knowledge is outdated. LLMs are encoding visual data, not just “language” data.

You misunderstand how the multimodal piece works. The fundamental unit of encoding here is still semantic. Not the same in your mind: you don’t need to know the word for sunset to experience the sunset.

Re: Andrej Karpathy – It will take a decade to work through the issues with agents

#880

Post by LarsDu88 » Sat, Oct 18, 2025, 5:56 PM UTC

danans wrote: post #835

LarsDu88 wrote: post #443

Earlier quoted context omitted.

This world model talk is interesting, and Yann Lecunn has broached on the same topic, but the fact is there are video diffusion models that are quite good at representing the "video world" and even counterfactually and temporally coherently generating a representation of that "world" under different perturbations. In fact you can go to a SOTA LLM today, and it will do quite well at predicting the outcomes of basic co…

> It's incredibly difficult to compress information without have at least some internal model of that information. Whether that model is a "world model" that fits the definition of folks like Sutton and LeCunn is semantic. Sutton's emphasizes his point by saying is that LLMs trying to reach AGI is futile because their world models are less capable that a squirrel's, in part because the squirrel has direct experiences…

This is actually a pretty good point, but quite honestly isn't this just an implementation detail? We can wire up a squirrel robot, give it a wifi connection to a Cerebras inference engine with a big context window, then let it run about during the day collecting a video feed while directing it to do "squirrel stuff".

Then during the night, we make it go to sleep and use the data collected during the day to continue finetuning the actual model weights in some data center somewhere.

After 2 years, this model would have a ton of "direct experiences" about the world.