Andrej Karpathy – It will take a decade to work through the issues with agents

You're right that there is no purely mathematical argument; it's almost non-sensical to claim such.

Instead you can simply make the rather reasonable observation that LLMs are a product of their training distribution, which only contains partial coverage of all possible observable states of the world. Some highly regular observable states are thus likely missing, but an embodied agent (like a human) would be able to correctly reason about these states due to other sensory modes that observe these states. It's very clear that the number of possible states far exceeds any text describing those states (the number of states transcribed to text is vanishingly small in comparison, but they are locally coherent and quite regular when observed through other sensory modes like vision). That's the closest you can make to a mathematical argument.

As a result one common failure case for an LLM might involve describing a specific physical environment and associated set of actions that are in an area of low (or no) probability mass in the training data and asking the LLM to reason about it. Here's an example where I asked ChatGPT the following:

  imagine i walked into a room through a doorway, then turned around and walked right back out of the room. from the perspective of a person in the room facing the door, which side would my right arm be? please explain.

It responded by incorrectly stating the following:

  Let’s break this down step by step.

  1. *You walk into the room through the doorway*:

   * As you're entering, the person in the room is facing the door. So, you're entering from the **outside**.
   * Your **right arm** will be on the **right side** of your body, just as it is normally when you're standing.

  2. *You turn around and walk right back out*:

   * When you turn around to walk out, you’ll be facing **away** from the room, looking back toward the doorway.
   * Since you're walking **back out**, the person inside the room is still facing the door, while you are walking away from them, so your **right arm** is still on the right side of your body.

  So, from the perspective of the person in the room, as you’re exiting, your *right arm* will still be on their *left side*. That’s because they’re facing the door, and your right side is to their left when you’re turning around and leaving.

  To visualize:

  * Imagine a door in front of them.
  * You walk through and turn around, so you're now facing away from them.
  * Your right arm is on your right side, but to them, it's on their left side since you’re now moving in the opposite direction.

I will tell my wife (who does our investing) of your bet: I've always felt a bit too invested in AI promises.

jb1991 says >"Which of course raises the interesting question of how I can make good on this bet."Have children...

Yes. I've run numerous 50Ks, 50 milers, 100ks and 100 milers. I felt like crap after 20 miles in almost all of them. Most of getting better at ultramarathons is learning to keep going when feeling like crap. Oddly, the one race that was an exception is probably the hardest one of them I did on paper - in that case I was going so slowly from the beginning that I never really hit a 20 mile wall.

So if you encounter an unknown intelligence, like I dunno some kind of extra dimensional pen pal with a wildly different biology and environment than our own... Would you be open to the possibilities:

- despite our difference we have the same kind of intelligence

- our intelligences intersect, but there are capacities that each has that the other doesn't

?

It seems like for either to be true there would have to be some place of common ground into which we could both generalize independently of our circumstance. Mathematics is often thought to be such a place for instance, there's plenty of sci fi about beaming prime numbers into space as an attempt to leverage that common ground. Are you saying there aren't such places? That SETI is hopeless?

Long form engineering tasks aren’t doable yet without supervision. But I can say in our shop, we won’t be hiring any more junior devs, ever, except as (in my region, free) interns or because of some extraordinary capabilities, insights, or skills. There just isn’t any business case for hiring junior devs to do the grunt work anymore.

But, the vast majority of work that is done in the world is not in the same order of magnitude of complexity or rigor that is required by long form engineering.

While models may not outperform an experienced developer, they will likely outperform her junior assistant, and a dev using ai effectively will almost certainly outperform a team of three without ai, in most cases.

The salient fact here is not that the human is outperformed by the model in a narrow field of extraordinary capability, but rather that the model can outperform that dev in 100 other disciplines, and outperform most people in almost any cerebral task.

My claim is not that models outperform people in all tasks, but that models outperform all people at many tasks, and I think that holds true with some caveats, especially when you factor in speed and scale.