Getting to AGI is not the problem. Finding the planet that it is going to run on will be.
Andrej Karpathy – It will take a decade to work through the issues with agents
221–230 of 1001 posts
Re: Andrej Karpathy – It will take a decade to work through the issues with agents
#222I think it's a shame that a 146 minute podcast released ~55 minutes ago has so much discussion. Everybody here is clearly just reacting to the title with their own biases. I know it's against the guidelines to discuss the state of a thread, but I really wish we could have thoughtful conversations about the content of links instead of title reactions.
Idk how this Dwarkesh Patel got so popular so fast. I'd never heard of him and he keeps popping up in my feeds.
Re: Andrej Karpathy – It will take a decade to work through the issues with agents
#223Amazing that he speaks the truth even tho $trillions and his stock options? depends on it. He and Dennis H. deserves all the respect.
Re: Andrej Karpathy – It will take a decade to work through the issues with agents
#224>What takes the long amount of time and the way to think about it is that it’s a march of nines. Every single nine is a constant amount of work. Every single nine is the same amount of work. When you get a demo and something works 90% of the time, that’s just the first nine. Then you need the second nine, a third nine, a fourth nine, a fifth nine. While I was at Tesla for five years or so, we went through maybe three…
The interview which I've watched recently with Rich Sutton left me with the impression that AGI is not just a matter of adding more 9s. The interviewer had an idea that he took for granted: that to understand language you have to have a model of the world. LLMs seem to udnerstand language therefore they've trained a model of the world. Sutton rejected the premise immediately. He might be right in being skeptical here…
> that to understand knowledge you have to have a model of the world.
Think of it this way: can you pass a test without understanding the test material? Certainly we all saw people we thought were idiots do well in class while we've also seen people we thought were geniuses fail. The test and understanding usually correlates but it's not perfect, right?
The reason I say understanding requires a world model (and I would not say LLMs understand) is because to understand you have to be able to detail things. Look at physics, or the far more detail oriented math. Physicists don't conclude things just off of experimental results. It's an important part, but not the whole story. They also write equations, ones which are counterfactual. You can call this compression if you want (I would and do), but it's only that because of the generalization. But it also only has that power because of the details and nuance.
With AI many of these people have been screaming for years (check my history) that what we're doing won't get us all the way there. Not because we want to stop the progress, but because we wanted to ensure continued and accelerate progress. We knew the limits and were saying "let's try to get ahead of this problem" but were told "that'll never be a problem. And if it is, we'll deal with it when we deal with it." It's why Chollet made the claim that LLMs have actually held AI progress back. Because the story that was sold was "AGI is solved, we just need to scale" (i.e. more money). I do still wonder how different things would be if those of us pushing back were able to continue and scale our works (research isn't free, so yes, people did stop us). We always had the math to show that scale wasn't enough, but it's easy to say "you don't need math" when you can see progress. The math never said no progress nor no acceleration, the math said there's a wall and it's easier to adjust now than when we're closer and moving faster. Sadly I don't think we'll ever shift the money over. We still evaluate success weirdly. Successful predictions don't matter. You're still heralded if you made a lot of money in VR and Bitcoin, right?
Re: Andrej Karpathy – It will take a decade to work through the issues with agents
#225Earlier quoted context omitted.
Reminds me of a time-honored aphorism in running: A marathon consists of two halves: the first 20 miles, and then the last 10k (6.2mi) when you're more sore and tired than you've ever been in your life.
This is 100% unrelated to the original article but I feel like there's an underreported additional first half. As a bigger runner who still loves to run, the first two or three miles before I have enough endorphins to get into the zen state that makes me love running is the first half, then it's 17 miles of this amazing meditative mindset. Then the last 10k sucks.
Just, ftr, endorphins cannot pass the blood brain barrier
http://hopkinsmedicine.org/health/wellness-and-prevention/th...
Re: Andrej Karpathy – It will take a decade to work through the issues with agents
#226>What takes the long amount of time and the way to think about it is that it’s a march of nines. Every single nine is a constant amount of work. Every single nine is the same amount of work. When you get a demo and something works 90% of the time, that’s just the first nine. Then you need the second nine, a third nine, a fourth nine, a fifth nine. While I was at Tesla for five years or so, we went through maybe three…
The interview which I've watched recently with Rich Sutton left me with the impression that AGI is not just a matter of adding more 9s. The interviewer had an idea that he took for granted: that to understand language you have to have a model of the world. LLMs seem to udnerstand language therefore they've trained a model of the world. Sutton rejected the premise immediately. He might be right in being skeptical here…
There is some evidence from Anthropic that LLMs do model the world. This paper[0] tracing their "thought" is fascinating. Basically an LLM translating across languages will "light up" (to use a rough fMRI equivalent) for the same concepts (e.g. bigness) across languages.
It does have clusters of parameters that correlate with concepts, not just randomly "after X word tends to have Y word." Otherwise you would expect all of Chinese to be grouped in one place, all of French in another, all of English in another. This is empirically not the case.
I don't know whether to understand knowledge you have to have a model of the world, but at least as far as language, LLMs very much do seem to have modeling.
[0]: https://www.anthropic.com/research/tracing-thoughts-language...
Re: Andrej Karpathy – It will take a decade to work through the issues with agents
#227Earlier quoted context omitted.
A transistor is very simple too, and here we are. Don’t dismiss something because it’s simple.
You got to look at how it scales. LLMs have already stopped increasing in parameter count as they don't get better by scaling them up anymore. New ideas are needed.
You’re right… but still, what was done until today is significant and useful already
Re: Andrej Karpathy – It will take a decade to work through the issues with agents
#228>What takes the long amount of time and the way to think about it is that it’s a march of nines. Every single nine is a constant amount of work. Every single nine is the same amount of work. When you get a demo and something works 90% of the time, that’s just the first nine. Then you need the second nine, a third nine, a fourth nine, a fifth nine. While I was at Tesla for five years or so, we went through maybe three…
Reminds me of a time-honored aphorism in running: A marathon consists of two halves: the first 20 miles, and then the last 10k (6.2mi) when you're more sore and tired than you've ever been in your life.
Why just run 20 miles then?
Re: Andrej Karpathy – It will take a decade to work through the issues with agents
#229I have massive respect for Andrej, my first encounter with "him" was following his tutorials/notes when he was a grad student/tutor for AI/ML. I was a lot disappointed when he went to work for Tesla, and I think that he had some achievement there, butnot nearly the impact I believe he potentially has. His switch (back?) to OpenAI was, in my mind, much more in keeping with where his spirit really lies. So, with that i…
How to tell if you regurgitated this comment vs being truly creative? If you can show me objectively, I’m sold.
Re: Andrej Karpathy – It will take a decade to work through the issues with agents
#230Now that Nvidia is the most valuable company, all this talk of actual AGI will be washed away by the huge amount of dollars driving the hype train. Most of these companies value is built on the idea of AGI being achievable in the near future. AGI being too close or too far away affects the value of these companies- too close and it'll seem too likely that the current leaders will win. Too far away and the level of sp…
It’s possible for AI to provide tremendous economic value without AGI
that is doubtful? sure it provides a lot of value but current levels are dotcom top level. Everyone knew internet had value but stocks push it too high