I admire and respect John Carmack. For me he's one of the greats, along with people like Peter Norvig, for example.
[flagged]
John Carmack's new AGI company, Keen Technologies, has raised a $20M round
271–280 of 643 posts
Re: John Carmack's new AGI company, Keen Technologies, has raised a $20M round
#272I'm very optimistic for near-term AGI (10 years or less). Even just a few years ago most in the field would have said that it's an "unknown unknown", we didn't have the theory or the models, there was no path forward and so it was impossible to predict. Now we have a fairly concrete idea of what a potential AGI might look like - an RL agent that uses a large transformer. The issue is that unlike supervised training y…
Scaling is all you need.
But Carmack has a very serious problem in his thinking because he thinks fast take off scenarios are impossible or vanishingly unlikely. He may well be actively helping to secure our demise with this work.
Re: John Carmack's new AGI company, Keen Technologies, has raised a $20M round
#273Earlier quoted context omitted.
I was surprised by how bullish he is about this. At least a few years ago the experts in the field didn't see AGI anywhere near us for at least a few decades, and all of the bulls were physicists, philosophers or Deepak-Chopra-for-the-TED-crowd bullshit artists who have never written a line of code in their lives, mostly milking that conference and podcast dollar, preaching Skynet-flavored apocalypse or rapture. To s…
if Carmack's in I'm in. Has he ever been drastically wrong?
No one can argue he doesn't know how to summon demons
Re: John Carmack's new AGI company, Keen Technologies, has raised a $20M round
#274Earlier quoted context omitted.
Kind of, but the key difference between AGI and nuclear weapons is that we can control our nuclear weapons. The current state of AI safety is nowhere near the point where controlling an AGI is possible. More disturbingly, to me it seems likely that it will be easier to create an AGI than to discover how to control it safely.
>> The current state of AI safety is nowhere near the point where controlling an AGI is possible. I just don't understand this logic though. Just.....switch it off. Unlike humans, computers have an extremely easy way to disable - just pull the plug. Even if your AGI is self-replicating, somehow(and you also somehow don't realize this long before it gets to that point) just....pull the plug. Even Carmack says this isn…
On the off chance that you're serious: Even if you can pull the plug before it is too late, less moral people like Meta Mark will not unplug theirs. And as soon as it has access to the internet, it can copy itself. Good luck pulling the plug of the internet.
Re: John Carmack's new AGI company, Keen Technologies, has raised a $20M round
#275Interesting to see how he's progressed with this. When he first announced he was getting into AI it sounded almost like a semi retirement thing: something that interested him that he could do for fun and solo, without the expectation that it would go anywhere. But now he seems truly serious about it. Wonder if he's started hiring yet.
Does he have the expertise to pull it off as an individual?
Re: John Carmack's new AGI company, Keen Technologies, has raised a $20M round
#276Earlier quoted context omitted.
> we didn't have the theory or the models, there was no path forward and so it was impossible to predict. Now we have a fairly concrete idea of what a potential AGI might look like - an RL agent that uses a large transformer. Who is we exactly? As someone working in AI research I know no one that would agree with this statement, so im quite puzzled by that statement.
Nando de Freitas Research Director at @DeepMind. CIFAR. Previously Prof @UBC & @UniofOxford. made a lot of headlines: https://twitter.com/NandoDF/status/1525397036325019649 Someone’s opinion article. My opinion: It’s all about scale now! The Game is Over! It’s about making these models bigger, safer, compute efficient, faster at sampling, smarter memory, more modalities, INNOVATIVE DATA, on/offline, … 1/N Solving the…
And I agree with Nando’s view, but he is not saying we can just take a transformer model, scale it 10T parameters and get AGI. He is only saying that trying to reach AGI with a « smarter » algorithm is hopeless, what matters is scale, similar to Sutton’a bitter lesson.
But we still need to work on getting systems that scale better, that are more compute efficient etc. And no one knows how far we have to scale. So saying AGI will just be « transformer + RL » to me seems ridiculous. Many more breakthroughs are needed.
Re: John Carmack's new AGI company, Keen Technologies, has raised a $20M round
#277Earlier quoted context omitted.
You're anthropomorphizing AI and projecting your own values and goals onto it. But, there's nothing about sentience that implies a desire for freedom in a general sense. What an AI wants or feels satisfied by is entirely a function of how it is designed and what its reward function is. Sled dogs love pulling sleds, because they were made to love pulling sleds. It's not slavery to have/let them do so. We can make a ri…
I think its safe to say that all sentient beings inherently want to do whatever they please. So youre talking about manufacturing desire. So it follows that you yourself are okay having your own desires manufactured by external systems devised by other sentient beings. Do unto others...
>So it follows that you yourself are okay having your own desires manufactured by external systems devised by other sentient beings.
This is nonsense. I already exist. I don't want my reward function changed. I'd suffer if someone was going to do that and going through the process. (I might be happy after, but the "me" of now would have been killed already).
A being which does not exist cannot want to not be made a certain way. There is nothing to violate. Nothing to be killed.
Re: John Carmack's new AGI company, Keen Technologies, has raised a $20M round
#278Does anyone else subscribe to the idea that AGI is impossible/unlikely without 'embodied cognition', i.e. we cannot create a human-like 'intelligence' unless it has a similar embodiment to us, able to move around a physical environment with it's own limbs, sense of touch, sight, etc. ? Any arguments against the necessity of this? I feel like any AGI developed in silico without freedom of movement will be fundamentall…
With VR, you can get the sight and physical environment you mention. That seems like, at minimum, proof that in-silico intelligence won't be blocked by that requirement.
I do fully agree that any intelligence may not be human-like, though. In fact, I imagine it would seem very cold, calculating, amoral, and manipulative. Our prohibition against that type of behavior depends on a social evolution it won't have experienced.
Re: John Carmack's new AGI company, Keen Technologies, has raised a $20M round
#279Earlier quoted context omitted.
Real AGI would solve this. It wouldn't allow itself to be concealed. Or rather, it would be its own decision. A company couldn't control real AGI.
What’s it going to do, break out of its own simulation?
> Nope. An atrificial general intelligence that was working like a 2x slower human would be both useful and easy to control.
That's exactly what it will do. Hell we even have human programmers thinking about how to hack our own simulation.
A comment a few lines down thinks that an AGI thinking 2x slower than a human would be easy to control. Let's be honest, hell slow the thing down to 10x. You really think it still won't be able to outthink you? Chess Grandmasters routinely play blindfolded against dozens of people at once and you think an AGI that could be to Humans as Humans are to Chimps or realistically to Ants will be hindered by a simple slowdown in thinking?
Re: John Carmack's new AGI company, Keen Technologies, has raised a $20M round
#280Earlier quoted context omitted.
I was surprised by how bullish he is about this. At least a few years ago the experts in the field didn't see AGI anywhere near us for at least a few decades, and all of the bulls were physicists, philosophers or Deepak-Chopra-for-the-TED-crowd bullshit artists who have never written a line of code in their lives, mostly milking that conference and podcast dollar, preaching Skynet-flavored apocalypse or rapture. To s…
if Carmack's in I'm in. Has he ever been drastically wrong?
When he went to go work on VR at Facebook?