Live data from Hacker News

John Carmack's new AGI company, Keen Technologies, has raised a $20M round

twitter.com

301–310 of 643 posts

Re: John Carmack's new AGI company, Keen Technologies, has raised a $20M round

#301
post #131

Earlier quoted context omitted.

We will have something that we ourselves define to be AGI, sure, but then it's easy to hit any goal that way. Is that machine really intelligent? What does that word even mean? Can it think for itself? Is it sentient? Similar to AI, AGI is going to be a new industry buzzword that you can throw at anything and mean nothing.

From the point when an AGI is capable of constructing a slightly better version of itself and has the urge to do so, everything can happen very fast.

People don't really consider the immense risk of "speed superintelligences" as a very quick and relatively easy follow-on step to the development of AGI.

Once developed, one solely needs to turn up the execution rate of an AGI, which would result in superhuman performance on most practical and economically meaningful metrics.

Imagine if for every real day that passed, one experienced 100 days of subjective time. Would that person be able to eclipse most of their peers in terms of intellectual output? Of course they would. In essence, that's what a speed superintelligence would be.

When most people think of AI outperforming humans, they tend to think of "quality superintelligences", AIs that can just "think better" than any human. That's likely to be a harder problem. But we don't even need quality superintelligences to utterly disrupt society as we know it.

We really need to stop arguing about time scales for the arrival of AGI, and start societal planning for its arrival whenever that happens. We likely already have the computational capacity for AGI, and have just not figured out the correct way to coordinate it. The human brain uses about 20 watts to do its thing, and humanity has gigawatts of computational capacity. Sure, the human brain should be considered to be "special purpose hardware" that dramatically reduces energy requirements for cognition. By a factor of more than 10^9 though? That seems unlikely.

Re: John Carmack's new AGI company, Keen Technologies, has raised a $20M round

#302
post #62
post #13

Carmack gave his opinions about AGI on a recent Lex Fridman interview. He has some good ideas.

I remember him saying we don't have "line of sight" to AGI, and there could just be "6 or so" breakthrough ideas needed to get there. And he said he was over 50% on us seeing "signs of life" by 2030. Something like being able to "boot up a bunch of remote Zoom workers" for your company. The "6 or so" breakthroughs sounds about right to me. But I don't really see the reason for being optimistic about 2030. It could ju…

>The "6 or so" breakthroughs sounds about right to me. But I don't really see the reason for being optimistic about 2030. It could just as easily be 2050, or 2100, etc.

Well if you read between the lines of the Gato paper there may be no more hurdles left and scale is the only boundary left.

>Not a recognition that we fundamentally still don't know how brains work! (or what intelligence is, etc.)

This is a really bad trope. We don't need to understand the brain to make an intelligence. Does Evolution understand how the brain works? Did we solve the Navier Stokes Equations before building flying planes? No.

Re: John Carmack's new AGI company, Keen Technologies, has raised a $20M round

#303
post #62

Earlier quoted context omitted.

I remember him saying we don't have "line of sight" to AGI, and there could just be "6 or so" breakthrough ideas needed to get there. And he said he was over 50% on us seeing "signs of life" by 2030. Something like being able to "boot up a bunch of remote Zoom workers" for your company. The "6 or so" breakthroughs sounds about right to me. But I don't really see the reason for being optimistic about 2030. It could ju…

There is an inverse relationship between the age of a futurist and the amount of time they think it will take for their predictions to become true. In other words, people making these sort of predictions about the future are biased towards believing they'll be alive to benefit from it.

> There is an inverse relationship between the age of a futurist and the amount of time they think it will take for their predictions to become true.

That's not true. The so-called Maes-Garreau law or effect does not replicate in actual surveys, as opposed to a few cherrypicked futurist examples.

Re: John Carmack's new AGI company, Keen Technologies, has raised a $20M round

#304
post #62

Earlier quoted context omitted.

I remember him saying we don't have "line of sight" to AGI, and there could just be "6 or so" breakthrough ideas needed to get there. And he said he was over 50% on us seeing "signs of life" by 2030. Something like being able to "boot up a bunch of remote Zoom workers" for your company. The "6 or so" breakthroughs sounds about right to me. But I don't really see the reason for being optimistic about 2030. It could ju…

There is an inverse relationship between the age of a futurist and the amount of time they think it will take for their predictions to become true. In other words, people making these sort of predictions about the future are biased towards believing they'll be alive to benefit from it.

> There is an inverse relationship between the age of a futurist and the amount of time they think it will take for their predictions to become true.

I think calling Carmack a Futurist is pretty insulting.

Re: John Carmack's new AGI company, Keen Technologies, has raised a $20M round

#305

Does anyone else subscribe to the idea that AGI is impossible/unlikely without 'embodied cognition', i.e. we cannot create a human-like 'intelligence' unless it has a similar embodiment to us, able to move around a physical environment with it's own limbs, sense of touch, sight, etc. ? Any arguments against the necessity of this? I feel like any AGI developed in silico without freedom of movement will be fundamentall…

[deleted]

Re: John Carmack's new AGI company, Keen Technologies, has raised a $20M round

#306
post #278

Does anyone else subscribe to the idea that AGI is impossible/unlikely without 'embodied cognition', i.e. we cannot create a human-like 'intelligence' unless it has a similar embodiment to us, able to move around a physical environment with it's own limbs, sense of touch, sight, etc. ? Any arguments against the necessity of this? I feel like any AGI developed in silico without freedom of movement will be fundamentall…

With VR, you can get the sight and physical environment you mention. That seems like, at minimum, proof that in-silico intelligence won't be blocked by that requirement. I do fully agree that any intelligence may not be human-like, though. In fact, I imagine it would seem very cold, calculating, amoral, and manipulative. Our prohibition against that type of behavior depends on a social evolution it won't have experie…

> With VR, you can get the sight and physical environment you mention

"physical environment" ? VR lets you operate on and get sensory input based on a fairly significantly degraded version of physical reality.

Re: John Carmack's new AGI company, Keen Technologies, has raised a $20M round

#307
post #285

Earlier quoted context omitted.

> Now we have a fairly concrete idea of what a potential AGI might look like - an RL agent that uses a large transformer. People have thought Deep RL would lead to AGI since practically the beginning of the deep learning revolution, and likely significantly before. It's the most intuitive approach by a longshot (even depicted in movies as agents receiving positive/negative reinforcement from their environment), but t…

> it isn't immediately clear that current RL algorithms will neatly scale with data & parameter count It may not be immediately clear, but it is nevertheless unfortunately clear from RL papers which provide adequate sample-size or compute ranges that RL appears to follow scaling laws (just like everywhere else anyone bothers to test). Yeah, they just get better the same way that regular ol' self-supervised or supervi…

That scaling will eventually hit a wall. What was it about nerds and S-curves?

Re: John Carmack's new AGI company, Keen Technologies, has raised a $20M round

#308
post #214

Earlier quoted context omitted.

At least we have lots of very complex simulated or pseudo-simulated environments already — throw your AGI agent into a sandbox mode game of GTA6, or like OpenAI and DeepMind already did, with DOTA2 and StarCraft II (with non-G-AIs). They have a vast almost-analog simulation space to figure out and interact with (including identifying or coming up with a goal). So while it is significant compute overhead, it at least…

I sure hope no one is planning to unleash their AGI in the real world after having it spend many (virtual) lifetimes playing GTA.

IMO, your take in the broader sense is an extremely profound and important point for AGI ethics. While GTA is seemingly extreme, I think that's going to be a problem no matter what simulation space we fabricate for training AGI agents — any simulation environment will encourage various behaviors by the biases encoded by the simulation's selectively enforced rules (because someone has to decide what rules the simulation implements...). An advanced intelligence will take learnings and interpretations of those rules beyond what humans would come up with.

If we can' make an AGI that we feel ok letting run amok in the world after living through a lot of GTA (by somehow being able to rapidly + intelligently reprioritize and adjust rules from multiple simulation/real environments? not sure), we probably shouldn't let that core AGI loose no matter what simulation(s) it was "raised on".

Re: John Carmack's new AGI company, Keen Technologies, has raised a $20M round

#309
post #254

Does anyone else subscribe to the idea that AGI is impossible/unlikely without 'embodied cognition', i.e. we cannot create a human-like 'intelligence' unless it has a similar embodiment to us, able to move around a physical environment with it's own limbs, sense of touch, sight, etc. ? Any arguments against the necessity of this? I feel like any AGI developed in silico without freedom of movement will be fundamentall…

Human like is actually very good question. It doesn't even come to embodiment, but in general how it would think and act. Lot of how we do is due to cultural, language, training and education. For example, which language would it "think" in? English? Some other? Something it's own? How would it formulate things? Based on what philosophical framework or similar? What about math? General reasoning? Then cultural norms…

The argument is that every question in your post is a red-herring. That is, humans don't actually function in the way that your questions suggest, even if they have the "experience" of doing so.
Post reply on HN