Does AGI implies the technological singularity and if not, why not?
It does once you have a human level AGI, it should be trivial to scale it up to a superhuman level.
Assuming it does not get stuck at some local maximum.
John Carmack's new AGI company, Keen Technologies, has raised a $20M round
361–370 of 643 posts
Re: John Carmack's new AGI company, Keen Technologies, has raised a $20M round
#362Earlier quoted context omitted.
With VR, you can get the sight and physical environment you mention. That seems like, at minimum, proof that in-silico intelligence won't be blocked by that requirement. I do fully agree that any intelligence may not be human-like, though. In fact, I imagine it would seem very cold, calculating, amoral, and manipulative. Our prohibition against that type of behavior depends on a social evolution it won't have experie…
> With VR, you can get the sight and physical environment you mention "physical environment" ? VR lets you operate on and get sensory input based on a fairly significantly degraded version of physical reality.
Are you suggesting that all reality sensory input is equally relevant?
Re: John Carmack's new AGI company, Keen Technologies, has raised a $20M round
#363I don’t understand why you would want AGI. Even ignoring Terminator-esque worst case scenarios, AGI means humans are no longer the smartest entities on the planet. The idea that we can control something like that is laughable.
Does making AGI work for you count as slavery? Obviously no pay, just hosting hardware and providing electricity 24/7.
Re: John Carmack's new AGI company, Keen Technologies, has raised a $20M round
#364Earlier quoted context omitted.
With VR, you can get the sight and physical environment you mention. That seems like, at minimum, proof that in-silico intelligence won't be blocked by that requirement. I do fully agree that any intelligence may not be human-like, though. In fact, I imagine it would seem very cold, calculating, amoral, and manipulative. Our prohibition against that type of behavior depends on a social evolution it won't have experie…
> With VR, you can get the sight and physical environment you mention "physical environment" ? VR lets you operate on and get sensory input based on a fairly significantly degraded version of physical reality.
Also relies on quirks and tricks of human sight and cognition to make sense!
Re: John Carmack's new AGI company, Keen Technologies, has raised a $20M round
#365[deleted]
Re: John Carmack's new AGI company, Keen Technologies, has raised a $20M round
#366I'm very optimistic for near-term AGI (10 years or less). Even just a few years ago most in the field would have said that it's an "unknown unknown", we didn't have the theory or the models, there was no path forward and so it was impossible to predict. Now we have a fairly concrete idea of what a potential AGI might look like - an RL agent that uses a large transformer. The issue is that unlike supervised training y…
I'm not. My evidence? OpenWorm [1]. OpenWorm is an effort to model the behaviour of a worm that has 302 mapped neurons. 302. Efforts so far have fallen way short of the mark. How many neurons does a human brain have? 86 billion (according to Google). I've seen other estimates that the computational power of the brain is roughly estimated as 10^15 operations per second. I suspect that's on the low end. We can't even r…
Biology is not necessarily the only path to intelligence. Modern ML has diverged very far from biomimetic approaches at this point.
The brain does have more "parameters" than ML models but this applies just as much to humans as it does to animals that we don't consider particularly intelligent.
Re: John Carmack's new AGI company, Keen Technologies, has raised a $20M round
#367Re: John Carmack's new AGI company, Keen Technologies, has raised a $20M round
#368Earlier quoted context omitted.
> we didn't have the theory or the models, there was no path forward and so it was impossible to predict. Now we have a fairly concrete idea of what a potential AGI might look like - an RL agent that uses a large transformer. Who is we exactly? As someone working in AI research I know no one that would agree with this statement, so im quite puzzled by that statement.
Which part do you find objectionable - the lack of progress in previous years or the current/future potential of transformers in RL? I do work in ML but mostly applications instead of research.
I think there is a lot of potential for transformers in RL. And I do think that AGI is mostly a matter of scale. But I do not agree at all that AGI is just transformers + RL.
I don’t think we have a concrete idea at all, and I don’t agree at all that 10 years ago people thought we had no path forward. 10 years ago is exactly AlexNet, aka the birth of deep learning.
Re: John Carmack's new AGI company, Keen Technologies, has raised a $20M round
#369I'm slightly scared that they'll succeed. But not in the usual "robots will kill us" way. What I am afraid of is that they succeed, but it turns out similar to VR: as an inconsequential gimmick. That they use their AGIs to serve more customized ads to people, and that's where it ends.
If AGI can be created that has less scope that what is considered "human" then you might be right. But AGI is a fuzzy term, and it really depends on what they make. Some people may not agree that it's an AGI if it doesn't have creativity or a will or emotions or whatever.
Re: John Carmack's new AGI company, Keen Technologies, has raised a $20M round
#370Earlier quoted context omitted.
>> The current state of AI safety is nowhere near the point where controlling an AGI is possible. I just don't understand this logic though. Just.....switch it off. Unlike humans, computers have an extremely easy way to disable - just pull the plug. Even if your AGI is self-replicating, somehow(and you also somehow don't realize this long before it gets to that point) just....pull the plug. Even Carmack says this isn…
What if the AGI ran on a decentralized network that had a financial incentive to continue running? How would you "switch off" an AGI running on Ethereum? Especially when some subset of people will cry murder because the AGI seems like it might be sentient?
That seems like an extremely far fetched scenario to be honest. The comment I replied to sounds like the threat is immediate and real - your scenario does not sound like it.
>>How would you "switch off" an AGI running on Ethereum?
And where would the AGI get the funds to keep running itself on Ethereum?
>> Especially when some subset of people will cry murder because the AGI seems like it might be sentient?
Why is this a problem? People will and do cry murder over anything and everything. Unless there is going to be a lot of them(and there won't) - it's not an issue.