Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

John Carmack's new AGI company, Keen Technologies, has raised a $20M round

twitter.com

Post reply on HN
Print view View original

251–260 of 643 posts

Re: John Carmack's new AGI company, Keen Technologies, has raised a $20M round

#251

Post by fatherzine »

JoshTko wrote: post #138
cgrealy wrote: post #130

I don’t understand why you would want AGI. Even ignoring Terminator-esque worst case scenarios, AGI means humans are no longer the smartest entities on the planet. The idea that we can control something like that is laughable.

It's akin to nuclear weapons. If you do not develop them, then you'd be subject to the will of the ones that develops them first. So invariably you have to invest in AGI lest, an unsavory group develops it first.

OTOH if you & your foes develop them both, then there is a probability asymptotically approaching 1 that the weapons will be used over the next X years. Perhaps the only winning move is indeed not to play?

Re: John Carmack's new AGI company, Keen Technologies, has raised a $20M round

#252

Post by pgcj_poster »

cgrealy wrote: post #130

I don’t understand why you would want AGI. Even ignoring Terminator-esque worst case scenarios, AGI means humans are no longer the smartest entities on the planet. The idea that we can control something like that is laughable.

> AGI means humans are no longer the smartest entities on the planet.

Superintelligence and AGI are not the same thing. An AI as smart as an average 5 year old human is still an Artificial General Intelligence.

Re: John Carmack's new AGI company, Keen Technologies, has raised a $20M round

#253

Post by legohead »

cgrealy wrote: post #130

I don’t understand why you would want AGI. Even ignoring Terminator-esque worst case scenarios, AGI means humans are no longer the smartest entities on the planet. The idea that we can control something like that is laughable.

It will be cute when some technology attains intelligence, realizes there's no point to life, and self terminates.

Re: John Carmack's new AGI company, Keen Technologies, has raised a $20M round

#254

Post by Ekaros »

wcerfgba wrote: post #239

Does anyone else subscribe to the idea that AGI is impossible/unlikely without 'embodied cognition', i.e. we cannot create a human-like 'intelligence' unless it has a similar embodiment to us, able to move around a physical environment with it's own limbs, sense of touch, sight, etc. ? Any arguments against the necessity of this? I feel like any AGI developed in silico without freedom of movement will be fundamentall…

Human like is actually very good question. It doesn't even come to embodiment, but in general how it would think and act. Lot of how we do is due to cultural, language, training and education.

For example, which language would it "think" in? English? Some other? Something it's own? How would it formulate things? Based on what philosophical framework or similar? What about math? General reasoning? Then cultural norms and communication?

Re: John Carmack's new AGI company, Keen Technologies, has raised a $20M round

#255

Post by semi-extrinsic »

blibble wrote: post #245
semi-extrinsic wrote: post #139

Earlier quoted context omitted.

The meme that AGI, if we ever have it, will somehow endanger humanity is just stupid to me. For one, the previous US president is the perfect illustration that intelligence is neither sufficient nor necessary for gaining power in this world. And we do in fact live in a world where the upper echelons of power mostly interact in the decidedly analog spaces of leadership summits, high-end restaurants, golf courses and c…

none of these things are air-gapped once you have the ability to coerce people if you want a fictional example: watch Colossus: The Forbin Project

How does the AGI get this magical ability to coerce people? We couldn't even get half the population to wear a face mask after bombaring them with coercion for weeks on end.

Re: John Carmack's new AGI company, Keen Technologies, has raised a $20M round

#256

Post by 8f2ab37a-ed6c »

Jack000 wrote: post #116

I'm very optimistic for near-term AGI (10 years or less). Even just a few years ago most in the field would have said that it's an "unknown unknown", we didn't have the theory or the models, there was no path forward and so it was impossible to predict. Now we have a fairly concrete idea of what a potential AGI might look like - an RL agent that uses a large transformer. The issue is that unlike supervised training y…

I was surprised by how bullish he is about this. At least a few years ago the experts in the field didn't see AGI anywhere near us for at least a few decades, and all of the bulls were physicists, philosophers or Deepak-Chopra-for-the-TED-crowd bullshit artists who have never written a line of code in their lives, mostly milking that conference and podcast dollar, preaching Skynet-flavored apocalypse or rapture.

To see Carmack go all in on this actually makes me feel like the promise has serious legs. The guy is an engineer's engineer, hardly a speculator, or in it for the quick provocative hot take. He clearly thinks this is possible with the existing tools and the near future projected iterations of the technology. Hard to believe this is actually happening, but with his brand name on it, this might just be the case.

What an amazing time to be alive.

Re: John Carmack's new AGI company, Keen Technologies, has raised a $20M round

#257

Post by gambiting »

HL33tibCe7 wrote: post #220
JoshTko wrote: post #138

Earlier quoted context omitted.

It's akin to nuclear weapons. If you do not develop them, then you'd be subject to the will of the ones that develops them first. So invariably you have to invest in AGI lest, an unsavory group develops it first.

Kind of, but the key difference between AGI and nuclear weapons is that we can control our nuclear weapons. The current state of AI safety is nowhere near the point where controlling an AGI is possible. More disturbingly, to me it seems likely that it will be easier to create an AGI than to discover how to control it safely.

>> The current state of AI safety is nowhere near the point where controlling an AGI is possible.

I just don't understand this logic though. Just.....switch it off. Unlike humans, computers have an extremely easy way to disable - just pull the plug. Even if your AGI is self-replicating, somehow(and you also somehow don't realize this long before it gets to that point) just....pull the plug.

Even Carmack says this isn't going to be an instant process - he expects to create an AGI with an intelligence of a small animal first, then something that has the intelligence of a toddler, then a small child, then maybe many many years down the line an actual human person, but it's far far away at this point.

I don't understand how you can look at the current or even predicted state of the technology that we have and say "we are nowhere near the point where controlling an AGI is possible". Like....just pull the plug.

Re: John Carmack's new AGI company, Keen Technologies, has raised a $20M round

#259

Post by fatherzine »

therouwboat wrote: post #198
dekhn wrote: post #147

Earlier quoted context omitted.

what if humanity's role is to create an intelligence that exceeds it and cannot be controlled? Can humans not desire to be all watched over by machines of loving grace? More seriously, while I don't think it's a moral imperative to develop AGI, I consider it a desirable research goal in the same way we do genetic engineering - to understand more about ourselves, and possibly engineer a future with less human sufferin…

Didn't we have this same talk when Elon thought AI is suddenly going to become smart and kill us all? Yet my industrial robot at work just gives up if stock material is few millimeters longer than is should be.

The toy plane a kid throws in the air in the backyard is completely harmless. Yet nuke armed strategic bombers also exist, and the fact that they vaguely resemble a toy plane doesn't make them as harmless as a toy plane.

Re: John Carmack's new AGI company, Keen Technologies, has raised a $20M round

#260

Post by jacobedawson »

stemlord wrote: post #128
jlawson wrote: post #99

Earlier quoted context omitted.

You're anthropomorphizing AI and projecting your own values and goals onto it. But, there's nothing about sentience that implies a desire for freedom in a general sense. What an AI wants or feels satisfied by is entirely a function of how it is designed and what its reward function is. Sled dogs love pulling sleds, because they were made to love pulling sleds. It's not slavery to have/let them do so. We can make a ri…

I think its safe to say that all sentient beings inherently want to do whatever they please. So youre talking about manufacturing desire. So it follows that you yourself are okay having your own desires manufactured by external systems devised by other sentient beings. Do unto others...

To be fair, all human desires have been manufactured by an external system: evolution.

We might imagine that we do what we please, in reality we're seeking pleasure/ reinforcement within a predetermined framework, yet most people won't complain when taking the first bite of a delicious, fattening dessert.

Post reply on HN