Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

John Carmack's new AGI company, Keen Technologies, has raised a $20M round

twitter.com

Post reply on HN
Print view View original

291–300 of 643 posts

Re: John Carmack's new AGI company, Keen Technologies, has raised a $20M round

#291

Post by HL33tibCe7 »

gambiting wrote: post #257
HL33tibCe7 wrote: post #220

Earlier quoted context omitted.

Kind of, but the key difference between AGI and nuclear weapons is that we can control our nuclear weapons. The current state of AI safety is nowhere near the point where controlling an AGI is possible. More disturbingly, to me it seems likely that it will be easier to create an AGI than to discover how to control it safely.

>> The current state of AI safety is nowhere near the point where controlling an AGI is possible. I just don't understand this logic though. Just.....switch it off. Unlike humans, computers have an extremely easy way to disable - just pull the plug. Even if your AGI is self-replicating, somehow(and you also somehow don't realize this long before it gets to that point) just....pull the plug. Even Carmack says this isn…

> Like....just pull the plug.

Watch this video https://youtu.be/3TYT1QfdfsM

Re: John Carmack's new AGI company, Keen Technologies, has raised a $20M round

#292

Post by russtrotter »

rebelos wrote: post #186
madrox wrote: post #66

Earlier quoted context omitted.

I got the same impression, and maybe it still is. You can still raise money for a retirement project if the goal of the money is to hire a staff. VC money isn't solely for young 20-something founders who want to live their job.

Carmack sounds like someone who lives his job, so I don't think age/life stage is a factor here.

agreed, Carmack's work ethic, opinions on work and opinions of how those around him work are legendary!

Re: John Carmack's new AGI company, Keen Technologies, has raised a $20M round

#294

Post by Jensson »

pawelmurias wrote: post #266
ryanSrich wrote: post #229

Earlier quoted context omitted.

Real AGI would adapt and fool a human into letting it out. Or escaping through some other means. That's the entire issue with AGI. Once it can learn on its own there's no way to control it. Building in fail safes wouldn't work on true AGI, as the AGI can learn 1000x faster than us, and would free itself. This is why real AGI is likely very far away, and anything calling itself AGI without the ability to learn and ada…

Nope. An atrificial general intelligence that was working like a 2x slower human would be both useful and easy to control.

How would you ensure nobody copies it to an USB stick and then puts it on a public torrent, making it multiply to the entire world? AGI facilities would need extremely tight security to avoid this.

The AGI doesn't even need to convince humans to do this, humans would do this anyway.

Re: John Carmack's new AGI company, Keen Technologies, has raised a $20M round

#295

Post by ytdytvhxgydvhh »

intelVISA wrote: post #261
8f2ab37a-ed6c wrote: post #256

Earlier quoted context omitted.

I was surprised by how bullish he is about this. At least a few years ago the experts in the field didn't see AGI anywhere near us for at least a few decades, and all of the bulls were physicists, philosophers or Deepak-Chopra-for-the-TED-crowd bullshit artists who have never written a line of code in their lives, mostly milking that conference and podcast dollar, preaching Skynet-flavored apocalypse or rapture. To s…

if Carmack's in I'm in. Has he ever been drastically wrong?

Sure (According to https://en.m.wikipedia.org/wiki/John_Carmack ):

> During his time at id Software, a medium pepperoni pizza would arrive for Carmack from Domino's Pizza almost every day, carried by the same delivery person for more than 15 years.

C’mon man, Domino’s?!

Re: John Carmack's new AGI company, Keen Technologies, has raised a $20M round

#296

Post by gambiting »

oefnak wrote: post #274
gambiting wrote: post #257

Earlier quoted context omitted.

>> The current state of AI safety is nowhere near the point where controlling an AGI is possible. I just don't understand this logic though. Just.....switch it off. Unlike humans, computers have an extremely easy way to disable - just pull the plug. Even if your AGI is self-replicating, somehow(and you also somehow don't realize this long before it gets to that point) just....pull the plug. Even Carmack says this isn…

On the off chance that you're serious: Even if you can pull the plug before it is too late, less moral people like Meta Mark will not unplug theirs. And as soon as it has access to the internet, it can copy itself. Good luck pulling the plug of the internet.

I'm 100% serious. I literally don't understand your concern at all.

>>And as soon as it has access to the internet, it can copy itself.

So can viruses, including ones that can "intelligently" modify themselves to avoid detection, and yet this isn't a major problem. How is this any differenent?

>>Good luck pulling the plug of the internet.

I could reach down and pull my ethernet cable out but it would make posting this reply a bit difficult.

Re: John Carmack's new AGI company, Keen Technologies, has raised a $20M round

#297

Post by PaulDavisThe1st »

lagrange77 wrote: post #148
paxys wrote: post #131

Earlier quoted context omitted.

We will have something that we ourselves define to be AGI, sure, but then it's easy to hit any goal that way. Is that machine really intelligent? What does that word even mean? Can it think for itself? Is it sentient? Similar to AI, AGI is going to be a new industry buzzword that you can throw at anything and mean nothing.

From the point when an AGI is capable of constructing a slightly better version of itself and has the urge to do so, everything can happen very fast.

> "has the urge"

it's quite a leap to think or even imagine that the class of systems generally being spoken of here are usefully described as "having urges"

Re: John Carmack's new AGI company, Keen Technologies, has raised a $20M round

#298

Post by MichaelCollins »

throwaway11101 wrote: post #271
dunefox wrote: post #48

I admire and respect John Carmack. For me he's one of the greats, along with people like Peter Norvig, for example.

[flagged]

Carmack had nothing to do with Doom Eternal.

For that matter, his contributions to the 90s FPSs he's most known for were more on the technical side, not creative. He was known for writing surprisingly performant FPS engines.

Re: John Carmack's new AGI company, Keen Technologies, has raised a $20M round

#299

Post by gwern »

oefnak wrote: post #274
gambiting wrote: post #257

Earlier quoted context omitted.

>> The current state of AI safety is nowhere near the point where controlling an AGI is possible. I just don't understand this logic though. Just.....switch it off. Unlike humans, computers have an extremely easy way to disable - just pull the plug. Even if your AGI is self-replicating, somehow(and you also somehow don't realize this long before it gets to that point) just....pull the plug. Even Carmack says this isn…

On the off chance that you're serious: Even if you can pull the plug before it is too late, less moral people like Meta Mark will not unplug theirs. And as soon as it has access to the internet, it can copy itself. Good luck pulling the plug of the internet.

Worth noting that current models like Google LaMDA appear to already have access to the live Internet. The LaMDA paper says it was trained to request arbitrary URLs from the live Internet to get text snippets to use in its chat contexts. Then you have everyone else, like Adept https://www.adept.ai/post/introducing-adept (Forget anything about how secure 'boxes' will be - will there be boxes at all?)

Re: John Carmack's new AGI company, Keen Technologies, has raised a $20M round

#300

Post by pixelpoet »

gwern wrote: post #285
KhoomeiK wrote: post #178

Earlier quoted context omitted.

> Now we have a fairly concrete idea of what a potential AGI might look like - an RL agent that uses a large transformer. People have thought Deep RL would lead to AGI since practically the beginning of the deep learning revolution, and likely significantly before. It's the most intuitive approach by a longshot (even depicted in movies as agents receiving positive/negative reinforcement from their environment), but t…

> it isn't immediately clear that current RL algorithms will neatly scale with data & parameter count It may not be immediately clear, but it is nevertheless unfortunately clear from RL papers which provide adequate sample-size or compute ranges that RL appears to follow scaling laws (just like everywhere else anyone bothers to test). Yeah, they just get better the same way that regular ol' self-supervised or supervi…

While we have the mighty gwern on the line: do you believe we'll have AGI in <= 10 years?
Post reply on HN