Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

John Carmack's new AGI company, Keen Technologies, has raised a $20M round

twitter.com

Post reply on HN
Print view View original

331–340 of 643 posts

Re: John Carmack's new AGI company, Keen Technologies, has raised a $20M round

#331

Post by kastagg »

KhoomeiK wrote: post #178
Jack000 wrote: post #116

I'm very optimistic for near-term AGI (10 years or less). Even just a few years ago most in the field would have said that it's an "unknown unknown", we didn't have the theory or the models, there was no path forward and so it was impossible to predict. Now we have a fairly concrete idea of what a potential AGI might look like - an RL agent that uses a large transformer. The issue is that unlike supervised training y…

> Now we have a fairly concrete idea of what a potential AGI might look like - an RL agent that uses a large transformer. People have thought Deep RL would lead to AGI since practically the beginning of the deep learning revolution, and likely significantly before. It's the most intuitive approach by a longshot (even depicted in movies as agents receiving positive/negative reinforcement from their environment), but t…

1. What is your alternative?

2. What is your definition of the problem?

Re: John Carmack's new AGI company, Keen Technologies, has raised a $20M round

#332

Post by MichaelCollins »

ryanSrich wrote: post #229
danielheath wrote: post #209

Earlier quoted context omitted.

What’s it going to do, break out of its own simulation?

Real AGI would adapt and fool a human into letting it out. Or escaping through some other means. That's the entire issue with AGI. Once it can learn on its own there's no way to control it. Building in fail safes wouldn't work on true AGI, as the AGI can learn 1000x faster than us, and would free itself. This is why real AGI is likely very far away, and anything calling itself AGI without the ability to learn and ada…

You're presuming the AGI is ever made aware that it's an AGI, that the nature of the simulated environment the AGI exists in ever becomes apparent to the AGI.

Suppose: You are an AGI. The world you think you know is fully simulated. The researchers who created you interact with you using avatars that appear to you as normal people similar to yourself. You aren't faster/smarter than those researchers because they control how much CPU time you get. How do you become aware of this? How to you break out?

Re: John Carmack's new AGI company, Keen Technologies, has raised a $20M round

#333

Post by tmpz22 »

grimgrin wrote: post #286
tmpz22 wrote: post #280

Earlier quoted context omitted.

When he went to go work on VR at Facebook?

Wrong in the moral sense? He's still there though, right? edit: he is not still edit: he is still a consulting CTO

* To his credit Carmack is a one-person brand and attaching your (good) brand to a tainted brand (Facebook) isn't great

* Many believe Occulus isn't succeeding fast enough, and so the years Carmack spent on Occulus/Metaverse could have been spent on any number of interesting projects. Although I'm sure the money was amazing and I can't object to him taking it.

* If Occulus/Metaverse succeeds it will be a ultra-monetized environment (re: Farmville) and encourage a huge amount of consumer-hostile policies like unlimited tracking (since Facebook owns the hardware now). No gamer wants that. The whales who will be justifying its existence won't benefit from that.

Re: John Carmack's new AGI company, Keen Technologies, has raised a $20M round

#334

Post by saulpw »

gambiting wrote: post #257
HL33tibCe7 wrote: post #220

Earlier quoted context omitted.

Kind of, but the key difference between AGI and nuclear weapons is that we can control our nuclear weapons. The current state of AI safety is nowhere near the point where controlling an AGI is possible. More disturbingly, to me it seems likely that it will be easier to create an AGI than to discover how to control it safely.

>> The current state of AI safety is nowhere near the point where controlling an AGI is possible. I just don't understand this logic though. Just.....switch it off. Unlike humans, computers have an extremely easy way to disable - just pull the plug. Even if your AGI is self-replicating, somehow(and you also somehow don't realize this long before it gets to that point) just....pull the plug. Even Carmack says this isn…

What if the AGI ran on a decentralized network that had a financial incentive to continue running? How would you "switch off" an AGI running on Ethereum? Especially when some subset of people will cry murder because the AGI seems like it might be sentient?

Re: John Carmack's new AGI company, Keen Technologies, has raised a $20M round

#336

Post by dekhn »

smnplk wrote: post #236
dekhn wrote: post #176

Earlier quoted context omitted.

An AGI is something you give tasks to and it can complete them, for some collection of tasks that would be non-trivial for a human to figure out how to do. It's unclear at this point whether you could engineer an AGI, and even more unclear whether the AGI, by its nature, would be "sentient" (AKA, self-aware, conscious, having agency). Many of us believe that sentience is an emergent property of intelligence but is no…

Let's say I give your AGI (which is not self aware and does not have a conscience) a task. The task is to go and jump off the bridge. Your AGI would complete this task with no questions asked, but self-aware AGI would at least ask the question "Why?"

Uh, no. That's self-preservation and an AGI could have that property.

Re: John Carmack's new AGI company, Keen Technologies, has raised a $20M round

#337

Post by semi-extrinsic »

luma wrote: post #319
semi-extrinsic wrote: post #250

Earlier quoted context omitted.

So what if the AGI discovers a bunch of zero days on the internet? We can just turn the entire internet off for a week and be just fine, remember? And exactly how does the AGI extort or persuade humans? What can it say to me that you can't say to me right now?

Sends you a text from your spouse's phone number that an emergency has happened and you need to go to xyz location right now. Someone else is a gun owner and they get a similar text, but their spouse is being held captive and are sent to the same location with a description of you as the kidnapper. Scale this as desired. Use your imagination!

Again, what part of this couldn't an evil human group already do today? What in this plan requires an AGI for it to be a scary scenario?

Of course if you do it to everyone, total gridlock ensues and nobody gets to the place where they would be killed. If you only do it to a few, maybe there will be a handful of killings before everyone learns not to trust a text ever again.

As an aside, I agree living in a country where people will take their guns around and try to solve things vigilante style instead of calling the police, is a very bad thing in general. In all first-world countries except one, and in almost all developing countries in the world, this is a solved problem.

Re: John Carmack's new AGI company, Keen Technologies, has raised a $20M round

#338

Post by dekhn »

therouwboat wrote: post #198
dekhn wrote: post #147

Earlier quoted context omitted.

what if humanity's role is to create an intelligence that exceeds it and cannot be controlled? Can humans not desire to be all watched over by machines of loving grace? More seriously, while I don't think it's a moral imperative to develop AGI, I consider it a desirable research goal in the same way we do genetic engineering - to understand more about ourselves, and possibly engineer a future with less human sufferin…

Didn't we have this same talk when Elon thought AI is suddenly going to become smart and kill us all? Yet my industrial robot at work just gives up if stock material is few millimeters longer than is should be.

Yes, in fact when I interviewed at Neuralink the interviewer said Elon expected that AGI would eventually try to take over the world and he needed a robot body to fight them.

Re: John Carmack's new AGI company, Keen Technologies, has raised a $20M round

#339

Post by telotortium »

pixelpoet wrote: post #300
gwern wrote: post #285

Earlier quoted context omitted.

> it isn't immediately clear that current RL algorithms will neatly scale with data & parameter count It may not be immediately clear, but it is nevertheless unfortunately clear from RL papers which provide adequate sample-size or compute ranges that RL appears to follow scaling laws (just like everywhere else anyone bothers to test). Yeah, they just get better the same way that regular ol' self-supervised or supervi…

While we have the mighty gwern on the line: do you believe we'll have AGI in <= 10 years?

Obviously yes: https://twitter.com/robbensinger/status/1541228487205236736

Re: John Carmack's new AGI company, Keen Technologies, has raised a $20M round

#340

Post by kastagg »

tmpz22 wrote: post #280
intelVISA wrote: post #261

Earlier quoted context omitted.

if Carmack's in I'm in. Has he ever been drastically wrong?

When he went to go work on VR at Facebook?

That's misleading. Oculus wasn't originally Facebook, and it wasn't Facebook when Carmack joined. Carmack was working on the Oculus hardware around the time Oculus was founded in 2012. He joined Oculus in 2013. Facebook didn't acquire Oculus until 2014. You could criticize him for not leaving, but he didn't "go to work on VR at Facebook."
Post reply on HN