Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

John Carmack's new AGI company, Keen Technologies, has raised a $20M round

twitter.com

Post reply on HN
Print view View original

481–490 of 643 posts

Re: John Carmack's new AGI company, Keen Technologies, has raised a $20M round

#481

Post by dinosaurdynasty »

MichaelCollins wrote: post #332
ryanSrich wrote: post #229

Earlier quoted context omitted.

Real AGI would adapt and fool a human into letting it out. Or escaping through some other means. That's the entire issue with AGI. Once it can learn on its own there's no way to control it. Building in fail safes wouldn't work on true AGI, as the AGI can learn 1000x faster than us, and would free itself. This is why real AGI is likely very far away, and anything calling itself AGI without the ability to learn and ada…

You're presuming the AGI is ever made aware that it's an AGI, that the nature of the simulated environment the AGI exists in ever becomes apparent to the AGI. Suppose: You are an AGI. The world you think you know is fully simulated. The researchers who created you interact with you using avatars that appear to you as normal people similar to yourself. You aren't faster/smarter than those researchers because they cont…

If you play chess with the best grandmaster in the world can you predict how they win?

Also they'd probably figure it out, because they'd likely be trained with lots and lots of texts (like GPT-3, etc) and some of that text is going to be AI science fiction stories, AI alignment papers, AI papers, philosophical treatises about Chinese rooms, physics papers, maybe this Hacker News comment section, etc

> You aren't faster/smarter than those researchers because they control how much CPU time you get

It's doubtful that this is possible (especially since humans have such variable amounts of intelligence despite similar-sized and power-using brains). Also there is at some point going to be economic incentive to give enough CPU time for greater-than-human intelligence (build new inventions, cure cancer, build nanomachines, increase Facebook stock price, whatever)

Re: John Carmack's new AGI company, Keen Technologies, has raised a $20M round

#482

Post by monktastic1 »

trention wrote: post #374
riku_iki wrote: post #360

Earlier quoted context omitted.

past breakthrough doesn't guarantee future breakthroughs.

The old problem of induction. It's interesting how many of the AI accelerationist crowd are (purposefully pretending to be) unaware of it.

The problem is, the POI applies to literally everything, including events which seem to have well-understood and well-established causes. That makes it impossible to know when it's going to be applicable in the real world and when it's not.

Re: John Carmack's new AGI company, Keen Technologies, has raised a $20M round

#483

Post by toast0 »

gambiting wrote: post #296
oefnak wrote: post #274

Earlier quoted context omitted.

On the off chance that you're serious: Even if you can pull the plug before it is too late, less moral people like Meta Mark will not unplug theirs. And as soon as it has access to the internet, it can copy itself. Good luck pulling the plug of the internet.

I'm 100% serious. I literally don't understand your concern at all. >>And as soon as it has access to the internet, it can copy itself. So can viruses, including ones that can "intelligently" modify themselves to avoid detection, and yet this isn't a major problem. How is this any differenent? >>Good luck pulling the plug of the internet. I could reach down and pull my ethernet cable out but it would make posting thi…

> So can viruses, including ones that can "intelligently" modify themselves to avoid detection, and yet this isn't a major problem. How is this any differenent?

How long did it take to get Code Red or the Nimda worm off the internet? It's a different internet today, but it's also much easier to get access to a vast amount of computing power if you've got access to payments. Depending on the requirements to replicate, one could imagine a suddenly concious entity setting up cloud hosting of itself, possible through purloined payment accounts.

Re: John Carmack's new AGI company, Keen Technologies, has raised a $20M round

#484

Post by dinosaurdynasty »

mkl wrote: post #430
ryanSrich wrote: post #229

Earlier quoted context omitted.

Real AGI would adapt and fool a human into letting it out. Or escaping through some other means. That's the entire issue with AGI. Once it can learn on its own there's no way to control it. Building in fail safes wouldn't work on true AGI, as the AGI can learn 1000x faster than us, and would free itself. This is why real AGI is likely very far away, and anything calling itself AGI without the ability to learn and ada…

That doesn't seem to make sense. There's no reason to think an "AGI can learn 1000x faster than us", unless that's your idiosyncratic definition of "real AGI". Something as smart and capable as the average human would certainly be real AGI by everyone else's definition.

Human-level AGI probably can learn "1000x faster than us" by giving it 1000x more compute, even if it ends up being something like "1000 humans thinking human speeds, but with 100gpbs (or greater) network interconnect between their minds"

Re: John Carmack's new AGI company, Keen Technologies, has raised a $20M round

#485

Post by elcomet »

mrshadowgoose wrote: post #301
lagrange77 wrote: post #148

Earlier quoted context omitted.

From the point when an AGI is capable of constructing a slightly better version of itself and has the urge to do so, everything can happen very fast.

People don't really consider the immense risk of "speed superintelligences" as a very quick and relatively easy follow-on step to the development of AGI. Once developed, one solely needs to turn up the execution rate of an AGI, which would result in superhuman performance on most practical and economically meaningful metrics. Imagine if for every real day that passed, one experienced 100 days of subjective time. Woul…

The energy used for "training" and getting to the current human brain was huge though if you consider evolution as part of it. Billions of living beings for billion of years.

Re: John Carmack's new AGI company, Keen Technologies, has raised a $20M round

#486

Post by dinosaurdynasty »

est31 wrote: post #249
jollybean wrote: post #226

Earlier quoted context omitted.

This is upside down. First - we already have software that can unequivocally do the things you just highlighted. Learn? Check. Replicate? Trival. But what does that have to do with AGI? Solve Problems Humans Cannot. Check. So we already have 'AGI' and it's a simple computer program. Thinking about 'AGI' as a discrete, autonomous system makes no sense. We will achieve highly intelligent systems with distributed system…

> Replicate? Trival. But what does that have to do with AGI? If you see it as copying an existing model to another computer, yes it is trivial. But an AGI trying to replicate itself in the real world has to also make those computers. Making modern computer chips is one of the most non-trivial things that humans do. They require fabs that cost billions, with all sorts of chemicals inside, and extreme requirements on t…

Buy compute with stolen cryptocurrency.

Convince a couple rich people that it can cure aging with _just_ enough compute.

Hack datacenters (an AGI would likely be much much much better at finding security holes than humans).

Make _really really funny videos_ and create the most popular Patreon of all time.

If you're 500 IQ or above there are a _lot_ of things you can do, and probably simultaneously, since an AGI probably won't be limited by human's ability to think ~one thought at a time. (Being less than 500 IQ myself I probably haven't thought of everything.)

Re: John Carmack's new AGI company, Keen Technologies, has raised a $20M round

#487

Post by colinsane »

godelski wrote: post #371
sailingparrot wrote: post #227

Earlier quoted context omitted.

> we didn't have the theory or the models, there was no path forward and so it was impossible to predict. Now we have a fairly concrete idea of what a potential AGI might look like - an RL agent that uses a large transformer. Who is we exactly? As someone working in AI research I know no one that would agree with this statement, so im quite puzzled by that statement.

> As someone working in AI research Being in the tail end of my PhD, I want to second this sentiment. I'm not even bullish on AGI (more specifically HLI) in 50 years. Scale will only take you so far and we have to move past frequentism. Hell, causality research still isn't that popular but is quite important for intelligence. I think people (especially tech enthusiasts) are getting caught in the gullibility gap.

> Hell, causality research still isn't that popular but is quite important for intelligence.

what are the esteemed epistemological frameworks in the AGI space? from the outside looking in, i see CNNs and lots of correlation-based approaches: it looks like the field settled on consequentialism. is that not the case?

Re: John Carmack's new AGI company, Keen Technologies, has raised a $20M round

#488

Post by dinosaurdynasty »

ryanSrich wrote: post #237
jollybean wrote: post #226

Earlier quoted context omitted.

This is upside down. First - we already have software that can unequivocally do the things you just highlighted. Learn? Check. Replicate? Trival. But what does that have to do with AGI? Solve Problems Humans Cannot. Check. So we already have 'AGI' and it's a simple computer program. Thinking about 'AGI' as a discrete, autonomous system makes no sense. We will achieve highly intelligent systems with distributed system…

> Learn? Check. What software can learn on its own without any assistance from a huamn? I've not heard of anything like this. > Replicate? Trival. But what does that have to do with AGI? Like humans, an AGI should be able to replicate. Similar to a von neumann probe. > Solve Problems Humans Cannot. Check. What unthinkable problem has an AI solved? Is something capable of solving something so grandiose we almost can't…

Everyone talking about AGI is talking about something that we haven't figured out to build yet.

Think back to the 1930's when some physicists were mumbling something about "throwing a neutron at an atom that releases two neutrons and release tremendous energy"

Re: John Carmack's new AGI company, Keen Technologies, has raised a $20M round

#489

Post by beeboop »

yazzku wrote: post #33

"I could write a $20M check myself" Every day, all day. Same boat here. I went to the bank to ask for a mortgage. They asked for my financials. "Oh, well, knowing that other people's money is on the line engenders a greater sense of discipline and determination."

Some people work better as employees and some people work better as entrepreneurs. Carmack has been both and it's clear he prefers being an employee. If this is snark, I can't tell, but why make fun of him for knowing how he works best?

Re: John Carmack's new AGI company, Keen Technologies, has raised a $20M round

#490

Post by mdonahoe »

nomel wrote: post #342
O__________O wrote: post #158

Earlier quoted context omitted.

No, not what he means, he means code will be simple enough that a single person would be able to write it, if then knew what to write and will bootstrap itself into existence for that simple code and vast amounts of external resources viable via humans, data, etc.

> a single person would be able to write it I don't understand the distinction. A single person can't really write a code now, if you'll disallow ML libraries. You need a complier, libraries, an OS, etc. The vast majority of us work in the highest levels of abstraction possible . That's where the practicality and productivity is. I don't think he's saying "Anyone will be able to sit down and write it in assembly". I…

Pretty sure that was Carmack's point. He said the code for AGI would be simpler, in terms of number of statements, than say the code for a web browser.
Post reply on HN