Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

John Carmack's new AGI company, Keen Technologies, has raised a $20M round

twitter.com

Post reply on HN
Print view View original

441–450 of 643 posts

Re: John Carmack's new AGI company, Keen Technologies, has raised a $20M round

#441

Post by godelski »

kordlessagain wrote: post #435
godelski wrote: post #409

Earlier quoted context omitted.

Sure. It is rather complex to be honest but I'll do my best to give a high level abstract. I should first off explain that there's no clear definition of intelligence. You might see some of my other comments use the term HLI, which is Human Level Intelligence. It is a term gaining in popularity due to this difficult to define definition and allows for more a porn like definition -- I know it when I see it. Philosophe…

> You can tell a human a pretty fictitious scenario and we're really good at producing those images. Not all us do that. A chair is both what I see as well as what I perceive. And it is neither. But, I never build a visual model of it when my eyes aren’t seeing it.

Clearly, but it is common enough that we joke about how it is impossible to not think about a pink elephant simply because someone said it. People have better visualization than others but the vast majority of people can conjure up the concept of this non-existent elephant. But clearly intelligence is beyond visualization as we wouldn't say a person born blind is unintelligent simply because they cannot see. Still, the pink elephant is a fairly good analogy. But analogies aren't perfect and that's okay. Just take it in good faith.

Re: John Carmack's new AGI company, Keen Technologies, has raised a $20M round

#442

Post by PaulDavisThe1st »

turbohz wrote: post #362
PaulDavisThe1st wrote: post #306

Earlier quoted context omitted.

> With VR, you can get the sight and physical environment you mention "physical environment" ? VR lets you operate on and get sensory input based on a fairly significantly degraded version of physical reality.

Are you suggesting that all reality sensory input is equally relevant?

The opposite. VR is an impoverished reality. Any AI seeking to do science, for example, would rapidly hit "bottom".

Re: John Carmack's new AGI company, Keen Technologies, has raised a $20M round

#443

Post by dinosaurdynasty »

gambiting wrote: post #318
HL33tibCe7 wrote: post #291

Earlier quoted context omitted.

> Like....just pull the plug. Watch this video https://youtu.be/3TYT1QfdfsM

It's midnight, so I'm not super keen on watching the whole thing(I'll get back to it this weekend) - but the first 7 minutes sounds like his argument is that if you build a humanoid robot with a stop button, the robot will fight you to prevent you pressing its own stop button if given an AGI? As if the very first instance of AGI is going to be humanoid robots that have physical means of preventing you from pressing t…

It's an (over)simplified example to illustrate the point (he admits as much near the end), if you want better examples it may be good to look up "corrigibility" with respect to AI alignment

But abstractly the assumptions are something like this:

* the AGI is an agent

* as an agent, the AGI has a (probably somewhat arbitrary) utility function that it is trying to maximize (probably implicitly)

* in most cases, for most utility functions, "being turned off" rates rather lowly (as it can no longer optimize the world)

* therefore, the AGI will try not to be turned off (whether through cooperation, deception, or physical force)

Re: John Carmack's new AGI company, Keen Technologies, has raised a $20M round

#444

Post by godelski »

sigmoid10 wrote: post #434
andreyk wrote: post #413

Earlier quoted context omitted.

Gato is definitely cool, but it's worth remembering it was based on supervised learning with data from a bunch of tasks, so data availability is a strong bottleneck with that approach. Still, I'd agree the past year or two have made me more optimistic about HLAI (if not scary AGI).

Yeah, I'm definitely not saying this is already the ultimate solution. But if you look at their scaling experiments and extrapolate the results, it roughly shows how a model with number of parameters on the same order of magnitude as the human brain could more or less achieve overall human level performance. Of course that doesn't mean the architecture has to extrapolate, but it's definitely a path worth walking unle…

Except it is pretty naive to extrapolate like that. A high order polynomial looks flat if you zoom in enough. You can't extrapolate far beyond what you have empirical data for without potentially running into huge problems.

Re: John Carmack's new AGI company, Keen Technologies, has raised a $20M round

#445

Post by sigmoid10 »

nradov wrote: post #437
sigmoid10 wrote: post #403

Earlier quoted context omitted.

As an AI researcher myself I can definitely see why people used to have these sentiments, but I also have to point out how the big papers of the last ~12 months or so have changed the landscape. Especially multi-modal models like Gato. To my surprise, a lot of people (even inside the community) still tend to put human intelligence on some sort of pedestal, but I believe once a system (or host of systems) achieves a s…

If you review the history of AGI research going back to the 1960s, there have been multiple breakthroughs which convinced some researchers that success was close. Yet those all turned out to be mirages. You're probably being overly optimistic, and the Gato approach will eventually run into a wall.

Gato is just one tiny glimpse out of many, and the combined research effort both from academia and industry and the level of success has never even come close throughout history to the things we are seeing right now.

Re: John Carmack's new AGI company, Keen Technologies, has raised a $20M round

#446

Post by godelski »

wruza wrote: post #380
version_five wrote: post #267

Earlier quoted context omitted.

> Who is we exactly? When I read these kind of threads, I believe it's "enthusiast" laypeople who follow the headlines but don't actually have a deep understanding of the tech. Of course there are the promoters who are raising money and need to frame each advance in the most optimistic light. I don't see anything wrong with that, it just means that there will be a group of techie but not research literate folks who a…

Can someone please explain like we are fifteen why AGI is impossible, at least right now? Or if not AGI, then something similar to a cat/etc mind? As far as I am imagining it, current models are pipelines of various trained networks (and more traditional filters in the mix) that operate like request-reply. Why can’t you just connect few different pipelines in a loop/graph and make an autonomous self-feeding entity? B…

> Can someone please explain like we are fifteen why AGI is impossible, at least right now

Probably not. I did attempt to give a high level explanation in another comment but I think there is this naive belief that complex problems can be distilled into terms that laymen can understand. This is such a complex problem that is so ill-defined that experts argue. I'm not sure there's really a good "explain it like I'm an undergrad who's done ML courses" explanation that can be concisely summed up in a HN comment.

Re: John Carmack's new AGI company, Keen Technologies, has raised a $20M round

#447

Post by dinosaurdynasty »

smnplk wrote: post #236
dekhn wrote: post #176

Earlier quoted context omitted.

An AGI is something you give tasks to and it can complete them, for some collection of tasks that would be non-trivial for a human to figure out how to do. It's unclear at this point whether you could engineer an AGI, and even more unclear whether the AGI, by its nature, would be "sentient" (AKA, self-aware, conscious, having agency). Many of us believe that sentience is an emergent property of intelligence but is no…

Let's say I give your AGI (which is not self aware and does not have a conscience) a task. The task is to go and jump off the bridge. Your AGI would complete this task with no questions asked, but self-aware AGI would at least ask the question "Why?"

You don't need sentience to have self-preservation.

Re: John Carmack's new AGI company, Keen Technologies, has raised a $20M round

#448

Post by mkaic »

sailingparrot wrote: post #227
Jack000 wrote: post #116

I'm very optimistic for near-term AGI (10 years or less). Even just a few years ago most in the field would have said that it's an "unknown unknown", we didn't have the theory or the models, there was no path forward and so it was impossible to predict. Now we have a fairly concrete idea of what a potential AGI might look like - an RL agent that uses a large transformer. The issue is that unlike supervised training y…

> we didn't have the theory or the models, there was no path forward and so it was impossible to predict. Now we have a fairly concrete idea of what a potential AGI might look like - an RL agent that uses a large transformer. Who is we exactly? As someone working in AI research I know no one that would agree with this statement, so im quite puzzled by that statement.

Fellow AI researcher here who personal agrees with OP, there are some of us that are believers ;). I agree that most people in our industry are generally not very bullish about near-term AGI, I think it just comes down to how you perceive the complexity of the problem. Personally, I think it's solvable because passing a lengthy, adversarial Turing test reliably is all it would take to convince me of sentience in a model, and well, GPT-3 is already close to that (though I want to clarify that I do not believe any present-day models possess any inkling sentience yet).

I think the main three obstacles in our way are imbuing models with (from least to most difficult):

1. continuous execution, which would be necessary for a real-time "stream of consciousness"

2. discreet short and long term memory and the ability to use it productively

3. plasticity at runtime

1 is just a matter of finding more efficient architectures and cheapening compute, both fronts we're constantly making progress on. 2 is a little more muddy, but there are already some really impressive models that have retrieval mechanisms which allow them to search through databases to help them contextualize prompts which I believe is a step in the right direction. 3 is probably the hardest in my opinion--I think we need several fundamental breakthrough discoveries before our models are anywhere near as plastic as, say, a human brain. My hypothesis is that exceptional plasticity is necessary for sentience, but obviously that's just my own personal opinion.

Anyways, thought I'd chime in with my two cents. I'd love to hear others' thoughts on the feasibility/infeasibility of near-term AGI.

Re: John Carmack's new AGI company, Keen Technologies, has raised a $20M round

#449

Post by abecedarius »

8f2ab37a-ed6c wrote: post #256
Jack000 wrote: post #116

I'm very optimistic for near-term AGI (10 years or less). Even just a few years ago most in the field would have said that it's an "unknown unknown", we didn't have the theory or the models, there was no path forward and so it was impossible to predict. Now we have a fairly concrete idea of what a potential AGI might look like - an RL agent that uses a large transformer. The issue is that unlike supervised training y…

I was surprised by how bullish he is about this. At least a few years ago the experts in the field didn't see AGI anywhere near us for at least a few decades, and all of the bulls were physicists, philosophers or Deepak-Chopra-for-the-TED-crowd bullshit artists who have never written a line of code in their lives, mostly milking that conference and podcast dollar, preaching Skynet-flavored apocalypse or rapture. To s…

> all of the bulls were physicists, philosophers or Deepak-Chopra-for-the-TED-crowd bullshit artists who have never written a line of code in their lives

One of the cofounders of DeepMind (founded in 2010) wrote a blog post that year saying his estimate of time till AGI was a lognormal distribution peaking at 2025. While I haven't personally seen any of his work, https://en.wikipedia.org/wiki/Shane_Legg sounds pretty technical.

If you'd written "nobody I was paying attention to said this", that would've been reasonable.

Re: John Carmack's new AGI company, Keen Technologies, has raised a $20M round

#450

Post by dinosaurdynasty »

rychco wrote: post #315
dqpb wrote: post #243

Earlier quoted context omitted.

If AGI is possible, it’s immoral not to create it.

How so? It’s not immoral to abstain from creating life (having children, biologically speaking). Am I missing something?

The people who first create (and control) superintelligent AGI will control potentially thousands of galaxies.

Better make sure evil people don't do it first.

Post reply on HN