John Carmack's new AGI company, Keen Technologies, has raised a $20M round • Hacker News Board

Re: John Carmack's new AGI company, Keen Technologies, has raised a $20M round

#591

Post by nradov » Sat, Aug 20, 2022, 3:53 PM UTC

wruza wrote: post #380

version_five wrote: post #267

Earlier quoted context omitted.

> Who is we exactly? When I read these kind of threads, I believe it's "enthusiast" laypeople who follow the headlines but don't actually have a deep understanding of the tech. Of course there are the promoters who are raising money and need to frame each advance in the most optimistic light. I don't see anything wrong with that, it just means that there will be a group of techie but not research literate folks who a…

Can someone please explain like we are fifteen why AGI is impossible, at least right now? Or if not AGI, then something similar to a cat/etc mind? As far as I am imagining it, current models are pipelines of various trained networks (and more traditional filters in the mix) that operate like request-reply. Why can’t you just connect few different pipelines in a loop/graph and make an autonomous self-feeding entity? B…

No one can explain why AGI is impossible because you can't prove a negative. But so far there is still no clear path to a solution. We can't be confident that we're on the right track towards human-level intelligence until we can build something roughly equivalent to, let's say, a reptile brain (or pick some other similar target if you prefer).

If you have an idea for a technical approach then go ahead and build it. See what happens.

Re: John Carmack's new AGI company, Keen Technologies, has raised a $20M round

#592

Post by lambdatronics » Sat, Aug 20, 2022, 4:28 PM UTC

wcerfgba wrote: post #239

Does anyone else subscribe to the idea that AGI is impossible/unlikely without 'embodied cognition', i.e. we cannot create a human-like 'intelligence' unless it has a similar embodiment to us, able to move around a physical environment with it's own limbs, sense of touch, sight, etc. ? Any arguments against the necessity of this? I feel like any AGI developed in silico without freedom of movement will be fundamentall…

If we want the intelligence to be recognizably human-like, then yes. As to whether the AGI could be trained in a simulation vs needing to really be physically embodied: I used to believe not, but then I learned about sim2real. (Ex: [0]) Training blank-slate deep learning algorithms on real hardware is prohibitive. It requires millions of hours of experience. Those hours are really expensive on current-day hardware (also cannot be accelerated, stuck with wall clock time). But if pre-training in simulations can be effective, then I think we have a decent shot in the near-term.

Human-level and human-like are not necessarily the same though. I doubt human intelligence is as general as we like to think it is. There's probably a lot about what we consider intelligence that is domain-specific. Training AGI on unfamiliar domains could be super valuable because it would be easier to surpass human effectiveness there.

My pet conjecture, though, is this: the ability to experiment is key to developing AGI. Passively consuming data makes it much harder to establish cause and effect. We create and discard hypotheses about potential causal links based on observed coincidences (ie, small samples) all the time. Doing experiments to confirm/refute these hypotheses is much less computationally expensive than doing passive causal inference, which enables us to consider many more potential causal linkages. That allows us to lower the statistical threshold for considering a potential causal relationship, which makes it more likely for us to find the linkages that do exist. The benefit of developing accurate causal models is that they are much more compact & computationally-effecient than trying to model the entire universe with a single probability distribution.

[0] https://openai.com/blog/solving-rubiks-cube/

Re: John Carmack's new AGI company, Keen Technologies, has raised a $20M round

#593

Post by Cyph0n » Sat, Aug 20, 2022, 4:29 PM UTC

SyneRyder wrote: post #585

Cyph0n wrote: post #92

Earlier quoted context omitted.

I watched some clips from the interview, good stuff. I personally don’t like Lex’s interviewing style though, so I couldn’t watch the whole thing.

Glad I'm not the only one. I'm 90 minutes in and finding some of Lex's questions really bad. Especially "What is the best programming language?", but I'm also astonished that Lex doesn't seem to know what an x86 is, hasn't really heard of Pascal, and was incredulous that anyone could write a game in assembly language (but as Carmack points out, everyone in the 80s was writing games in assembly, BASIC just wasn't fast…

Yeah, bad questions is one part of it. Lex also tends to interject with his own observations or anecdotes that end up interrupting the guest's flow. I'm tuning in because of the guest, not the host. In a guest-oriented format like this, I feel like the goal should be to get the guest to talk. It's just a personal preference for me, so YMMV.

Re: John Carmack's new AGI company, Keen Technologies, has raised a $20M round

#594

Post by marvin » Sat, Aug 20, 2022, 4:33 PM UTC

polishdude20 wrote: post #516

HL33tibCe7 wrote: post #182

Earlier quoted context omitted.

It sounds like you haven’t really thought through AI safety in any real detail at all. Airgapping and the necessity of human input are absolutely not ways to prevent an AGI gaining access to a system. A true, superintelligent AGI could easily extort (or persuade) those humans. If you think concerns over AGI are “stupid”, you haven’t thought about it enough. It’s a massive display of ignorance. The Computerphile AI sa…

What I don't understand is what motivates this world destroying AGI? Like, it's got motives right? How does it get them? Do we program them in? If we do, is the fear that it won't stop at anything in its way to fulfill its objective? If so, what stops it from removing that objective from its motivation? If it can discover zero days to fulfill its objective, wouldn't it reason about itself and it's human-set motives a…

This is a whole field of philosophy. You can read this to get started: https://en.wikipedia.org/wiki/AI_alignment

Tl;dr: It's hard to define what humans would actually want a powerful genie to do in the first place, and if we figure that out, it's also hard to make the genie do it without getting into a terminal conflict with human wishes.

Meaning: Doing it in the first place, without going off on some fatal tangent we hadn't thought about, and also preventing it from getting side-tracked by instrumental objectives that are inherently fatally bad to us.

The nightmare scenario is that we tell it to do X, but it is actually programmed to do Y which looks superficially familiar to X. While working to do Y, it will also consume most of Earth's easily available resources and prevent humans from turning it off, since those two objectives greatly increase the probability of achieving Y.

To illustrate via the only example we currently have experience with: Humans have an instrumental objective of accumulating resources and surviving for the immediate future, because those two objectives greatly increase the likelihood that we will propagate our genes. This is a fundamental property of systems that try to achieve goals, so it's something that also needs to be navigated for superhuman intelligence.

Re: John Carmack's new AGI company, Keen Technologies, has raised a $20M round

#595

Post by nradov » Sat, Aug 20, 2022, 4:36 PM UTC

kastagg wrote: post #416

MichaelCollins wrote: post #188

Earlier quoted context omitted.

The philosophy side of the matter seems meaningless, it interrogates the meaning of language, not the capabilities of technology. When people ask "Could machines think?" the question isn't really about machines, it's about precisely what we mean by the word 'think'. Can a submarine swim? Who cares! What's important is that a submarine can do what a submarine does. Whether or not the action of a submarine fits the mea…

The people who designed the first submarines had relatively specific engineering problems and applications in mind. They could bypass any philosophical interrogation of the word "swim" because they didn't define the problem as swimming and didn't need to. "AGI" isn't like that. Nobody really knows what it means, and it's impossible to get down to brass tacks until you choose a problem definition. When philosophers po…

Then let's define the problem as passing a rigorous Turing Test. And by rigorous I mean one lasting several days conducted by a jury of 12 tenured professors drawn from multiple academic fields including philosophy, mathematics, history, computer science, neuroscience, psychology, law, etc.

Re: John Carmack's new AGI company, Keen Technologies, has raised a $20M round

#596

Post by marvin » Sat, Aug 20, 2022, 4:37 PM UTC

dqpb wrote: post #534

trention wrote: post #258

AGI will be more dangerous that nuclear weapons. People are not allowed to start a nuclear weapon company. At all. Why are people allowed to casually start an AGI company?

I wonder if somebody could have started a nuclear weapon company before nuclear weapons were proven.

Obviously. No company would practically have been able to do it, due to the investment required, but there would be no laws stopping it.

Re: John Carmack's new AGI company, Keen Technologies, has raised a $20M round

#597

Post by nradov » Sat, Aug 20, 2022, 4:38 PM UTC

dinosaurdynasty wrote: post #484

mkl wrote: post #430

Earlier quoted context omitted.

That doesn't seem to make sense. There's no reason to think an "AGI can learn 1000x faster than us", unless that's your idiosyncratic definition of "real AGI". Something as smart and capable as the average human would certainly be real AGI by everyone else's definition.

Human-level AGI probably can learn "1000x faster than us" by giving it 1000x more compute, even if it ends up being something like "1000 humans thinking human speeds, but with 100gpbs (or greater) network interconnect between their minds"

Nah. You are extrapolating from zero data points.

Re: John Carmack's new AGI company, Keen Technologies, has raised a $20M round

#598

Post by marvin » Sat, Aug 20, 2022, 4:40 PM UTC

gambiting wrote: post #375

cgrealy wrote: post #326

Earlier quoted context omitted.

Imagine a bunch of chimps capture a human. They put them in a cage surrounded by tigers and spikes and remove all weapons from the human. The human uses their phone to call for rescue. That is us trying to contain an AGI. We probably cannot even conceive of the ways it can get out of any pitiful cage we put it in. That, or it’s so dumb, it’s not worth making in the first place.

I still don't understand. That presuposes that the AGI will just materialize out of thin air, and immediately have human level intelligence. That once day you just have a bunch of dumb computers, the next day you have an AGI hell-bent on escaping at all cost. That's not going to happen - even Carmack believes so. The process to get AGI is going to take a long time, and we'll go through lots and lots of iterations of…

They will inevitably reach the point of a world-ending threat, but I'm very confident Carmack is right that this won't happen so quickly we can't see the signs of danger.

There's a lot of policy we can do to slow things down once we get near that point, which is something rarely talked about among AI safety researchers, but the fundamental existential danger of this technology is obvious.

Re: John Carmack's new AGI company, Keen Technologies, has raised a $20M round

#599

Post by drstewart » Sat, Aug 20, 2022, 4:40 PM UTC

godelski wrote: post #446

wruza wrote: post #380

Earlier quoted context omitted.

Can someone please explain like we are fifteen why AGI is impossible, at least right now? Or if not AGI, then something similar to a cat/etc mind? As far as I am imagining it, current models are pipelines of various trained networks (and more traditional filters in the mix) that operate like request-reply. Why can’t you just connect few different pipelines in a loop/graph and make an autonomous self-feeding entity? B…

> Can someone please explain like we are fifteen why AGI is impossible, at least right now Probably not. I did attempt to give a high level explanation in another comment but I think there is this naive belief that complex problems can be distilled into terms that laymen can understand. This is such a complex problem that is so ill-defined that experts argue. I'm not sure there's really a good "explain it like I'm an…

>but I think there is this naive belief that complex problems can be distilled into terms that laymen can understand

Naive people like Richard Feynman, who said if you can't explain an idea to an 8 year old, you don't understand it? Can you tell us why you think Nobel Prize winner Richard Feynman is naive?

Re: John Carmack's new AGI company, Keen Technologies, has raised a $20M round

#600

Post by dataangel » Sat, Aug 20, 2022, 4:43 PM UTC

tux1968 wrote: post #175

Trasmatta wrote: post #2

Interesting to see how he's progressed with this. When he first announced he was getting into AI it sounded almost like a semi retirement thing: something that interested him that he could do for fun and solo, without the expectation that it would go anywhere. But now he seems truly serious about it. Wonder if he's started hiring yet.

He mentioned, in his Lex Friedman interview, that accepting investor money was a way to keep himself serious and motivated. He feels an obligation to those putting their money in.

"I want other people's skin in the game instead of mine" is not necessarily virtuous.