Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

John Carmack’s ‘Different Path’ to Artificial General Intelligence

dallasinnovates.com

Post reply on HN
Print view View original

151–160 of 506 posts

Re: John Carmack’s ‘Different Path’ to Artificial General Intelligence

#151

Post by hooby »

loveparade wrote: post #26

I'm always taken aback by this fascination and heroification with John Carmack (or similar figures) on HN. I love Doom and Quake too, but that doesn't qualify him any more than any other random senior engineer to work on something totally different. From his interviews it looks like he understands little about the technical details of ML, or about as much as anyone can learn in a few months, and is just banking on PR…

I fully understand (and share) your aversion to personality cults. And it's definitely possible that there exists some actual personality cult/fandom around Carmack.

BUT - I strongly believe, that he has earned quite some respect during his career.

And - importantly in this case - he is well known for NOT blowing things out of proportion, indulge in wishful thinking, hyping up unrealistic expectations or jumping to premature conclusions. He usually knows what he's talking about.

This is not people blindly believing everything he says - but more a case of his statements holding up really well under critical inspection most of the time.

Re: John Carmack’s ‘Different Path’ to Artificial General Intelligence

#152

Post by nwoli »

loveparade wrote: post #26

I'm always taken aback by this fascination and heroification with John Carmack (or similar figures) on HN. I love Doom and Quake too, but that doesn't qualify him any more than any other random senior engineer to work on something totally different. From his interviews it looks like he understands little about the technical details of ML, or about as much as anyone can learn in a few months, and is just banking on PR…

There isn’t even that much to learn about AI at this stage. Gradient descent and a lot of model variations. I really doubt carmack hasn’t been able to get deep knowledge of this, most smart curious undergraduates are already up to date with it.

Re: John Carmack’s ‘Different Path’ to Artificial General Intelligence

#153

Post by nwoli »

jasmer wrote: post #115

"So I asked Ilya Sutskever, OpenAI’s chief scientist, for a reading list. He gave me a list of like 40 research papers and said, ‘If you really learn all of these, you’ll know 90% of what matters today.’ And I did. I plowed through all those things and it all started sorting out in my head." Wow, that's going to be one of the more glib things I've read in a while. This is a bit of a Tom Cruise moment. I mean, I get i…

He got those papers 4 years ago and have implemented them in the meantime. That doesn’t sound infeasible at all

Re: John Carmack’s ‘Different Path’ to Artificial General Intelligence

#154

Post by GuB-42 »

arcturus17 wrote: post #9
Macuyiko wrote: post #2

Fun interview but barely any meat for those in the field. Just very general questions and answers, saying that the road is murky, but nothing about e.g. if Transformers / attention are the way to go forward, multi-modal models, reinforcement learning + self-supervised learning.

I do not question for a second that Carmack is a computing genius, but “murky” is an understatement. There is nothing of technical, scientific or business value in this interview. It’s just generic words strung together.

Plot twist, John Carmack's AI is answering the interview.

Re: John Carmack’s ‘Different Path’ to Artificial General Intelligence

#155

Post by int_19h »

yreg wrote: post #124
fmajid wrote: post #101

Earlier quoted context omitted.

And ethics. Enslaving intelligent, self-aware AIs is no better than the old transatlantic slave trade. Terminating an AI without its consent is no different than murder. Putting a compulsion in an AI so it craves doing whatever it is you want it to do is no different from what a cult does or drugging someone for exploitation.

That depends on whether you believe it is sentient.

Indeed, and similar arguments have been made in defense of slavery as well, back when it was socially acceptable to defend it.

Re: John Carmack’s ‘Different Path’ to Artificial General Intelligence

#156

Post by rl3 »

perryizgr8 wrote: post #149
rl3 wrote: post #137

Carmack makes four points—some of which I agree with—that are unfortunately disturbing when taken in totality: a) We’ll eventually have universal remote workers that are cloud-deployable. b) That we’ll have something on the level of a toddler first, at which point we can deploy an army of engineers, developmental psychologists, and scientists to study it. c) The source code for AGI will be a few tens of thousands of…

> what’s stopping a random person from intentionally or unintentionally ending human civilization? If AGI is really an intelligent agent, our random supervillain would have to do what any real-life villain would need to do: convince his minions of his plan using persuasion or money. I don't think the overall danger would increase at all. If the AGI is something less than a human, then what are you worried about?

Intelligent agents need not mirror human psychology or emotions. The creation of something extremely powerful that doesn’t think like we do is a very real possibility.

In human beings, what we consider normal is actually a very fragile and delicate balance. Changes in chemical operation of the brain have outsized effects on emotion, perception, and even sanity.

With A[G]I, I think it’s helpful to think of code or architectural changes as analogous in some respects to chemical changes. In other words, if all it takes to spin up an AGI is 30,000 lines of code, then I bet rendering the thing psychotic intentionally or unintentionally would just take a few lines somewhere.

Agents capable of recursive self-improvement at silicon speeds that can easily be rendered psychotic or malevolent even by accident, is not something that I think the public should have access to, let alone anyone.

If it’s less than human, it can still have superhuman capability. The paperclip maximizer is the classic example of a tool AI run amok. Whether it counts as AGI is up for debate. Is tool AI a path to AGI? I think it is.

Re: John Carmack’s ‘Different Path’ to Artificial General Intelligence

#157

Post by int_19h »

Yoric wrote: post #120
jasmer wrote: post #115

"So I asked Ilya Sutskever, OpenAI’s chief scientist, for a reading list. He gave me a list of like 40 research papers and said, ‘If you really learn all of these, you’ll know 90% of what matters today.’ And I did. I plowed through all those things and it all started sorting out in my head." Wow, that's going to be one of the more glib things I've read in a while. This is a bit of a Tom Cruise moment. I mean, I get i…

> Defeating the Rust borrow checker takes longer than that! Fighting the Rust borrow checker to hate leads and hate to the Dark Side leads. Harmony with the Rust borrow checker is what you must achieve, padawan.

Or, come to the Dark Side. We have comptime!

Re: John Carmack’s ‘Different Path’ to Artificial General Intelligence

#158

Post by foldr »

rplst8 wrote: post #150
foldr wrote: post #142

Earlier quoted context omitted.

I'm sure he can, but has Bill gates done actual groundbreaking research into vaccines, or just made intelligent decisions about what kind of research to fund? I think the latter.

Sometimes real progress is made only by finding the proper place to "push" on an otherwise stubborn problem that won't budge.

Sure, I guess. My point is that Bill Gates is going something quite different from John Carmack (funding research outside his area of expertise, rather than doing the research himself), so that the comparison is not very illuminating.

Re: John Carmack’s ‘Different Path’ to Artificial General Intelligence

#159

Post by yreg »

int_19h wrote: post #155
yreg wrote: post #124

Earlier quoted context omitted.

That depends on whether you believe it is sentient.

Indeed, and similar arguments have been made in defense of slavery as well, back when it was socially acceptable to defend it.

People believed human slaves are not sentient?

Re: John Carmack’s ‘Different Path’ to Artificial General Intelligence

#160

Post by golergka »

fmajid wrote: post #101
thom wrote: post #93

I don't want to be an utter bore, but 60% chance of AGI by 2030, taken at face value, would have prompted me as an interviewer to more urgently ask deeper questions about safety. For a long time I have tried to comfort myself that brains are complicated, AI is hard, and maybe there's just a complexity barrier that is going to prevent any sort of hard takeoff scenario. Maybe we can have subhuman AGI that can do lots o…

And ethics. Enslaving intelligent, self-aware AIs is no better than the old transatlantic slave trade. Terminating an AI without its consent is no different than murder. Putting a compulsion in an AI so it craves doing whatever it is you want it to do is no different from what a cult does or drugging someone for exploitation.

Humans are hard-wired to universally like or dislike certain things — we suffer when we're hungry and we're afraid of death. Most of human morality is based on these reactions.

But AI can be configured to desire anything you want, you just have to pick a fitting reward function. So, is turning off the AI that is expecting to be turned off and desires it an amoral thing?

Post reply on HN