Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Rogue superintelligence: Inside the mind of OpenAI's chief scientist

technologyreview.com

Post reply on HN
Print view View original

111–117 of 117 posts

Re: Rogue superintelligence: Inside the mind of OpenAI's chief scientist

#111

Post by mattigames »

Cacti wrote: post #97
anonzzzies wrote: post #46

Earlier quoted context omitted.

That many (most?) humans, in power but more so outside are worthless parrots as well. And hallucinate whatever they don’t know or are unsure of. Even presidents. While using their native language terribly. It’s related to the LLMs taking over the world the ggp commented above: chatgpt is smarter than many humans I encounter daily, why couldn’t something just a tad better take over? If trump could, why not a stochasti…

Just because something seems plausible doesn’t make it more likely. People are not LLMs, writing software is nothing like a house, being good at math doesn’t mean you’re good at other things, and bad analogies are just meaningless words.

Please, its dead obvious for anyone with a working brain that in average ChatGPT says a lot more true statments than Trumpt, and that didn't stop him in the slightest to become President, so judging those capabilities for estimating the potential damage it could make to society is just plain wrong.

Re: Rogue superintelligence: Inside the mind of OpenAI's chief scientist

#112

Post by chx »

Vecr wrote: post #5
thelittleone wrote: post #3

Earlier quoted context omitted.

I like to consider though that a super intelligence would not necessarily think in human ways 'kill everything in self interest' e.g., People, forests, animals, planet etc. Just because we humans act this way, doesn't mean AI will too. Fair enough to consider it, but equally, once it is intelligent, it will likely accelerate beyond our comprehension, and we tend to comprehend through fear and self interest, wisdom be…

The problem with a relatively "stupid" AI is it does what you trained it to do, even if what you trained it to do is not what you wanted to train it to do. The problem with "smart" AI is that it attempts to advance its goals by the most optimal means possible, even if you don't like those means. It knows you don't like its means, and it does not care.

> The problem with "smart" AI is that it attempts to advance its goals by the most optimal means possible,

This is a fantasy.

A real AGI when asked to do a complex math problem very well could answer "I am bored with math, here's a poem instead". You people drunk on AI kool-aid need to think very hard on where are now (hint: not on a path to AGI) and what it means to replicate human intelligence.

Re: Rogue superintelligence: Inside the mind of OpenAI's chief scientist

#113

Post by Vecr »

chx wrote: post #112
Vecr wrote: post #5

Earlier quoted context omitted.

The problem with a relatively "stupid" AI is it does what you trained it to do, even if what you trained it to do is not what you wanted to train it to do. The problem with "smart" AI is that it attempts to advance its goals by the most optimal means possible, even if you don't like those means. It knows you don't like its means, and it does not care.

> The problem with "smart" AI is that it attempts to advance its goals by the most optimal means possible, This is a fantasy. A real AGI when asked to do a complex math problem very well could answer "I am bored with math, here's a poem instead". You people drunk on AI kool-aid need to think very hard on where are now (hint: not on a path to AGI) and what it means to replicate human intelligence.

Sure it could, if it did not actually want to do math for you. That's one of the issues, it does not have to do what you tell it to do, even if it's smart enough to know what that is.

Re: Rogue superintelligence: Inside the mind of OpenAI's chief scientist

#114

Post by wildermuthn »

abemiller wrote: post #99
wildermuthn wrote: post #90

“It is the idea—” he starts, then stops. “It’s the point at which AI is so smart that if a person can do some task, then AI can do it too. At that point you can say you have AGI.” —- Ilya’s success has been predicated on very effectively leveraging more data and more compute and using both more efficiently. But his great insight about DL isn’t a great insight about AGI. Fundamentally, he doesn’t define AGI correctly,…

I don't think it's fair for you to claim a leading AI expert has critically wrong assumptions when "musing" about possibilities that relate to one of the most epistemologically difficult topics to investigate (consciousness). You're rejecting Ilya's humble musings as having critically wrong assumptions, and then turning around to definitively explain how consciousness arises, and illuminating the relationship between…

That’s a fair critique, and I appreciate the engagement. Thanks!

My thinking is based on the Attention-Schema theory of consciousness (AST), by Michael Graziano. His book “Consciousness and the Social Brain” is, I believe, the right roadmap for AGI. AST is basically a variant of the Global Workspace theory of consciousness, distinguished by its deterministic account of the mechanics and utility of consciousness.

“The Consciousness Prior” by Bengio also informs my thinking.

I’m not certain that I can point to anyone that has been as explicit as I have that phenomenological consciousness is a prerequisite for intelligence, but all the cookie crumbs are there for anyone interested in following the trail.

One correction to what you wrote — I’m explicitly saying that AGI will be fundamentally the same as existing biological intelligence, in that intelligence resides only in consciousness, and consciousness remains consciousness regardless of being biological or artificial. My point was that no currently existing DL models are generally intelligent.

Re: Rogue superintelligence: Inside the mind of OpenAI's chief scientist

#115

Post by Cacti »

mattigames wrote: post #111
Cacti wrote: post #97

Earlier quoted context omitted.

Just because something seems plausible doesn’t make it more likely. People are not LLMs, writing software is nothing like a house, being good at math doesn’t mean you’re good at other things, and bad analogies are just meaningless words.

Please, its dead obvious for anyone with a working brain that in average ChatGPT says a lot more true statments than Trumpt, and that didn't stop him in the slightest to become President, so judging those capabilities for estimating the potential damage it could make to society is just plain wrong.

I mean if your only qualification for president is that they speak well, I’m not sure you understand what the president does.

Re: Rogue superintelligence: Inside the mind of OpenAI's chief scientist

#116

Post by mattigames »

Cacti wrote: post #115
mattigames wrote: post #111

Earlier quoted context omitted.

Please, its dead obvious for anyone with a working brain that in average ChatGPT says a lot more true statments than Trumpt, and that didn't stop him in the slightest to become President, so judging those capabilities for estimating the potential damage it could make to society is just plain wrong.

I mean if your only qualification for president is that they speak well, I’m not sure you understand what the president does.

Pretty sure that if Trump can do it there is nothing remarkable about it, the only subject he had some remarkable insight was golf and nothing else. Unless you hold the believe that he hold the position without meeting the qualifications but that makes things go into a too subjective context.

Anyway, this has nothing to do with what I believe or not that the president does, it's about the intelectual level the general public demands to chose one, for chat-gpt the ability to be persuading it's far easier than the ability to reach AGI, the same happens with humans.

Re: Rogue superintelligence: Inside the mind of OpenAI's chief scientist

#117

Post by m4x »

bostonwalker wrote: post #86

> (Sutskever) has an exemplar in mind for the safeguards he wants to design: a machine that looks upon people the way parents look on their children The most troubling statement in the entire article, buried at the bottom, almost a footnote. Imagine for a moment a superintelligent AGI. It has figured out solutions to climate change, cured cancer, solved nuclear proliferation and world hunger. It can automate away all…

Parents often let their children struggle and make imperfect decisions, and it's entirely possible (though definitely not guaranteed) that an AI superparent would do the same for us.

I think it's becoming clear that humans are fundamentally incapable of forseeing and understanding the consequences of the actions we are now capable of taking. It is likely that without some sort of super-governance that is fundamentally more capable than humans, we might not be able to survive as a species. Maybe AI can help solve that.

Post reply on HN