Live data from Hacker News

Rogue superintelligence: Inside the mind of OpenAI's chief scientist

technologyreview.com

51–60 of 117 posts

Re: Rogue superintelligence: Inside the mind of OpenAI's chief scientist

#51
post #9

These people are too full of themselves. The physicists that invented The Bomb didn't have any special insight into the philosophical and societal implications. It takes a special kind of person to be able to invent something so big it can change the world, but it's about 0% chance that those people can control how the technology then gets used. I wish they'd focus more on the technical advances and less on trying to…

That's an overly broad generalization, Leo Szilard certainly had insight into the implications:

> "With an enduring passion for the preservation of human life and political freedom, Szilard hoped that the US government would not use nuclear weapons, but that the mere threat of such weapons would force Germany and Japan to surrender. He also worried about the long-term implications of nuclear weapons, predicting that their use by the United States would start a nuclear arms race with the USSR. He drafted the Szilárd petition advocating that the atomic bomb be demonstrated to the enemy, and used only if the enemy did not then surrender. The Interim Committee instead chose to use atomic bombs against cities over the protests of Szilard and other scientists. Afterwards, he lobbied for amendments to the Atomic Energy Act of 1946 that placed nuclear energy under civilian control."

https://en.wikipedia.org/wiki/Leo_Szilard

Re: Rogue superintelligence: Inside the mind of OpenAI's chief scientist

#52

I, for one, am glad that Ilya has the reins on OpenAI and that Sam is out of the picture. It does seem that he weighs the ethics of what is being built more heavily than Sam. I’m also hoping that OpenAI cools down on the regulatory moat they were trying to build as a thinly veiled profit seeking strategy.

It's something to think about. Even if Sam is 97% aligned and Ilya is 98.5% aligned towards the ideal mentality to have wrt AI safety, that 1.5% could be worth millions of lives in averaged risk

Re: Rogue superintelligence: Inside the mind of OpenAI's chief scientist

#53

This is a very delusional idea: "He thinks ChatGPT just might be conscious (if you squint)" It's a technology with literally no intelligence or understanding of the world of any kind. Its just statistics on data. It is as conscious as a calculator.

I often observe that those dismissing this idea tend to be less informed about current insights into human cognition, philosophy, and concepts such as the information theoretic view of consciousness, neural correlates of consciousness, the free energy principle, and predictive coding.

The human mind is "just statistics on data".

People more informed than you are taking this seriously. You should pay attention and start inquiring why that's the case.

Re: Rogue superintelligence: Inside the mind of OpenAI's chief scientist

#54
post #24

Earlier quoted context omitted.

chatGPT is very much aware that it's looking at a shadow and it deduces what's beyond the shadow by looking at millions of different permutations of the shadow. Additionally breaking the fourth wall is trivial to it. I'm not entirely sure what you mean by fourth wall but chatGPT can definitely talk about it's own existence.

It can generate text that talks about its existence. It's not conscious, it isn't talking. It's just a.co puter program that takes an input and produces an output. Stop anthropomorphizing it.

The superintelligences running our simulation probably say the same things about us.

Re: Rogue superintelligence: Inside the mind of OpenAI's chief scientist

#55
post #9

These people are too full of themselves. The physicists that invented The Bomb didn't have any special insight into the philosophical and societal implications. It takes a special kind of person to be able to invent something so big it can change the world, but it's about 0% chance that those people can control how the technology then gets used. I wish they'd focus more on the technical advances and less on trying to…

There is an even more important lesson in this in that the people who worked on the bomb thought the world would be over in short order and were totally wrong.

Feynman talked about how there was an idea of "normal" people walking around not knowing they were basically doomed and going to die in a certain nuclear holocaust in a few years.

Von Neumann thought the U.S. should launch a nuclear first strike at Moscow. Obviously, if war is inevitable then you don't have to be the father of game theory to figure out you should strike first.

It was just a year ago that literally everyone was predicting we would be in a recession right now. We can't predict that but we can predict how AGI plays out even when we haven't bothered to define a measure of what AGI even is. Even people who grew up "knowing" we would all have sentient robots by 1997. I can't think of a single prediction I have heard in my lifetime that has turned out to be true other than the government debt going up.

Re: Rogue superintelligence: Inside the mind of OpenAI's chief scientist

#56

This is a very delusional idea: "He thinks ChatGPT just might be conscious (if you squint)" It's a technology with literally no intelligence or understanding of the world of any kind. Its just statistics on data. It is as conscious as a calculator.

I very much disagree that it's not intelligent. GPT-4 is clearly intelligent, and we use the lobotomized version of it. So imagine what GPT-5 can do internally.

Conscious is hard to say, partly because we can't define it either, so it means something different for you and me.

Re: Rogue superintelligence: Inside the mind of OpenAI's chief scientist

#57

Earlier quoted context omitted.

I like to consider though that a super intelligence would not necessarily think in human ways 'kill everything in self interest' e.g., People, forests, animals, planet etc. Just because we humans act this way, doesn't mean AI will too. Fair enough to consider it, but equally, once it is intelligent, it will likely accelerate beyond our comprehension, and we tend to comprehend through fear and self interest, wisdom be…

I commented this in the depths of the Altman Fired thread: meet Altman's Basilisk. Depending on how you create an AI, it absolutely would think in human ways. Not only that, you could coax it to think in specific human ways, as just a single human establishing the axioms for the AI. Seems like the big gotcha here is that AGI, artificial general intelligence as we contextualize it around LLM sources, is not an abstrac…

[dead]

Re: Rogue superintelligence: Inside the mind of OpenAI's chief scientist

#58
post #44

I, for one, am glad that Ilya has the reins on OpenAI and that Sam is out of the picture. It does seem that he weighs the ethics of what is being built more heavily than Sam. I’m also hoping that OpenAI cools down on the regulatory moat they were trying to build as a thinly veiled profit seeking strategy.

I was speaking with a woman earlier this week who just finished her master's dissertation with a focus on Sam Altman's recent influence campaign, and she was scared of how charming he was. She obviously was trying to maintain distance for impartiality, but his interview style, and the seeming total sincerity of his communication style... It was terrifying to her in how well he could draw everyone in. She was absolute…

People want to believe that their lives will radically change for the better very soon. It's a religion. A lot of claims but not much evidence.

Re: Rogue superintelligence: Inside the mind of OpenAI's chief scientist

#59
post #9

These people are too full of themselves. The physicists that invented The Bomb didn't have any special insight into the philosophical and societal implications. It takes a special kind of person to be able to invent something so big it can change the world, but it's about 0% chance that those people can control how the technology then gets used. I wish they'd focus more on the technical advances and less on trying to…

You are talking about the people not only doing the work and creating the wealth, but doing the innovations and creating new things. Then you say these are the people who are "too full of themselves". That they should give up on how the commodity they create is used - and by your definition it is a commodity. That they should focus on the needed innovations and that's it. Well if the workers doing the work don't have…

I'm saying a bunch of millionaire silicon valley broskis are too full of themselves because they keep on and on about how what they are working on will radically transform all of society in a pinch, and only they can save us from it.

I can assure you their random dev #56 isn't taking any decisions already, this is all PR and grandstanding from the already-millionnaires leadership with their power plays about who knows best how to save all of us from AGI.

Re: Rogue superintelligence: Inside the mind of OpenAI's chief scientist

#60

Earlier quoted context omitted.

Chatgpt's consciousness would be akin to plato's cave. Even if chatgpt were more intelligent, wouldn't it be staring at different shadows? Given that chatgpt has consciousness, would it be able to break the fourth wall? There seems to be an implicit assumption that it must break that in order to prove its consciousness to us. Maybe that's how AGI will come to be, because we desire to train it that way.

chatGPT is very much aware that it's looking at a shadow and it deduces what's beyond the shadow by looking at millions of different permutations of the shadow. Additionally breaking the fourth wall is trivial to it. I'm not entirely sure what you mean by fourth wall but chatGPT can definitely talk about it's own existence.

> chatGPT is very much aware that it's looking at a shadow…

Is a mirror aware because it creates a reflection? You’re making a very hard claim here with little evidence

Post reply on HN