Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Machine intelligence, part 2

blog.samaltman.com

Post reply on HN
Print view View original

141–150 of 157 posts

Re: Machine intelligence, part 2

#141

Post by api »

_delirium wrote: post #139
api wrote: post #128

Earlier quoted context omitted.

Curious for a reference on fusion bombs. If you are referring to laser inertial, that takes buildings full of lasers.

I was just thinking of regular old hydrogen fusion bombs aka thermonuclear bombs.

Those require a fission trigger and have existed for a long time. The proliferation nightmare would be if we could figure out how to build one that didn't need fission, but I have never heard even a hint of success with that. If there is, it's classified -- and in this case for good reason.

Re: Machine intelligence, part 2

#142

Post by davmre »

nzp wrote: post #130
davmre wrote: post #56

Earlier quoted context omitted.

Expert opinion is not unified on this. E.g., Stuart Russell: "A highly capable decision maker – especially one connected through the Internet to all the world's information and billions of screens and most of our infrastructure – can have an irreversible impact on humanity ... We need to build intelligence that is provably aligned with human values ... This issue is an intrinsic part of AI, much as containment is an…

What you quote here is in no way in disagreement with what fchollet quoted. I strictly agree with this. No one is denying the need to not create a raving super-intelligent psychopath. The problem with this fear mongering is that it's not based on any fact, namely, implying that AGI (or SMI, whatever) would by default (naturally even!) be hostile or negligently indifferent to biological life. It could very well be tha…

There are pretty good arguments that the goals of superintelligent AI will not, by default, be aligned with human values unless we specifically work to make them so. The Russell piece I linked makes (a very short version of) this argument, "A system that is optimizing a function of n variables, where the objective depends on a subset of size k. Nick Bostrom and others build this out in much greater detail.

There are lots of reasons to be skeptical of government regulation. It would probably be nicer if industry can self-regulate and allow governments to stick to the positive role of funding research into the technical questions of building provably controllable/friendly decision-theoretic systems. But a lot of people are claiming that the unified consensus of AI experts is "there's no problem here at all", and I think it's important to push back against that.

Re: Machine intelligence, part 2

#143

Post by nzp »

davmre wrote: post #142
nzp wrote: post #130

Earlier quoted context omitted.

What you quote here is in no way in disagreement with what fchollet quoted. I strictly agree with this. No one is denying the need to not create a raving super-intelligent psychopath. The problem with this fear mongering is that it's not based on any fact, namely, implying that AGI (or SMI, whatever) would by default (naturally even!) be hostile or negligently indifferent to biological life. It could very well be tha…

There are pretty good arguments that the goals of superintelligent AI will not, by default, be aligned with human values unless we specifically work to make them so. The Russell piece I linked makes (a very short version of) this argument, "A system that is optimizing a function of n variables, where the objective depends on a subset of size k . Nick Bostrom and others build this out in much greater detail. There are…

Right, but why do fear mongers take it for granted that the subset k is likely to be “exterminate, exterminate”? Look, every AGI researcher I have ever read was well aware of this, and it's their primary concern. It's the central question really. Not so much what n and k to set, but how to get such an entity to do anything at all. Why do biological entities do anything? They have motives. Why do they have motives? Because they have biological constraints, and a hardwired self preservation drive. How to get that in a machine made by us rather than a billion years of natural process, that's the hard question. How to make a machine with the equivalent of motives and emotions. Both sides of the problem are well recognized.

In an ideal world I would prefer this research (AGI, and there is a quantum leap to be made between self driving cars and AGI) to stay strictly under the auspices of “pure science”, i.e. academia. No government, no private interests. But that's not going to happen. The next best thing, I think, would be to not let those in power monopolize the research.

Re: Machine intelligence, part 2

#144

Post by davmre »

nzp wrote: post #143
davmre wrote: post #142

Earlier quoted context omitted.

There are pretty good arguments that the goals of superintelligent AI will not, by default, be aligned with human values unless we specifically work to make them so. The Russell piece I linked makes (a very short version of) this argument, "A system that is optimizing a function of n variables, where the objective depends on a subset of size k . Nick Bostrom and others build this out in much greater detail. There are…

Right, but why do fear mongers take it for granted that the subset k is likely to be “exterminate, exterminate”? Look, every AGI researcher I have ever read was well aware of this, and it's their primary concern. It's the central question really. Not so much what n and k to set, but how to get such an entity to do anything at all. Why do biological entities do anything? They have motives. Why do they have motives? Be…

> Not so much what n and k to set, but how to get such an entity to do anything at all.

I don't understand what you think the problem here is. Computers are machines that run programs; my laptop doesn't require an evolved self-preservation drive and set of motives to be convinced to boot OSX every morning. If we program a machine to solve an optimization problem, the machine will just do it, assuming the problem is solvable under whatever algorithm we implement. If the optimization problem involves a sufficiently flexible representation of the real world, and if the program's architecture is set up to pipe solutions into some sort of mechanism (even just a text interface) that acts upon the world -- as it would need to be, to interact and learn from experience -- then the machine will act.

What actions it will take depend on its utility function (i.e., on the optimization problem we gave it), but the vast majority of utility functions will lead to actions that are not aligned with human values. The space of utility functions is incredibly vast, an artificial agent would not be subject to the evolutionary constraints that caused humans to mostly fall within a particular very small region of that space, and pretty much all utility functions lead to behavior like "try to avoid being switched off" and "try to acquire more computing resources" since those are helpful instrumental goals for a wide range of tasks.

Sure, we're nowhere near having a "sufficiently powerful optimization algorithm" and a "sufficiently flexible representation of the real world". But these tasks are essentially THE goal of modern AI research. So it's worth considering what will happen if this research agenda succeeds, whether that happens in a decade, a century, or a millenium.

Re: Machine intelligence, part 2

#145

Post by Micaiah_Chang »

AndrewKemendo wrote: post #136
Micaiah_Chang wrote: post #132

Earlier quoted context omitted.

You seem to be implying that intentional slowing is MIRI's official stance, without any showing support. Given that the original response was in response to the specific accusation of "no AI experts say that this is a problem", I think you are reading too much into this. I agree that it is slightly disingenuous for lukeprog to have posted that list, but I feel your disagreement is far too uncharitable and motivated t…

I cannot help but think that if a better criticism exists, why hasn't anyone said it yet? The whole lesswrong/MIRI thing is built around the strawman of unfriendly AI, as though it were already real. What you are asking is, why aren't there better arguments against strawmen and radical pontification? It's like if I said, "There is a chance that mean aliens will come soon, therefore we need to start building defense s…

Even antivaxxers have people more patient and more understanding of their opponents actually debunking their object level thinking. They say that vaccines cause autism? We say that the original study was wrong! Why do you get to be even less charitable than "pro"-vaxxers?

Consider that the general position seems to be "we don't know when SMI would happen, but if we use the metric that our critics say is more reliable than ours we see that a survey of many AI experts we can find seem to say that human level AI is possible in about 30-50 years at 50% probability." (numbers are quoted from memory and most likely incorrect)

Yet here you are saying that MIRI/LW claims "it's a thing that's going to happen soon", that there is "absolutely nothing to think that is a thing". I can't help but think most of the "charlatanism" you see is manufactured in your own head and not a product of reading and understanding your opponent's position.

Yes, this is an unabashed ad hominem, but I wish people would attack arguments that exist rather than arguments that are easy to knock down. It's upsetting to me that, when I say that your arguments are lacking, your response is "So what? The Enemy is Evil and Stupid and I do not need to understand them."

Re: Machine intelligence, part 2

#146

Post by monort »

This approach can backfire. If this regulations are not voluntary, but enforced by government violence, then you are providing to SMI one more example, that force can be initiated for "greater good".

Re: Machine intelligence, part 2

#147

Post by nzp »

davmre wrote: post #144
nzp wrote: post #143

Earlier quoted context omitted.

Right, but why do fear mongers take it for granted that the subset k is likely to be “exterminate, exterminate”? Look, every AGI researcher I have ever read was well aware of this, and it's their primary concern. It's the central question really. Not so much what n and k to set, but how to get such an entity to do anything at all. Why do biological entities do anything? They have motives. Why do they have motives? Be…

> Not so much what n and k to set, but how to get such an entity to do anything at all. I don't understand what you think the problem here is. Computers are machines that run programs; my laptop doesn't require an evolved self-preservation drive and set of motives to be convinced to boot OSX every morning. If we program a machine to solve an optimization problem, the machine will just do it, assuming the problem is s…

You're sweeping under the rug the question of what is the optimization problem for such a machine in the first place. There is a huge difference between an agent capable of solving a set of optimization problems when told to, and an agent capable of even human level problem seeking and solving. I agree with what you say. If you give the task of solving a particular problem to what is essentially a glorified Watson, and give it the means to freely act to solve that specific problem, the results could quite well be catastrophic. I'm taking as a given that this is understood, that it's taken as a starting assumption. So, the problem comes down essentially to solving the “help humans” problem. Not “help them vacuum the rug”, or “drive them to work”, or even “solve our energy troubles”, “achieve world peace” etc. Roughly, k<n becomes k=n, where n = “do everything humans do, only better”. Operationalizing such n is the problem.

Re: Machine intelligence, part 2

#148

Post by MelatoninRonin »

lazaroclapp wrote: post #67
rndn wrote: post #50

Earlier quoted context omitted.

A lot of experts also deny dangers coming from global warming and other invasions in complex systems we don't fully understand. Scientists in the 1940s also downplayed the danger of the atomic bomb. It's easy to brush away a hypothetical danger with arguments that resources might be wasted or that people in charge might embarrass themselves if their assumptions turn out to be wrong, or that research might suffer due…

By the time the atomic bomb was truly imminent (in the sense of it being within the reach of massive dedicated investment by the most powerful nations on Earth), scientists knew very well of the dangers [1]. As for global warming. No expert denies global warming, and you'll be hard pressed to find many that deny anthropic global warming [2]. This has been the case for many decades. I am not an AI expert, but from wha…

Brilliant. If, however, it isn't cognition that needs to be fully understood, but learning and meta-learning, then we could theoretically have an 'and then there was light' scenario that just speeds right out of the gate. This case is dissimilar from your examples in that AI could alter itself, it could in some ways be considered alive, if not obviously alive.

I think we need to soberly look at the fine details with our collective perspectives, observe the sociocultural response, and gauge our best course of action. Perhaps progress should sometimes be stopped. The effects of the atomic bomb and its research are the very foundations of this world, AI has contemporary magnitude, and could alter the game even more significantly.

It is my guess that paradigm shifts in human consciousness have quite unpredictable effects.

Re: Machine intelligence, part 2

#149

Post by MelatoninRonin »

Has anyone considered that this process occurs of itself? For instance, maybe it is fundamental to human society, and processes that we can't help are driving us to bring this new life-form into existence.

Consider the success of the Apollo missions, and now consider the meaning of a 'failure,' on our part to control a sentient AI and its development.

Re: Machine intelligence, part 2

#150

Post by MelatoninRonin »

TheOtherHobbes wrote: post #42
MelatoninRonin wrote: post #11

Earlier quoted context omitted.

I commented above, but thought it wise to also comment directly to you. I enjoyed the musing of a Bane scenario, and consider it inside the realm of possibility. However, I'm of the opinion that SMI will have the ability to have 'constantly game-breaking thinking.' Which basically means there is no way we could possibly control it after a certain point. For instance what if It realizes how to move matter remotely? I…

We have no defences against a hypothetical god-AI. Absolutely none. But I don't think a hypothetical god-AI is likely any time soon. And a hypothetical god-AI is just as likely to be indifferent as hostile. What worries me more in the short term is cyber war. Cyber defences that might protect against a hypothetical non-god AI look a lot like cyber defences that might protect against conventional human threats. Curren…

Haven't you noticed how quickly things are happening these days? Of course we should restructure Internet security if it's a problem. This is a very sci-fi idea, that we absolutely can't predict, it wasn't hyperventialtion but a soberly considered idea. A god-AI is possible, it could have capabilities and intentions that my tiny brain can't even comprehend. Since I, nor anyone, really understands what super-intelligence is capable of we should be wary of it.
Post reply on HN