Yann LeCun: AI one-percenters seizing power forever is real doomsday scenario • Hacker News Board

Re: Yann LeCun: AI one-percenters seizing power forever is real doomsday scenario

#871

Post by mayd » Sun, Nov 05, 2023, 12:39 PM UTC

4bpp wrote: post #501

ImPostingOnHN wrote: post #469

Earlier quoted context omitted.

> No, I'm proposing replacing a high risk scenario (I disagree with Hinton here) I disagree with you here, I think the risk is low, not high. > it's about getting governments to SIGN AN INTERNATIONAL TREATY. It is NOT about having US (or anyone else) policing the rest of the world We haven't been able to do that for climate change. When we do, then I'll be convinced enough that it would be feasible for AI. Until then…

To continue contributing to climate change takes very little: you need some guy willing to spin a stick for long enough to start a fire, or to feed and protect a flock of cows. To continue contributing to AI, you need to maintain a global multi-billion supply chain with cutting edge technology that might have a four-digit bus factor. The mechanisms that advance climate change are also grandfathered in to the point th…

the collective noun for cows is herd - not flock.

https://en.wikipedia.org/wiki/List_of_animal_names>

Re: Yann LeCun: AI one-percenters seizing power forever is real doomsday scenario

#872

Post by gooseus » Mon, Nov 06, 2023, 12:33 PM UTC

homefree wrote: post #864

gooseus wrote: post #860

Earlier quoted context omitted.

> Your latter point about AGI exploring the universe makes a lot of implicit assumptions about its reasoning. Absolutely, that's actually kind of my point... Anyone who tries to predict how some AGI will behave will be making a lot of implicit assumptions about how that AGI will see the world. This is why I said: > Personally, I think the way these people talk about what a potential AGI would do reveals a lot about m…

It’d be a longer conversation that’s hard to do via HN comments, but I think the main divide is I get the impression you’re giving the AGI implicit human-like reasoning, but the idea behind the orthogonality thesis or alignment generally is that you don’t get these things for free. It’s not that humans hate ants or apes, it’s that we pursue goals without thinking too hard about them. A house being built may destroy a…

Yes, I do assume that any AGI that is "a lot smarter than us" such that it is "game over" will have to possess human-like reasoning that would also allow it to adjust its own goals, otherwise it's going to be restricted in a way that makes it less capable than us.

It seems to me that you want to have it both ways... a machine that is so smart and strategic that there is no way any human intelligence could ever outsmart or outplay it; but also so narrow and limited in how its goals are defined that it is incapable of adjusting its own course of action.

I'll be honest, I don't understand how anyone who builds real machines in real life can seriously consider a machine built with such a narrowly defined goal (produce paper-clips) that also possesses the kind of capabilities you're imagining as a side effect (a lot smarter than us, game over).

I find a lot of these AI concepts lack a rigor that is commonplace in normal computer science... for instance, you can show that one problem can be reduced to another problem which has been proven to have certain limits, therefore the first problem can not break those limits without breaking the proof [comparison sorting will always be O(nlogn)].

When it comes to the "alignment problem" as it pertains to AI that have human-level capabilities, it seems to me that you have a similar situation... doesn't this problem just reduce to the same ethical, moral, and philosophical issues we have in aligning human intelligence with some cultural or civic ideal for behavior? Isn't the "alignment problem" the same problem as "how do you raise a good citizen"?

Re: Yann LeCun: AI one-percenters seizing power forever is real doomsday scenario

#873

Post by homefree » Mon, Nov 06, 2023, 4:11 PM UTC

gooseus wrote: post #872

homefree wrote: post #864

Earlier quoted context omitted.

It’d be a longer conversation that’s hard to do via HN comments, but I think the main divide is I get the impression you’re giving the AGI implicit human-like reasoning, but the idea behind the orthogonality thesis or alignment generally is that you don’t get these things for free. It’s not that humans hate ants or apes, it’s that we pursue goals without thinking too hard about them. A house being built may destroy a…

Yes, I do assume that any AGI that is "a lot smarter than us" such that it is "game over" will have to possess human-like reasoning that would also allow it to adjust its own goals, otherwise it's going to be restricted in a way that makes it less capable than us. It seems to me that you want to have it both ways... a machine that is so smart and strategic that there is no way any human intelligence could ever outsma…

There's some overlap with 'how you raise a good citizen', but we're also aligned somewhat already by our shared evolutionary history (and even then there are still major problems so if anything that suggests a lot of caution).

> "I'll be honest, I don't understand how anyone who builds real machines in real life can seriously consider a machine built with such a narrowly defined goal (produce paper-clips) that also possesses the kind of capabilities you're imagining as a side effect (a lot smarter than us, game over)."

The specific bit of this was that paperclips just happen to satisfy its reward function really well (vs. it being narrowly constrained to make paperclips intentionally). The example is meant to be about how you can get an unanticipated result when you don't know what the intelligence is solving for. A human looks at that and thinks 'that's a dumb goal', but the point of the alignment problem is that that isn't some universal truth, a different intelligence would not get that shared understanding for free. Humans have a lot of baked in baseline wants and even then like you said we're ourselves not perfectly aligned.

> "It seems to me that you want to have it both ways... a machine that is so smart and strategic that there is no way any human intelligence could ever outsmart or outplay it"

Imagine a chimp or ant trying to outsmart or out play a human - and this difference is smaller than the difference we're talking about here. To simplify it further imagine a regular human brain unconstrained by biological energy limitations scaled up to run a billion times faster. It thinks faster than you - there isn't a competition there you're essentially standing still.