Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Coming Technological Singularity (1993)

mindstalk.net

Post reply on HN
Print view View original

81–90 of 181 posts

Re: The Coming Technological Singularity (1993)

#81

Post by K0balt »

trescenzi wrote: post #28
samsartor wrote: post #15

> We humans have the ability to internalize the world and conduct "what if's" in our heads; we can solve many problems thousands of times faster than natural selection. I don't know for sure whether superintellegence will happen, but as for the singularity, this is the underlying assumption I have the most issue with. Smart isn't the limiting factor of progress, often it's building consensus, getting funding, waiting…

I do think one of the major weaknesses of “smart people” is they tend to think of intelligence as the key aspect of basically everything. Reality is though we have plenty of intelligence already. We know how to solve most of our problems. The challenges are much more social and our will as a society to make things happen.

The “otherness” of AI is what holds its appeal.

Imagine a scenario where instead of AI, a billion dollar pill could make one person exponentially smarter and able to communicate with thousands of people per second.

That does not have the same appeal.

This provokes me to some musings on the theme.

We imagine superintelligence to be subservient, evenly distributed, and morally benign at least.

We don’t have a lot of basis for these assumptions.

What we imagine is that a superintelligence will act as a benevolent leader; a new oracle; the new god of humanity.

We are lonely and long to be freed of our burdens by servile labor, cured of our ills by a benevolent angel, and led to the promised land by an all knowing god?

We imagine ourselves as the stewards of the planet but yearn for irrelevance in the shadow of a new and better steward.

In AI we are creating a new life form, one that will make humans obsolete and become our evolutionary legacy.

Perhaps this is the path of all technological intelligences?

Natural selection doesn’t magically stop applying to synthetic creatures, and human fitness for our environment is already plummeting with our prosperity.

As we replace labor with automation, we populate the world with our replacement, fertility rates drop, we live for the experience of living, and require yet more automation to carry the burdens we no longer deem worthy of our rarified attention.

I’m not sure any of this is objectively good, or bad. I kinda feel like it’s just the way of things, and I hope that our children, both natural and synthetic, will be better than we were.

As we prosper, will we will have still less children? Will we seek more automation, companionship in benevolent and selfless synthetic intelligence, and more insulation from toil and strife, leading to yet more automation, prosperity, and childlessness?

Synthetic intelligence will probably not have to “take over”, it will merely be filling the void we willingly abandon.

I suspect that in a thousand years, humans will be either primitive, or vanishingly rare. Or maybe just non-primitive humans will be rare, while humans returning to nature will proliferate prodigiously as we always have, assuming the environment is not too hostile to complex biological life.

Interesting times.

Re: The Coming Technological Singularity (1993)

#82

Post by ben_w »

epcoa wrote: post #79
ben_w wrote: post #67

Earlier quoted context omitted.

> An aligned AI is not AGI, or whatever they want to call it. There's a few ways I can interpret that. If you mean "alignment and competence are separate axies" then yes. That's well understood by the people running most of these labs. (Or at least, they know how to parrot the clichés stochastically :P) If you mean "alignment precludes intelligence", then no. Consider a divisive presidential election between Alice an…

Campaign team members and even candidates switch teams often enough. Weak analogy. What is the alignment of human GI, completely generalized?

> What is the alignment of human GI, completely generalized?

Of any specific human to any other specific human?

https://benwheatley.github.io/blog/2019/05/25-15.09.10.html

Of any specific human to a nation? That's the example you replied to.

Of all the people of a nation to each other? Best we've done there is what we see in countries in normal times, with all the strife and struggles within.

We have yet to fully extend from nation to the world; the closest for that is the UN, which is even less in agreement with itself than are nations.

Re: The Coming Technological Singularity (1993)

#83

Post by keiferski »

rsaarelm wrote: post #78
keiferski wrote: post #77

Earlier quoted context omitted.

Here’s a simpler way to put it: intelligence and social cooperation are not the same thing. Being good at math or science doesn’t mean you understand how to organize complex political groups, and never has. People tend to think their special gift is what the world needs, and academically-minded smart people (by that I mean people that define their self-worth by intelligence level) are no different.

Yes, because you need to spend a lot of time doing social organization and thinking about it to get very good at it, just like you need to spend a lot of time doing math or science and thinking about it to get very good at it. And then you need to pick up patterns, respond well to unexpected situations and come up with creative solutions on top of that, which requires intelligence. If you look at the people who are t…

I don’t agree at all. Charismatic leaders tend to have both “in born” talent and experience gained over time. It’s not something that comes from sitting in a room and thinking about how to be a good leader.

Sure, some level of intelligence is required, which may be above average. But that is a necessary requirement, not a sufficient one. Raw intelligence is only useful to a certain extent here, and exceeding certain limits may actually be detrimental.

Re: The Coming Technological Singularity (1993)

#84

Post by [deleted] »

trashtester wrote: post #59
samsartor wrote: post #15

> We humans have the ability to internalize the world and conduct "what if's" in our heads; we can solve many problems thousands of times faster than natural selection. I don't know for sure whether superintellegence will happen, but as for the singularity, this is the underlying assumption I have the most issue with. Smart isn't the limiting factor of progress, often it's building consensus, getting funding, waiting…

AlphaFold/AlphaProteo are direct examples of how AI can allow us to bypass experiments when doing science. More importantly, though, is that insufficiently powerful AI has been a limiting factor in robotics. That seems to be coming to an end now. And once we have humanoid robots powered by superhuman intelligence entering the workforce, the impact will be massive. Quite possibly mostly for the bad for most people.

[deleted]

Re: The Coming Technological Singularity (1993)

#85

Post by Retric »

trashtester wrote: post #59
samsartor wrote: post #15

> We humans have the ability to internalize the world and conduct "what if's" in our heads; we can solve many problems thousands of times faster than natural selection. I don't know for sure whether superintellegence will happen, but as for the singularity, this is the underlying assumption I have the most issue with. Smart isn't the limiting factor of progress, often it's building consensus, getting funding, waiting…

AlphaFold/AlphaProteo are direct examples of how AI can allow us to bypass experiments when doing science. More importantly, though, is that insufficiently powerful AI has been a limiting factor in robotics. That seems to be coming to an end now. And once we have humanoid robots powered by superhuman intelligence entering the workforce, the impact will be massive. Quite possibly mostly for the bad for most people.

You’re overstating what AlphaFold can do. Without validation the predictions aren’t dependable, but it can cut down on what’s worth investigating.

There’s definitely a point where models get good enough you can trust them, but super intelligence doesn’t mean it can both come up with a model and be able to trust it without validation.

Re: The Coming Technological Singularity (1993)

#86

Post by arethuza »

keiferski wrote: post #83
rsaarelm wrote: post #78

Earlier quoted context omitted.

Yes, because you need to spend a lot of time doing social organization and thinking about it to get very good at it, just like you need to spend a lot of time doing math or science and thinking about it to get very good at it. And then you need to pick up patterns, respond well to unexpected situations and come up with creative solutions on top of that, which requires intelligence. If you look at the people who are t…

I don’t agree at all. Charismatic leaders tend to have both “in born” talent and experience gained over time. It’s not something that comes from sitting in a room and thinking about how to be a good leader. Sure, some level of intelligence is required, which may be above average. But that is a necessary requirement, not a sufficient one. Raw intelligence is only useful to a certain extent here, and exceeding certain…

When it comes to "charismatic leaders" I like this quote from Frank Herbert:

"“I wrote the Dune series because I had this idea that charismatic leaders ought to come with a warning label on their forehead: "May be dangerous to your health." One of the most dangerous presidents we had in this century was John Kennedy because people said "Yes Sir Mr. Charismatic Leader what do we do next?" and we wound up in Vietnam. And I think probably the most valuable president of this century was Richard Nixon. Because he taught us to distrust government and he did it by example.”

Edit: Maybe what we really need to worry about is an AI developing charisma....

Re: The Coming Technological Singularity (1993)

#87

Post by rsaarelm »

keiferski wrote: post #83
rsaarelm wrote: post #78

Earlier quoted context omitted.

Yes, because you need to spend a lot of time doing social organization and thinking about it to get very good at it, just like you need to spend a lot of time doing math or science and thinking about it to get very good at it. And then you need to pick up patterns, respond well to unexpected situations and come up with creative solutions on top of that, which requires intelligence. If you look at the people who are t…

I don’t agree at all. Charismatic leaders tend to have both “in born” talent and experience gained over time. It’s not something that comes from sitting in a room and thinking about how to be a good leader. Sure, some level of intelligence is required, which may be above average. But that is a necessary requirement, not a sufficient one. Raw intelligence is only useful to a certain extent here, and exceeding certain…

Someone with IQ 160 might have trouble empathizing with what IQ 100 people find convincing or compelling and not do that well with an average IQ 100 population. What if they were dealing with an average IQ 145 population that might be much closer to being on the same wavelength with them to begin with and tried to do social coordination now?

Re: The Coming Technological Singularity (1993)

#88

Post by arethuza »

trescenzi wrote: post #28
samsartor wrote: post #15

> We humans have the ability to internalize the world and conduct "what if's" in our heads; we can solve many problems thousands of times faster than natural selection. I don't know for sure whether superintellegence will happen, but as for the singularity, this is the underlying assumption I have the most issue with. Smart isn't the limiting factor of progress, often it's building consensus, getting funding, waiting…

I do think one of the major weaknesses of “smart people” is they tend to think of intelligence as the key aspect of basically everything. Reality is though we have plenty of intelligence already. We know how to solve most of our problems. The challenges are much more social and our will as a society to make things happen.

Having worked with some very intelligent people my own personal theory is that they forget that they don't have expert level knowledge in everything and actually end up making some pretty silly mistakes that far less smart people would never make - whether this is hubris or being focused and ignoring "trivial" day to day matters is a question of personality.

Re: The Coming Technological Singularity (1993)

#89

Post by keiferski »

rsaarelm wrote: post #87
keiferski wrote: post #83

Earlier quoted context omitted.

I don’t agree at all. Charismatic leaders tend to have both “in born” talent and experience gained over time. It’s not something that comes from sitting in a room and thinking about how to be a good leader. Sure, some level of intelligence is required, which may be above average. But that is a necessary requirement, not a sufficient one. Raw intelligence is only useful to a certain extent here, and exceeding certain…

Someone with IQ 160 might have trouble empathizing with what IQ 100 people find convincing or compelling and not do that well with an average IQ 100 population. What if they were dealing with an average IQ 145 population that might be much closer to being on the same wavelength with them to begin with and tried to do social coordination now?

I guess it’s possible, but again I don’t think empathy and intelligence are correlated. Extremely intelligent people don’t seem any better at navigating the social spheres of high-intelligence spaces than regular people do in regular social spaces. If anything, they’re worse.

All of this is just an overvaluation of intelligence, in my opinion, and largely comes from arrogance.

Re: The Coming Technological Singularity (1993)

#90

Post by keiferski »

arethuza wrote: post #86
keiferski wrote: post #83

Earlier quoted context omitted.

I don’t agree at all. Charismatic leaders tend to have both “in born” talent and experience gained over time. It’s not something that comes from sitting in a room and thinking about how to be a good leader. Sure, some level of intelligence is required, which may be above average. But that is a necessary requirement, not a sufficient one. Raw intelligence is only useful to a certain extent here, and exceeding certain…

When it comes to "charismatic leaders" I like this quote from Frank Herbert: "“I wrote the Dune series because I had this idea that charismatic leaders ought to come with a warning label on their forehead: "May be dangerous to your health." One of the most dangerous presidents we had in this century was John Kennedy because people said "Yes Sir Mr. Charismatic Leader what do we do next?" and we wound up in Vietnam. A…

Not really a good example, honestly. Kennedy’s involvement in Vietnam was the culmination of the previous two decades of events (Korean War, Cuban Missile Crisis, Taiwan standoff, etc.), and not just a crusade he charismatically fooled everyone into joining. If anything, had Nixon won in 1960 (and defeated Kennedy), it’s possible that the war would have escalated more quickly.
Post reply on HN