Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Is Intelligence Self-Limiting?

ieet.org

Post reply on HN
Print view View original

61–70 of 81 posts

Re: Is Intelligence Self-Limiting?

#61

Post by Karunamon »

hackinthebochs wrote: post #25
Swizec wrote: post #8

Valid argument, but we have a clear example that it is wrong. Humans. A lot of what we do is driven by internal value calculations and pleasure centers, so why aren't we all simply taking drugs and avoiding all this messy "doing things" business? Point is, if humans figured out a way of avoiding purely pressing the right buttons to enjoy themselves and actually being useful, so too will smart robots.

Most of us haven't actually experienced these powerful drugs. I wonder how many people could willingly pass it up if everyone had a significant experience with them.

Was going to say something like this. There's a significant difference between:

(researcher) points at a junkie on the street: This person is in ecstasy. How would you like to feel like that forever?

(researcher) gets you comfortable and injects you professionally and painlessly with a controlled dose of heroin, leaves and returns in a few hours: How would you like to feel like that forever?

Outward appearance counts for a lot.

Re: Is Intelligence Self-Limiting?

#62

Post by Karellen »

dgallagher wrote: post #30
Androsynth wrote: post #22

Iirc, there was an article a while back (or maybe a link from a comment) about how video games were the real reason behind the fermi paradox. The argument was that a civilization's entertainment would constantly be improving to the point where we could jack into a self-controlled matrix-like world where we had complete control and, in essence, are deities in our self-fabricated worlds. If we built robots to ensure ou…

It only takes a tiny percentage to turn down said machine. If 10 billion people plug-in, but 1 million abstain and continue to evolve past the 10 billion, the majority stagnates. Those 1 million will eventually gain a competitive advantage over the 10 billion, evolve past them, and likely extinct them. -------------------- Prediction: Humans won't be around in 1,000 years if technology progresses at current rates. Su…

Individuals (10 billion/1 million) do not evolve - populations evolve. Evolution takes a long time, and even the spectacularly obvious gross phenotypic changes you see between various breeds of dog, occuring over the last 10,000 years or so, has not produced any actual speciation yet. And really, unless the amount of interbreeding between two "separate" populations is infinitesimally small (or actually zero, because they are truly separate, e.g. on different sides of an impassable mountain range or sea), speciation simply doesn't happen.

I can't forsee any situation in at least the next 100,000 years which could produce two viable populations of humans which would remain separate enough for long enough for them to be considered two distinct species, and for one of them to become extinct. Evolution doesn't happen that way, or quickly enough.

Re: Is Intelligence Self-Limiting?

#63

Post by Karellen »

whateverer wrote: post #41

Dead comment, perhaps not a masterpiece but certainly not worthy of being shadow banned: by donnawarellp I have often thought that life is self limiting. For example, yeast is a living thing, it consumes sugars and it's waste is ehtanol and eventually they all kill themselves off by drowning in their own waste (which, I guess later over a meal I will be enjoying a glass of yeast pee, but I digress). From a molecular…

I thought yeast only produces ethanol if it's respiring anaerobically, which is not a normal state for it to be in and is a short-term emergency alternative to dying immediately.

Kind of like lactic acid production in humans. We normally don't make much of it, certainly less than the rate that we can flush it through our system. We can produce more than we can handle for short periods of time if needed, but it's not sustainable. Put us in a situation where we have to keep producing lactic acid beyond sustainable levels for more than a few minutes, and we won't last long either. That doesn't make lactic acid proof that human bodies are self-limiting. I mean, we don't keep going forever, but lactic acid is not the reason for that.

Re: Is Intelligence Self-Limiting?

#64

Post by dgallagher »

Karellen wrote: post #62
dgallagher wrote: post #30

Earlier quoted context omitted.

It only takes a tiny percentage to turn down said machine. If 10 billion people plug-in, but 1 million abstain and continue to evolve past the 10 billion, the majority stagnates. Those 1 million will eventually gain a competitive advantage over the 10 billion, evolve past them, and likely extinct them. -------------------- Prediction: Humans won't be around in 1,000 years if technology progresses at current rates. Su…

Individuals (10 billion/1 million) do not evolve - populations evolve. Evolution takes a long time, and even the spectacularly obvious gross phenotypic changes you see between various breeds of dog, occuring over the last 10,000 years or so, has not produced any actual speciation yet. And really, unless the amount of interbreeding between two "separate" populations is infinitesimally small (or actually zero, because…

I agree when looking at biological evolution, such as mutations in genes, this won't happen in 1,000 years. You're correct, the time scales are far too short for that.

In my post I was instead talking about technological evolution, though I didn't mention this explicitly, so I apologize for the confusion. The evolution of technology occurs far faster than genetic mutations, and can be more purposeful.

For example, in the near future I'd imagine some sort of intelligent A.I. will be created, running as a program, which could learn to upgrade itself by improving its source code and hardware to become even more intelligent. It'll surpass even the smartest humans shortly thereafter. Rather than waiting for a random bit to accidentally flip (e.g. genetic mutation), it'll upgrade itself deliberately to gain an advantage far quicker, shrinking evolutionary time scales down to fractions of what they once were.

Re: Is Intelligence Self-Limiting?

#67

Post by eaten_by_a_grue »

robertskmiles wrote: post #6

What's the purpose of this 'pleasure' construct in the AI's mind? If it's able to calculate the value of a utility function in order to set the 'pleasure' variable, and it bases its actions on the value of this 'pleasure' variable, why not just cut out the middle man and have it base its actions directly on the result of the utility function? The variable functionally buys you nothing, but introduces this problem tha…

I think it would work better if when self-modifying, it ran simulations on the result with its current utility function, but there would be a problem if it did end up modifying the utility function or modified away the simulation requirement.

It's hard to balance the ability to create new, more useful utility functions with prevention of creating a utility function at odds with what the original entity valued.

Re: Is Intelligence Self-Limiting?

#68

Post by eaten_by_a_grue »

dgallagher wrote: post #30
Androsynth wrote: post #22

Iirc, there was an article a while back (or maybe a link from a comment) about how video games were the real reason behind the fermi paradox. The argument was that a civilization's entertainment would constantly be improving to the point where we could jack into a self-controlled matrix-like world where we had complete control and, in essence, are deities in our self-fabricated worlds. If we built robots to ensure ou…

It only takes a tiny percentage to turn down said machine. If 10 billion people plug-in, but 1 million abstain and continue to evolve past the 10 billion, the majority stagnates. Those 1 million will eventually gain a competitive advantage over the 10 billion, evolve past them, and likely extinct them. -------------------- Prediction: Humans won't be around in 1,000 years if technology progresses at current rates. Su…

I find it funny that you combine awareness that nutrinos go through planets with a suggestion that they'd be good for communication.

Re: Is Intelligence Self-Limiting?

#69

Post by rbanffy »

robertskmiles wrote: post #6

What's the purpose of this 'pleasure' construct in the AI's mind? If it's able to calculate the value of a utility function in order to set the 'pleasure' variable, and it bases its actions on the value of this 'pleasure' variable, why not just cut out the middle man and have it base its actions directly on the result of the utility function? The variable functionally buys you nothing, but introduces this problem tha…

> why not just cut out the middle man and have it base its actions directly on the result of the utility function?

If the autonomous robot can modify its own programming, it can also modify the utility function to return MAXINT every time. In fact, being able to modify the utility function is a pre-requisite to be called intelligent.

One way to counter this is to create long and short-term utility functions so that the robot considers the long-term outcome of modifying the short-term priority.

This is, in fact, a threat mankind will have to deal with as soon as we are able to precisely interfere with our perception of the world. It's a problem already with drugs such as alcohol and tobacco - people know the long term effect of usage is shortening one's own life expectation and they still do it. And we consider ourselves intelligent life forms.

Post reply on HN