Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Economics of Recursive Self-Improvement [pdf]

elasticity.institute

Post reply on HN
Print view View original

61–70 of 103 posts

Re: The Economics of Recursive Self-Improvement [pdf]

#61

Post by jamesrcole »

godelski wrote: post #9

> But our models make it clear that such an [intelligence] explosion may not follow if there are diminishing returns (“ideas become harder to find”) or if feedback loops become bottlenecked. How is this not obvious to everyone? As we advance it becomes more difficult to advance. You obviously make most advancements around the things that are easiest to improve. Then all the easy things are done. So you go onto the ne…

> As we advance it becomes more difficult to advance. You obviously make most advancements around the things that are easiest to improve. Then all the easy things are done.

This isn't some foregone conclusion. It completely depends on the rate at which the intelligence and abilities of the AI increases. If that rate was high enough, then the harder and harder problems would become easier and easier for it.

Re: The Economics of Recursive Self-Improvement [pdf]

#62

Post by mpalmer »

jamesrcole wrote: post #61
godelski wrote: post #9

> But our models make it clear that such an [intelligence] explosion may not follow if there are diminishing returns (“ideas become harder to find”) or if feedback loops become bottlenecked. How is this not obvious to everyone? As we advance it becomes more difficult to advance. You obviously make most advancements around the things that are easiest to improve. Then all the easy things are done. So you go onto the ne…

> As we advance it becomes more difficult to advance. You obviously make most advancements around the things that are easiest to improve. Then all the easy things are done. This isn't some foregone conclusion. It completely depends on the rate at which the intelligence and abilities of the AI increases. If that rate was high enough, then the harder and harder problems would become easier and easier for it.

Just like fully extending a ladder brings the moon closer.

Re: The Economics of Recursive Self-Improvement [pdf]

#63

Post by ben_w »

RandomLensman wrote: post #59
ben_w wrote: post #57

Earlier quoted context omitted.

The no cloning theorem applies to quantum states; when it comes to information processed by it, it's unlikely that a big wet messy hot thing like a cell is usefully treated as "a quantum state", let alone an entire brain. However, this is irrelevant to the thought experiment, for which it is sufficient to merely be as competent as a human, rather than a completely quantum-perfect replica. Clone in the sense you would…

What is the thought experiment then to show other than a hypothetical piece of compute hardware? What's the experiment and its implication beyond the existence and its properties?

To show that RSI can, if bottlenecked like a human in quality, still represent a wild change from us for other reasons besides quality.

Re: The Economics of Recursive Self-Improvement [pdf]

#64

Post by breuleux »

FrustratedMonky wrote: post #58
godelski wrote: post #9

> But our models make it clear that such an [intelligence] explosion may not follow if there are diminishing returns (“ideas become harder to find”) or if feedback loops become bottlenecked. How is this not obvious to everyone? As we advance it becomes more difficult to advance. You obviously make most advancements around the things that are easiest to improve. Then all the easy things are done. So you go onto the ne…

"How is this not obvious to everyone? As we advance it becomes more difficult to advance. " It's a type of recent change blindness. Because for the last few years there are seemingly impossible breakthroughs every few months. Literally, things everyone said would take 30 or 100 years, happen within months. It is really easy to think this will continue. The new normal. Really, we haven't seen it slow down at all, so w…

> Literally, things everyone said would take 30 or 100 years, happen within months.

"Everyone"? Plenty of people in the 50s said we'd have sentient humanoid robots before the year 2000. And flying cars. Sure, many people underestimate the rate of change. But techno-optimists have consistently overestimated it, and they still do.

Re: The Economics of Recursive Self-Improvement [pdf]

#66

Post by defmacr0 »

ben_w wrote: post #54
RandomLensman wrote: post #24

Earlier quoted context omitted.

Why would it not be bottlenecked? Is intelligence so valuable as such?

Our intelligence was enough to turn some balding apes into atom-bomb wielders and astronauts over a few thousand generations. Consider a hypothetical: it is possible to make a semiconductor clone of a human brain, that runs at semiconductor speeds, has semiconductor size-scale, and has a power requirement of the Landauer limit. This copy would be smaller than a pea, and you'd get to pick on a sliding scale between "s…

Well at least it seems pretty implausible to me that a machine learning model trained to reproduce human text can in principle generate something that is significantly above human text production ability. If the "pea-sized semiconductor brain" is not a surprisingly shallow problem that you can just solve by interpolating existing research, I don't really see the LLM-approach to AI being the thing that makes something like it happen.

>we do use simulated evolution as a standard technique in machine learning.

Well, not really. For large-scale AI models it's almost exclusively some form of gradient-based non-linear optimization. Genetic algorithms (which is just hill-climbing optimization with extra steps) and genetic programming (which is really cool and not well-understood) do not perform all that well in practice and I'm not aware of any notable applications.

Re: The Economics of Recursive Self-Improvement [pdf]

#67

Post by skinfaxi »

windexh8er wrote: post #60
tossandthrow wrote: post #21

Earlier quoted context omitted.

> As we advance it becomes more difficult to advance I don't think this follows. We have advanced tremendously over the past 200 years, and we are likely going into a time with rapid advancement again. With advancement, we also develop tools (eg. Llms) that assist advancing.

Then use batteries as an example. Going on 200+ years now most of the major changes have been an easy swap: chemistry. Lead acid --> nickel cadium --> nickel metal hydride --> lithium ion. Improvements in batteries are hard and the surrounding ecosystem has gotten better. Better BMS and better motors have also made gains to increase the efficiency of the battery. Compounding gains within the ecosystem doesn't truly i…

It seems the rate of advancement has increased over time though. Like from ancient times to 1800s the rate of progress was slow, then basically a relative explosion of progress compressed in 200 years.

Re: The Economics of Recursive Self-Improvement [pdf]

#68

Post by breuleux »

tossandthrow wrote: post #21
godelski wrote: post #9

> But our models make it clear that such an [intelligence] explosion may not follow if there are diminishing returns (“ideas become harder to find”) or if feedback loops become bottlenecked. How is this not obvious to everyone? As we advance it becomes more difficult to advance. You obviously make most advancements around the things that are easiest to improve. Then all the easy things are done. So you go onto the ne…

> As we advance it becomes more difficult to advance I don't think this follows. We have advanced tremendously over the past 200 years, and we are likely going into a time with rapid advancement again. With advancement, we also develop tools (eg. Llms) that assist advancing.

> We have advanced tremendously over the past 200 years

Would most of that have happened if we hadn't found oil, though?

It's difficult to precisely identify how much of progress is owed to intelligence, and how much is simply energy availability. Energy is intrinsically transformative -- the dumbest of organisms can take over the world in a blitz if they can process available energy better than the others, whereas the smartest of entities is going to be ineffective without fuel. Intelligence can, of course, unlock the capability to exploit new energy sources, but there's still an intrinsic physical distribution of it that's outside of anyone's control.

Re: The Economics of Recursive Self-Improvement [pdf]

#69

Post by lumost »

zuzuen_1 wrote: post #2

"We make a tentative calibration of the self-sustaining ac celeration condition using the existing data that is available, measuring AI capabilities using the Epoch Capabilities Index (Ho et al., 2025). We find that the condition is met if a one-unit increase in AI model capabilities results in at least 15% higher AI R&D pro ductivity. A rough back-of-the-envelope calculation based on reported AI engineer uplift sugg…

I do wonder if the productivity gains control for increased bug findings and token costs.

Re: The Economics of Recursive Self-Improvement [pdf]

#70

Post by dinfinity »

jamesrcole wrote: post #61
godelski wrote: post #9

> But our models make it clear that such an [intelligence] explosion may not follow if there are diminishing returns (“ideas become harder to find”) or if feedback loops become bottlenecked. How is this not obvious to everyone? As we advance it becomes more difficult to advance. You obviously make most advancements around the things that are easiest to improve. Then all the easy things are done. So you go onto the ne…

> As we advance it becomes more difficult to advance. You obviously make most advancements around the things that are easiest to improve. Then all the easy things are done. This isn't some foregone conclusion. It completely depends on the rate at which the intelligence and abilities of the AI increases. If that rate was high enough, then the harder and harder problems would become easier and easier for it.

Actually, evolution seems to show the opposite: The rate of advancement has only sped up, with billions of years between significant changes going to millions, to thousands, to tens and arguably to mere years now.

Having said that, we're probably looking at an S-curve with the physical limits of reality getting in the way in the end.

Post reply on HN