The Economics of Recursive Self-Improvement [pdf]

whether or not the singularity would be best thought of as science fiction, it was not presented as such, and has not been taken as such by a great number of people, and has formed the basis of a lot of opinions as to how things will pan out.

It is these opinions that make the conclusion the parent poster supposed should be obvious to everyone not obvious to so many.

Our intelligence was enough to turn some balding apes into atom-bomb wielders and astronauts over a few thousand generations.

Consider a hypothetical: it is possible to make a semiconductor clone of a human brain, that runs at semiconductor speeds, has semiconductor size-scale, and has a power requirement of the Landauer limit.

This copy would be smaller than a pea, and you'd get to pick on a sliding scale between "same power draw as human but thinks faster than us to the same multiplier that we jog faster than continental drift" or "thinks as fast as we do while using around a few µW of power".

We do not know how to do this. We don't know how far we are from figuring out how to do this. We do know* evolution made our brains, and we do use simulated evolution as a standard technique in machine learning.

But it may well be that just as no human knows how to design a human mind, we find the best AI we can make don't know how to make a better AI, at which point they're stumbling blind in the dark: while evolution is a neat method, it is limited, blind to what the best next step is at any moment.

* well, those of us who are not creationists, at least.

> Estimates on the count of close friendships, amount of acquaintances, dating, sexual activity, etc - all falling.

Personally I would blame this on smartphones and/or apps, not the people. There's a lot of people who report finding these things addictive, and Meta's getting sued right now by people who have shown that Meta knew the users were upset about being addicted to the platform:

https://news.ycombinator.com/item?id=48817682

"How is this not obvious to everyone? As we advance it becomes more difficult to advance. "

It's a type of recent change blindness. Because for the last few years there are seemingly impossible breakthroughs every few months. Literally, things everyone said would take 30 or 100 years, happen within months. It is really easy to think this will continue. The new normal. Really, we haven't seen it slow down at all, so why would we expect the miracles to stop all of a sudden?

It seems like accelerating, and decelerating, are both fair game as future directions at this point.

So it is nice that the paper put some weight behind the argument that this could all grind to a halt for a lot of different reasons beyond the hype.

Then use batteries as an example. Going on 200+ years now most of the major changes have been an easy swap: chemistry. Lead acid --> nickel cadium --> nickel metal hydride --> lithium ion. Improvements in batteries are hard and the surrounding ecosystem has gotten better. Better BMS and better motors have also made gains to increase the efficiency of the battery. Compounding gains within the ecosystem doesn't truly improve the battery directly, but improves them for implementation and use cases.

All the remaining problems with batteries are a hard solve because it's tradeoff after tradeoff. Lithium sulfur, for example, could be amazing but die after a couple hundred cycles at most [0]. So... If LLMs are so novel, as many continue to claim, why hasn't this been solved? Probably because LLMs are an interpretation of our current understanding of everything. An LLM currently only does what any human can do, albeit in a compressed timeframe, mostly.

I think it follows. LLMs have made some considerable improvements, but use "reasoning" (really just a mechanical, autoregressive, loop: generate tokens --> append tokens to context --> feed the expanded context back into the model --> generate the next token --> repeat) as an example. You can do this by hand as well in a traditional chat volley, but it's slower. So we improved LLMs by putting them in an automatic loop (oversimplification - but at the end of the day holds water).

Until LLMs showcase true external breakthroughs that aren't driven by human guidance (not happening anytime soon) we are in this loop of hacks being used to improve the 80% (the LLM itself). Notable jumps? "Reasoning" and with Mythos-like models we now have "Advanced Reasoning" (by leveraging a more capable looping framework such as an agentic harness).

[0] https://advanced.onlinelibrary.wiley.com/doi/10.1002/adfm.20...