There is no hard takeoff

That depends heavily on what kind of task we're talking about.

The human brain uses just 12 watts, and with that it can still perform certain tasks that a computer using 100 times as much power can't even approach.

I get the point you were trying to make. In certain tasks it's extremely inefficient, yes. But in general, it's still crazy how power efficient it is.

> Once AI learns to construct efficient mathematical models of the universe

There's not going to be just one model. Whether an AI (in the short term) can be more efficient than a human at discovering these efficient mathematical models remains to be seen. It seems like the creativity and discovery needed to do this is exactly what neural nets can be good at, but then you're back to something fairly inefficient (with todays algorithms)

The text, dated August 10, 2023, discusses the development and limitations of artificial intelligence (AI) and the concept of a sudden "hard takeoff" or "FOOM" (Fast Onset of Overwhelming Might) in AI capabilities.

    Historical Perspective: The author begins by referencing Elon Musk's 2014 comment about AI being like "summoning the demon." They highlight the rapid advancements in AI, such as beating human players in Go, Chess, and Shogi, and playing Atari games.

    Complexity Comparison: The author argues that the universe's complexity is not just a matter of scale compared to a Go game but a difference in kind. They illustrate this by imagining the universe tiled with tiny Go boards, emphasizing that predicting the universe requires understanding at the atomic level, not just the "stone level" of Go.

    Dynamics Models: Modern self-play systems like MuZero and GPT-4 are described as dynamics models that predict the next state based on the current state and action. The author connects intelligence with prediction and compression, suggesting that feeding the entire internet into a model could theoretically "win the universe."

    Practical Limitations: The author challenges the idea that AI can easily dominate complex systems like the stock market. They use the example of hedge funds, explaining that dominating the market requires more compute power than the entire market, which is unrealistic.

    Inclusion of Computers: In Go, the model doesn't need to include other computers, but in modeling the universe, computers must be included. The author argues that unless there's a staggering advantage, understanding them from self-play is unlikely.

    Preventing FOOM: The author warns against capping FLOPS in training runs, as it could create a dangerous situation if one person breaks the restriction. They advocate for preventing a 51% attack on compute to avoid FOOM.

    Efficiency and Innovation: The text dismisses the idea that a single group could achieve a 1e20x efficiency increase, arguing that more intelligence leads to new tricks, but they become harder to find.

    Revolution and Gradual Change: The author predicts an information revolution that will transform intelligence, similar to how the industrial revolution transformed energy. They stress that this change won't happen overnight but will follow a gradual exponential curve.

    Conclusion: The author concludes that unless a "terrifying powder keg" is built, there will be no sudden FOOM. They emphasize the complexity of the universe compared to games and call for letting the markets evolve naturally. The closing statement, "the singularity is nearer," hints at a belief in the eventual convergence of human and machine intelligence, but not in an abrupt or catastrophic manner.

Your intuition however is correct in the sense that there is a principal-agent problem at play with all kinds of hedge funds, where if the hedge fund manager isn't the one investing his own money he is by default incentivized to do things besides just maximizing hedge fund profits. But there are indeed managers who have such an ability to generate edge that they do in fact invest their own money solely, usually money they generated while working for other hedge fund managers before striking out on their own, and these people are terrifying forces to watch in action indeed.

It’s like asking for a drink (legal age 21) and getting rebuked with a “you aren’t even old enough to drive” (legal age 16). That doesn’t mean you can drink when you are 16!

Edit: If you want a hard necessary and sufficient condition for “real intelligence”, mine is “when it can do all of our jobs, i.e. wholesale replace every human”.