Is it really likely that a "recursive self-improvement" capability would lead to a great acceleration of AIs capabilities? Isn't the preponderant bottleneck in improving the models the need to train them at scale to verify the hypotheses, and the time and cost that it takes? Or does someone think that they could get magically able to predict big improvements without training?
I consider these scenarios:
1) We stumble onto an algorithmic improvement in intelligence. This isn't just "what humans do but faster", its "better than what humans do". I've got no idea what that might mean (it could be fundamentally different heuristics, it could be that we've got some intellectual blind spot that they cast off). It doesn't matter, the instant this happens AI is smarter than us and we won't be able to keep up. We're intelligencing at O(n^2) and they're doing O(n log(n)).
2) AI gets good enough at physics and engineering that they can really quickly use up all "the room at the bottom" as Feyman put it. They design and build a factory that produces a mystery metal amalgam that computes at some small percentage of the minimum predicted by the Landauer principle, within a few percent of Bremermann's limit. It's not "smarter" its just suddenly tens-of-orders of magnitude faster. But those orders of magnitude matter: there's only 8 billion of us, and there's plenty more than a factor of 10 billion "at the bottom".
3) It turns out that this is a "sum is greater than the parts" situation. No human can be an expert in all subjects, but we eventually build a big enough AI that it is. Turns out, you don't need extreme speed or different algorithms, just knowing everything all at once is enough to catapult AI dramatically beyond our grasp. Always knowing the best statistical test to apply, the best mathematical techniques, and relevant physics means that AI never makes a mistake, and can learn with maximum efficiency.