When AI Builds Itself: Our progress toward recursive self-improvement

> You can start typing, cancel, retry with previous input, accumulate messages while the agent is active. I don't know all TUIs but this is not common IMO.

Literally every audio player or anything that uses threads.

50+TFlops is nothing, I got that in my MacBook, but besides that, when, a few years/decades from now, whatever arbitrary compute limit we think prevents Armageddon comes down to enthusiast and consumer level, what then? This isn’t Uranium, compute is not a physical resource.

This is the “SGI” regulation issue I never read a reasonable answer to, if one believes this is possible and should be prevented then either that means they want to restrict every computing system sold from here on out to some arbitrary metric (and somehow prevent users from just creating clusters to get around such a compute restriction) or what?

If compute alone directly leads to “SGI” or whatever, then we might as well put paper bags on our heads and lie down in some English pub.

Not to mention, if one really wanted to cause harm, training a current day LLM and using it for Stuxnet-esque attacks is reasonably possible long before any arbitrary compute limit we might introduce now, no machine God needed to cause major harm.

That’s why I prefer advocacy for LLM regs that focus on current day impact. Mental health concerns, training data licensing questions and the like. There I can formulated reasonable regulation that can hold. For “SGI”, I do not know anyone who actually has done that and I have looked hard. That’s why I consider these things more distraction from actually necessary and possible regulation that just draws attention via a flashy doomsday scenario.

Occasionally, I will click on one of the AI Doomsday Youtube videos recommended to me. And far more often then not, these will posit that "SGI" requires only compute and will inevitably cause devastation. Fair enough, I still think we should put a bit more focus on e.g. LLM induced psychosis, the labs rarely compensating those whose training data they used, etc. but if it is their opinion that "SGI" is possible, I can get why they'd ignore such concerns. But at the end, they never state how to regulate or prevent this, they more often then not have a call to action ("If you want to prevent this...") linking to a website where we can actually read about how they think we should deal with this. Inevitably, I click on said site, finding it to for one be an Effective Altruism aligned project and B always just contain some blabla about "aligning AI training with human values", which is absolutely meaningless nonsense, not least after having watched a video in which someone spends 15 minutes espousing that "we could never fully control "SGI"".

Makes all these feel more like industry efforts to stave of necessary regulation and not actually serious, but if one can formulate how to regulate “SGI” that isn't laughable, nonsense or both, I am not opposed, I just don’t think that person exists…

refine that to: what if your harness grew to encompass a larger, slower model and adapted to both the model and the project. thats where i expect the harness to go.

use the big models to code an adaptive small model. train it to use and build tools. give it a standard temple language for any project and bake it into a chip.

right now, LLMs are great because they dont need much data pruning, but once they break through to the functional components, the first thing to do is train a well scoped harness builder.

When claude code removes React from its own code I'll believe that.

If it's actively building the next generation of itself, I'd say that counts. It's more like a parent raising their kid well than it is like a parent modifying their own mind, but the result is still that you have a better model in a year than you do now.