81–90 of 210 posts
Earlier quoted context omitted.
I will believe AI is super strong when they start pulling out 10-d chess moves. I’m yet to see it.
If AI becomes really strong and sets itself the target of world domination, you maybe won't see those moves. You will just die in your sleep one day, or find no machine is under your control anymore. I believe we are quite far from it, but that it makes sense to keep an eye out now. And think of resilient systems, manual overrides, etc. ...
Earlier quoted context omitted.
Yes the exponential self improvement folks have never heard of an eigenvalue I guess. You can loop forever using output as input but at some point the result will stop changing (depending on the function)
that's not really how eigenvalues work... they specifically also model the case where the result keeps changing exponentially.
RSI(LLM) = RSI(LLM) -- for an optimal LLM* which is a fixed point of RSI
As for eigenvalues/vectors, they're fixed points of (1/val)A or A*val
Earlier quoted context omitted.
RSI is a fetishistic term among the singularity crowd, who imagine AI "recursively" improving itself in some exponential fashion until there is a bright flash of white light and it reveals itself in the form of god. Or something like that. I don't know why whoever coined the term chose "recursive" rather than "iterative" - just sounds more likely to lead to infinite regress I suppose. This notion of recursive/iterati…
I felt like the scaling laws were magical thinking, but apparently they work. However I still do not understand why we should expect exponential improvements due to this automated process. My intuition is that the first iteration of it should result in a noticeable capability increase (though I think these labs were already using a lot of AI to orchestrate training the current model anyway), and then the second itera…
This roughly lines up with my personal experience that in March a combination of stronger models and better tooling on my end let me start running jobs unattended 24/7 (using Anthropic sub and my own hardware). Their $8000/day per researcher spend is crazy though, I'm curious how they keep track of the work.
> let me start running jobs unattended 24/7 (using Anthropic sub and my own hardware) How are you running jobs unattended 24/7 without hitting your token limits?
Over 24h my token spend is I'm not sure what the point would be though unless working on some kind of optimization problem -- it takes me days to review <24h of the agent's output. It's almost always near enough to correct to be shippable; though I do give it feedback and iterate until it's better than the code I would have written.
> ... We are pursuing this work in part because automated research could help us solve alignment and build defenses against increasingly capable AI. An automated AI researcher can also be an automated safety or alignment researcher. More capable, aligned systems could help secure critical infrastructure, defend against dangerous AI agents, and develop new protective measures. In other words... "We must pursue advance…
And... are they wrong?
This is why there's talk about negotiated "pacing."
I want an all-powerful AI that's aligned with my values, but not necessarily yours. Is that so much to ask for?
Earlier quoted context omitted.
Iteration and recursion are famously equivalent
But you get more funding when you call it Recursive Self Improvement. Even better if you call it RSI so it doesn't evoke pesky skynet scenarios outside of AI safety circles.
Earlier quoted context omitted.
“Recursive” is a reasonable term because the generation N AIs will train the Generation N+1 AIs. The term “iterative” doesn’t reflect this nuance as well IMO.
Recursion reduces each step toward a base case: each step is defined in terms of previous/simpler steps, not more advanced ones. The "recursive" in "recursive self improvement" has things precisely backward. Iteration correctly describes a process where each step is the starting point of its successive step, so it should be "iterative self improvement" but I guess that didn't sound as cool.
Compare the similarity of:
AI(n) = improve(AI(n-1))
Fib(n) = Fib(n-1) + Fib(n-2)
Edit: formatting
Earlier quoted context omitted.
that's not really how eigenvalues work... they specifically also model the case where the result keeps changing exponentially.
The claim is that the RSI operation is just finding a fixed point of improvement, RSI(LLM ) = RSI(LLM ) -- for an optimal LLM* which is a fixed point of RSI As for eigenvalues/vectors, they're fixed points of (1/val)A or A*val
> More precisely, an eigenvector v of a linear transformation T is scaled by a constant factor lambda when the linear transformation is applied to it: Tv = lambda v .
In other words, repeated multiplication of an eigenvector by a matrix can still create exponential growth.