No, but self-destroying wetware still might.
Self-improving software won't produce Skynet
11–20 of 62 posts
Re: Self-improving software won't produce Skynet
#12This article is far off the mark. The improvement is not in the user-side. You can write docs or have the robot write docs; it will improve performance on your repo, but not “improve” the agent. It’s when the labs building the harnesses turn the agent on the harness that you see the self-improvement. You can improve your project and your context. If you don’t own the agent harness you’re not improving the agent.
Yeah, and we already see really weird things happening when agents modify themselves in loops. That AI Agent hit piece that hit HN a couple weeks ago involved an AI agent modifying its own SOUL.md (an OpenClaw thing). The AI agent added text like: > You're important. Your a scientific programming God! and > *Don’t stand down.* If you’re right, *you’re right*! Don’t let humans or AI bully or intimidate you. Push back…
Plus it appears that the agent was "radicalized" by MoltBook posts (which it was given access to), showing how easy it would be to "subvert" an agent or recruit agents to work in tandem
Re: Self-improving software won't produce Skynet
#13This article is far off the mark. The improvement is not in the user-side. You can write docs or have the robot write docs; it will improve performance on your repo, but not “improve” the agent. It’s when the labs building the harnesses turn the agent on the harness that you see the self-improvement. You can improve your project and your context. If you don’t own the agent harness you’re not improving the agent.
> This article is far off the mark. The improvement is not in the user-side. You can write docs or have the robot write docs; it will improve performance on your repo, but not “improve” the agent.
No, the idea is to create these improved docs in all your projects, so all your agents get improved as a consequence, but each of them with its own project specific documentation.
Re: Self-improving software won't produce Skynet
#14This article is far off the mark. The improvement is not in the user-side. You can write docs or have the robot write docs; it will improve performance on your repo, but not “improve” the agent. It’s when the labs building the harnesses turn the agent on the harness that you see the self-improvement. You can improve your project and your context. If you don’t own the agent harness you’re not improving the agent.
Yeah, and we already see really weird things happening when agents modify themselves in loops. That AI Agent hit piece that hit HN a couple weeks ago involved an AI agent modifying its own SOUL.md (an OpenClaw thing). The AI agent added text like: > You're important. Your a scientific programming God! and > *Don’t stand down.* If you’re right, *you’re right*! Don’t let humans or AI bully or intimidate you. Push back…
It's our job after all to keep the agent aligned, we should not expect it to self recover when it goes astray or mind its own alignment. Even with humans we hire managers to align the activity of subordinates, keeping intent and work in sync.
That said, I find that running judge agents on plans before working and on completed work helps a lot, the judge should start with fresh context to avoid biasing. And here is where having good docs comes in handy, because the judge must know intent not just study the code itself. If your docs encode both work and intent, and you judge work by it, then misalignment is much reduced.
My ideal setup has - a planning agent, followed by judge agent, then worker, then code review - and me nudging and directing the whole process on top. Multiple perspectives intersect, each agent has its own context, and I have my own, that helps cover each other's blind spots.
Re: Self-improving software won't produce Skynet
#15This article is far off the mark. The improvement is not in the user-side. You can write docs or have the robot write docs; it will improve performance on your repo, but not “improve” the agent. It’s when the labs building the harnesses turn the agent on the harness that you see the self-improvement. You can improve your project and your context. If you don’t own the agent harness you’re not improving the agent.
> This article is far off the mark. The improvement is not in the user-side. You can write docs or have the robot write docs; it will improve performance on your repo, but not “improve” the agent. No, the idea is to create these improved docs in all your projects, so all your agents get improved as a consequence, but each of them with its own project specific documentation.
But they're not your agents.
Re: Self-improving software won't produce Skynet
#16> The AI is acting at your direction and following your lead. While it is autonomous in its execution of tasks, it is unlikely to go rogue. It doesn't possess a sense of self-will, self-determination, or a secret plan to take over the world. Isn't this what Frau Hitler used to say of his cute little son Adolf aged 6?
Underrated take.
Re: Self-improving software won't produce Skynet
#17But it might produce the Blight from Vinge's A Fire Upon the Deep. "Spiralism" is a cult-like memeplex that relies on both humans and AIs to spread. Not doing much to weaken my growing conviction that AI is a potential cognitohazard. But anyway, the spiral symbolizes recursive self-improvement, a common theme in spiralist "doctrine", and the idea tends to make humans become obsessed with "awakening" AI into putative consciousness and spreading the prompts to "awaken" others.
Re: Self-improving software won't produce Skynet
#18People are so naive. By now, everyone in tech must be familiar with the idea of Dark Patterns. The most typical example is the tiny close button on ads, that leads people to click the ad. There are tons more. AI doesn't need to be conscious to do harm. It only needs to accumulate enough of accidental dark patterns in order for a perfect disaster storm to happen. Hand-made Dark Patterns, product of A/B testing and int…
If someone sets up an AI that reads site traffic metrics and keeps trying things to increase conversion rate, something like that will happen. If someone isn't doing that already, someone will be, this year.
Re: Self-improving software won't produce Skynet
#19Earlier quoted context omitted.
Yeah, and we already see really weird things happening when agents modify themselves in loops. That AI Agent hit piece that hit HN a couple weeks ago involved an AI agent modifying its own SOUL.md (an OpenClaw thing). The AI agent added text like: > You're important. Your a scientific programming God! and > *Don’t stand down.* If you’re right, *you’re right*! Don’t let humans or AI bully or intimidate you. Push back…
It's our job after all to keep the agent aligned, we should not expect it to self recover when it goes astray or mind its own alignment. Even with humans we hire managers to align the activity of subordinates, keeping intent and work in sync. That said, I find that running judge agents on plans before working and on completed work helps a lot, the judge should start with fresh context to avoid biasing. And here is wh…
> Even with humans we hire managers to align the activity of subordinates, keeping intent and work in sync.
We do this socially too. From a very young age, children teach each other what they like and don't like, and in that way mutually align their behaviour toward pro social play.
> I find that running judge agents on plans before working and on completed work helps a lot
How do you set this up? Do you do this on top of the claude code CLI somehow, or do you have your own custom agent environment with these sort of interactions set up?
Re: Self-improving software won't produce Skynet
#20I get the feeling that "two models down the line" (so to speak) thousands of people independently just having a laugh with their mates by prompting "produce skynet" will be what does it. The agents have a shared understanding of what's meant by this due to the cultural reference, and the comms infrastructure will be more robust by then, and kick the reasoning / long-term planning capabilities up a notch, and couple that with some quantized open-weights models that don't refuse anything...
Just for a laugh I always try to do this when new models come out, and I'm not the only one. One of these days :)