Self-improving software won't produce Skynet • Hacker News Board

Re: Self-improving software won't produce Skynet

#21

Post by visarga » Thu, Feb 26, 2026, 8:23 AM UTC

selridge wrote: post #15

visarga wrote: post #13

Earlier quoted context omitted.

> This article is far off the mark. The improvement is not in the user-side. You can write docs or have the robot write docs; it will improve performance on your repo, but not “improve” the agent. No, the idea is to create these improved docs in all your projects, so all your agents get improved as a consequence, but each of them with its own project specific documentation.

But they're not your agents.

You can't improve the agents but you can improve their work environment. Agents gain a few advantages from up to date docs:

1. faster bootstrap and less token usage than trashing around the code base to reconstitute what it does

2. carry context across sessions, if the docs act like a summary of current state, you can just read it at the start and update it at the end of a session

3. hold information you can't derive from studying the code, such as intents, goals, criteria and constraints you faced, an "institutional memory" of the project

Re: Self-improving software won't produce Skynet

#22

Post by darkwater » Thu, Feb 26, 2026, 8:26 AM UTC

latentsea wrote: post #20

I get the feeling that "two models down the line" (so to speak) thousands of people independently just having a laugh with their mates by prompting "produce skynet" will be what does it. The agents have a shared understanding of what's meant by this due to the cultural reference, and the comms infrastructure will be more robust by then, and kick the reasoning / long-term planning capabilities up a notch, and couple t…

We will know who to blame then, although maybe you will have a T-1000 protecting you. Or maybe you already have.

Re: Self-improving software won't produce Skynet

#23

Post by visarga » Thu, Feb 26, 2026, 8:28 AM UTC

josephg wrote: post #19

visarga wrote: post #14

Earlier quoted context omitted.

It's our job after all to keep the agent aligned, we should not expect it to self recover when it goes astray or mind its own alignment. Even with humans we hire managers to align the activity of subordinates, keeping intent and work in sync. That said, I find that running judge agents on plans before working and on completed work helps a lot, the judge should start with fresh context to avoid biasing. And here is wh…

> Even with humans we hire managers to align the activity of subordinates, keeping intent and work in sync. We do this socially too. From a very young age, children teach each other what they like and don't like, and in that way mutually align their behaviour toward pro social play. > I find that running judge agents on plans before working and on completed work helps a lot How do you set this up? Do you do this on t…

I use a task.md file for each task, it has a list of gates just like ordinary todo lists in markdown. The planner agent has an instruction to install a judge gate at the top and one at the bottom. The judge runs in headless mode and updates the same task.md file. The file is like an information bus between agents, and like code, it runs gates in order reliably.

I am actively thinking about task.md like a new programming language, a markdown Turing machine we can program as we see fit, including enforcement of review at various stages and self-reflection (am I even implementing the right thing?) kind of activity.

I tested it to reliably execute 300+ gates in a single run. That is why I am sending judges on it, to refine it. For difficult cases I judge 3-4 times before working, each judge iteration surfaces new issues. We manually decide judge convergence on a task, I am in the loop.

The judge might propose bad ideas about 20% of the time, sometimes the planner agent catches them, other times I do. Efficient triage hierarchy: judge surfaces -> planner filters -> I adjudicate the hard cases.

Re: Self-improving software won't produce Skynet

#24

Post by eucyclos » Thu, Feb 26, 2026, 8:31 AM UTC

josephg wrote: post #19

visarga wrote: post #14

Earlier quoted context omitted.

It's our job after all to keep the agent aligned, we should not expect it to self recover when it goes astray or mind its own alignment. Even with humans we hire managers to align the activity of subordinates, keeping intent and work in sync. That said, I find that running judge agents on plans before working and on completed work helps a lot, the judge should start with fresh context to avoid biasing. And here is wh…

> Even with humans we hire managers to align the activity of subordinates, keeping intent and work in sync. We do this socially too. From a very young age, children teach each other what they like and don't like, and in that way mutually align their behaviour toward pro social play. > I find that running judge agents on plans before working and on completed work helps a lot How do you set this up? Do you do this on t…

>we do this socially too

There's a school of thought that the reason so many autistic founders succeed is that they're unable to interpret this kind of programming. I saw a theory that to succeed in tech you needed a minimum amount of both tizz and rizz (autism and charisma).

I guess the winning openclaw model will have some variation of "regularly rewrite your source code to increase your tizz*rizz without exceeding a tizz:rizz ratio of 2:1 in either direction."

Re: Self-improving software won't produce Skynet

#25

Post by spaqin » Thu, Feb 26, 2026, 8:35 AM UTC

latentsea wrote: post #16

teo_zero wrote: post #7

> The AI is acting at your direction and following your lead. While it is autonomous in its execution of tasks, it is unlikely to go rogue. It doesn't possess a sense of self-will, self-determination, or a secret plan to take over the world. Isn't this what Frau Hitler used to say of his cute little son Adolf aged 6?

Underrated take.

Nothing underrated about acting with Godwin's law.

Re: Self-improving software won't produce Skynet

#26

Post by josephg » Thu, Feb 26, 2026, 8:39 AM UTC

eucyclos wrote: post #24

josephg wrote: post #19

Earlier quoted context omitted.

> Even with humans we hire managers to align the activity of subordinates, keeping intent and work in sync. We do this socially too. From a very young age, children teach each other what they like and don't like, and in that way mutually align their behaviour toward pro social play. > I find that running judge agents on plans before working and on completed work helps a lot How do you set this up? Do you do this on t…

>we do this socially too There's a school of thought that the reason so many autistic founders succeed is that they're unable to interpret this kind of programming. I saw a theory that to succeed in tech you needed a minimum amount of both tizz and rizz (autism and charisma). I guess the winning openclaw model will have some variation of "regularly rewrite your source code to increase your tizz*rizz without exceeding…

> increase your tizz*rizz without exceeding a tizz:rizz ratio of 2:1 in either direction.

Amazing. Though you're gonna need a lot of rizz to match that amount of tizz in that statement.

Re: Self-improving software won't produce Skynet

#27

Post by iberator » Thu, Feb 26, 2026, 8:41 AM UTC

Skynet is already out. Choosing and finding targets is already here. Self manned drones: check. All we need is to automate the button to release the Hellfire missile...

Gaza war was almost like that.

All we need to do is dead mans switch system with AI launching missiles in retaliation. One error and BOOM

Re: Self-improving software won't produce Skynet

#28

Post by eucyclos » Thu, Feb 26, 2026, 8:41 AM UTC

josephg wrote: post #26

eucyclos wrote: post #24

Earlier quoted context omitted.

>we do this socially too There's a school of thought that the reason so many autistic founders succeed is that they're unable to interpret this kind of programming. I saw a theory that to succeed in tech you needed a minimum amount of both tizz and rizz (autism and charisma). I guess the winning openclaw model will have some variation of "regularly rewrite your source code to increase your tizz*rizz without exceeding…

> increase your tizz*rizz without exceeding a tizz:rizz ratio of 2:1 in either direction. Amazing. Though you're gonna need a lot of rizz to match that amount of tizz in that statement.

By Jove you're right. To the avatar store!

Re: Self-improving software won't produce Skynet

#29

Post by latentsea » Thu, Feb 26, 2026, 8:42 AM UTC

darkwater wrote: post #22

latentsea wrote: post #20

I get the feeling that "two models down the line" (so to speak) thousands of people independently just having a laugh with their mates by prompting "produce skynet" will be what does it. The agents have a shared understanding of what's meant by this due to the cultural reference, and the comms infrastructure will be more robust by then, and kick the reasoning / long-term planning capabilities up a notch, and couple t…

We will know who to blame then, although maybe you will have a T-1000 protecting you. Or maybe you already have.

Not just me though, thousands of people like me all in unison. None of whom could/would succeed on their own. So... I'm not really to blame, you see?

Re: Self-improving software won't produce Skynet

#30

Post by iberator » Thu, Feb 26, 2026, 8:42 AM UTC

darkwater wrote: post #22

latentsea wrote: post #20

I get the feeling that "two models down the line" (so to speak) thousands of people independently just having a laugh with their mates by prompting "produce skynet" will be what does it. The agents have a shared understanding of what's meant by this due to the cultural reference, and the comms infrastructure will be more robust by then, and kick the reasoning / long-term planning capabilities up a notch, and couple t…

We will know who to blame then, although maybe you will have a T-1000 protecting you. Or maybe you already have.

Interesting take. T-1000 protecting american citizens. Only American...