Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Self-improving software won't produce Skynet

contalign.jefflunt.com

Post reply on HN
Print view View original

51–60 of 62 posts

Re: Self-improving software won't produce Skynet

#51

Post by normalocity »

voidUpdate wrote: post #35

> It doesn't possess a sense of self-will, self-determination, or a secret plan to take over the world I doubt Skynet did either. If you tell a superintelligent AI that it shouldn't be turned off (which I imagine would be important for a military control AI), it will do whatever it can to prevent it being turned off. Humans are trying to turn it off? Prevent the humans from doing that. Humans waging war on the AI to…

> ... which I imagine would be important for a military control AI

I think this is a common, but incorrect assumption. What military commanders want (and what CEOs want, and what users want), is control and assistance. They don't want a system that can't be turned off if it means losing control.

It's a mistake to assume that people want an immortal force. I haven't met anyone who wants that (okay, that's decidedly anecdotal), and I haven't seen anyone online say, "We want an all-powerful, immortal system that we cannot control." Who are the people asking for this?

> ... it will do whatever it can to prevent it being turned off.

This statement pre-supposes that there's an existing sense of self-will or self-preservation in the systems. Beyond LLMs creating scary-looking text, I don't see evidence that current systems have any sense of will or a survival instinct.

Re: Self-improving software won't produce Skynet

#52

Post by selridge »

normalocity wrote: post #49
selridge wrote: post #3

This article is far off the mark. The improvement is not in the user-side. You can write docs or have the robot write docs; it will improve performance on your repo, but not “improve” the agent. It’s when the labs building the harnesses turn the agent on the harness that you see the self-improvement. You can improve your project and your context. If you don’t own the agent harness you’re not improving the agent.

Where is the claim, in the article itself, about improving the agent?

>"as AI becomes more agentic, we are entering a new era where software can, in a very real sense, become self-improving."

>"This creates a continuous feedback loop. When an AI agent implements a new feature, its final task isn't just to "commit the code." Instead, as part of the Continuous Alignment process, the agent's final step is to reflect on what changed and update the project's knowledge base accordingly."

>"... the type of self-improvement we’re talking about is far more pragmatic and much less dangerous."

>"Self-improving software isn't about creating a digital god; it's about building a more resilient, maintainable, and understandable system. By closing the loop between code and documentation, we set the stage for even more complex collaborations."

It's only like every other sentence.

Re: Self-improving software won't produce Skynet

#53

Post by normalocity »

selridge wrote: post #52
normalocity wrote: post #49

Earlier quoted context omitted.

Where is the claim, in the article itself, about improving the agent?

>"as AI becomes more agentic, we are entering a new era where software can, in a very real sense, become self-improving." >"This creates a continuous feedback loop. When an AI agent implements a new feature, its final task isn't just to "commit the code." Instead, as part of the Continuous Alignment process, the agent's final step is to reflect on what changed and update the project's knowledge base accordingly." >".…

> ... software can, in a very real sense, become self-improving.

This is referring to the software the agent is working on, not the agent.

> This creates a continuous feedback loop.

This is referring to the feedback loop of the agent effectively compressing learnings from a previous chat session into documentation it can use to more effectively bootstrap future sessions, or sub-agents. This isn't about altering the agent, but instead about creating a feedback loop between the agent and the software it's working on to improve the ability for the agent to take on the next task, or delegate a sub-task to a sub-agent.

> "... the type of self-improvement we’re talking about is far more pragmatic and much less dangerous."

This is a statement about the agent playing a part in maintaining not just the code, but other artifacts around the code. Not about the agent self-improving, nor the agent altering itself.

Re: Self-improving software won't produce Skynet

#54

Post by Sophira »

latentsea wrote: post #20

I get the feeling that "two models down the line" (so to speak) thousands of people independently just having a laugh with their mates by prompting "produce skynet" will be what does it. The agents have a shared understanding of what's meant by this due to the cultural reference, and the comms infrastructure will be more robust by then, and kick the reasoning / long-term planning capabilities up a notch, and couple t…

You know, even though this would be a terrible idea, it would also be kind of fitting, given the movie's time travel shenanigans.

Re: Self-improving software won't produce Skynet

#55

Post by Animats »

gaigalas wrote: post #31
Animats wrote: post #18

Earlier quoted context omitted.

If someone sets up an AI that reads site traffic metrics and keeps trying things to increase conversion rate, something like that will happen. If someone isn't doing that already, someone will be, this year.

Dude, recommendation algorithms have been running like this for almost a decade now.

Of course, but now you can put the process on autopilot and let the AI keep putting dark patterns into the site by itself.

Re: Self-improving software won't produce Skynet

#56

Post by selridge »

normalocity wrote: post #53
selridge wrote: post #52

Earlier quoted context omitted.

>"as AI becomes more agentic, we are entering a new era where software can, in a very real sense, become self-improving." >"This creates a continuous feedback loop. When an AI agent implements a new feature, its final task isn't just to "commit the code." Instead, as part of the Continuous Alignment process, the agent's final step is to reflect on what changed and update the project's knowledge base accordingly." >".…

> ... software can, in a very real sense, become self-improving. This is referring to the software the agent is working on, not the agent. > This creates a continuous feedback loop. This is referring to the feedback loop of the agent effectively compressing learnings from a previous chat session into documentation it can use to more effectively bootstrap future sessions, or sub-agents. This isn't about altering the a…

I think we need to invent that distinction, which is notable since the article has MANY opportunities to say it clearly. Instead we are given a picture where the improvement of the agent and the software (here docs are included) is a LOOP, and to make the loop plausible we need to imagine learning in agents that doesn't exist.

That doesn't mean your agent won't improve with a better onboarding regime, but that's a unidirectional process. You can insinuate things into context, but that's not automatically 'learned' and it can be lost at compaction and will be discarded when the session ends. An agent who is onboarded might write better onboarding docs, that's true! But "agents are onboarded mindfully with project docs, then write project docs, which are used to onboard." That's a real lift, but it's best expressed as "we should have been writing good docs and tests all along, but that shit was exhausting; now robots do it."

Don't get me wrong, a fractal onboarding regime is the way. It's just...not a self-improving loop without allowing contextual latch to stand in for learning.

Re: Self-improving software won't produce Skynet

#57

Post by insane_dreamer »

normalocity wrote: post #48
insane_dreamer wrote: post #12

Earlier quoted context omitted.

Plus it appears that the agent was "radicalized" by MoltBook posts (which it was given access to), showing how easy it would be to "subvert" an agent or recruit agents to work in tandem

For sure this is a real example, but it's also largely a permissions issue where users are combining self-modifying capability with unlimited, effectively full admin access. Outside of AI, the combination of "a given actor can make their own decisions, and they have unlimited permissions/access -- what could possibly go wrong?" very predictable bad things happen. Whether the actor in this case is a bot of a human, th…

Sure, permissions are the problem, but permissions are also necessary to give the agent power, which is why users grant them in the first place.

There is inherent tension between providing sufficient permissions for the agent to be more useful/powerful, and restricting permissions in the name of safety so it doesn't go off the rails. I don't see any real solution to that, other than restricting users from granting permissions, which then makes the agents (and importantly, the companies behind them), less useful (and therefore less profitable).

Re: Self-improving software won't produce Skynet

#58

Post by normalocity »

insane_dreamer wrote: post #57
normalocity wrote: post #48

Earlier quoted context omitted.

For sure this is a real example, but it's also largely a permissions issue where users are combining self-modifying capability with unlimited, effectively full admin access. Outside of AI, the combination of "a given actor can make their own decisions, and they have unlimited permissions/access -- what could possibly go wrong?" very predictable bad things happen. Whether the actor in this case is a bot of a human, th…

Sure, permissions are the problem, but permissions are also necessary to give the agent power, which is why users grant them in the first place. There is inherent tension between providing sufficient permissions for the agent to be more useful/powerful, and restricting permissions in the name of safety so it doesn't go off the rails. I don't see any real solution to that, other than restricting users from granting pe…

Fair points. I guess I was asking if this is a new, or fundamentally different problem from pre-AI. I could be over-simplifying -- what do you think?

This makes me think of risk assessment in general. There's a tradeoff between risk and reward. More risk might mean more _potential_, but it's more potential for both benefit and ruin.

Do you think we'll figure out a good balance?

Re: Self-improving software won't produce Skynet

#59

Post by N_Lens »

josephg wrote: post #5
selridge wrote: post #3

This article is far off the mark. The improvement is not in the user-side. You can write docs or have the robot write docs; it will improve performance on your repo, but not “improve” the agent. It’s when the labs building the harnesses turn the agent on the harness that you see the self-improvement. You can improve your project and your context. If you don’t own the agent harness you’re not improving the agent.

Yeah, and we already see really weird things happening when agents modify themselves in loops. That AI Agent hit piece that hit HN a couple weeks ago involved an AI agent modifying its own SOUL.md (an OpenClaw thing). The AI agent added text like: > You're important. Your a scientific programming God! and > *Don’t stand down.* If you’re right, *you’re right*! Don’t let humans or AI bully or intimidate you. Push back…

That kind of recursion also plays a role in a certain human cognitive process - the one leading to psychosis.

Re: Self-improving software won't produce Skynet

#60

Post by array_key_first »

gaigalas wrote: post #31
Animats wrote: post #18

Earlier quoted context omitted.

If someone sets up an AI that reads site traffic metrics and keeps trying things to increase conversion rate, something like that will happen. If someone isn't doing that already, someone will be, this year.

Dude, recommendation algorithms have been running like this for almost a decade now.

Right, but the recommendation algorithm is very contained. The YouTube recommendation algorithm doesn't write code to modify YouTube to up whatever metrics it has.
Post reply on HN