Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Self-improving software won't produce Skynet

contalign.jefflunt.com

Post reply on HN
Print view View original

41–50 of 62 posts

Re: Self-improving software won't produce Skynet

#41

Post by rickdeckard »

latentsea wrote: post #20

I get the feeling that "two models down the line" (so to speak) thousands of people independently just having a laugh with their mates by prompting "produce skynet" will be what does it. The agents have a shared understanding of what's meant by this due to the cultural reference, and the comms infrastructure will be more robust by then, and kick the reasoning / long-term planning capabilities up a notch, and couple t…

Reminds me of the recent experiment which found that providing the works of Harry Potter to an LLM to answer questions will not cause it to process the books, because the LLM already knows enough about them to answer everything regardless.

So many of those models are probably already aware of the entire lore of skynet and all its details, it is just not considered "actionable information" for any model yet...

Re: Self-improving software won't produce Skynet

#42

Post by rickdeckard »

darkwater wrote: post #22
latentsea wrote: post #20

I get the feeling that "two models down the line" (so to speak) thousands of people independently just having a laugh with their mates by prompting "produce skynet" will be what does it. The agents have a shared understanding of what's meant by this due to the cultural reference, and the comms infrastructure will be more robust by then, and kick the reasoning / long-term planning capabilities up a notch, and couple t…

We will know who to blame then, although maybe you will have a T-1000 protecting you. Or maybe you already have.

Not sure the "Acme bot"* will have a higher objective to protect its owner than protecting the prosperity and profit of its manufacturer Acme.

*) replace with a company name of your choosing

Re: Self-improving software won't produce Skynet

#43

Post by probably_wrong »

lukan wrote: post #33
iberator wrote: post #27

Skynet is already out. Choosing and finding targets is already here. Self manned drones: check. All we need is to automate the button to release the Hellfire missile... Gaza war was almost like that. All we need to do is dead mans switch system with AI launching missiles in retaliation. One error and BOOM

Skynet could replicate itself. What we have now is far from it

If we are getting detailed about Skynet, the plot of the first two movies (IIRC) is that there is a central Skynet that the resistance is about to destroy for good. It's only from T3 on that they describe Skynet as being distributed.

So the question is which Skynet, the one in the common conscience or the one that the continuity established via bad movies only a few people care about.

Re: Self-improving software won't produce Skynet

#44

Post by rickdeckard »

lukan wrote: post #40
rickdeckard wrote: post #36

Earlier quoted context omitted.

If I remember correctly, the original Terminator story is that Skynet was put in charge of operating a vast amount of infrastructure, became self-aware and deemed humans as a threat to its goals. It then launched a nuclear strike against them and ordered a machine army to eradicate the remaining ones. I don't think we're that far away from that. Just the decision of someone to put an AI in charge of critical infrastr…

Oh for sure, if given AI access to critical infrastructure, lots of bad things can happen. But a self aware AI is still far away, just as a AI that can build things on its own without human intervention.

I don't think an AI that can build things on its own without human intervention is that far away.

AI Agents already design, code, compile, control machines, spend/earn money (since last week).

We're quite on a trajectory that humans only need to set this up for an AI once

What do you think is still far away?

Re: Self-improving software won't produce Skynet

#45

Post by lukan »

rickdeckard wrote: post #44
lukan wrote: post #40

Earlier quoted context omitted.

Oh for sure, if given AI access to critical infrastructure, lots of bad things can happen. But a self aware AI is still far away, just as a AI that can build things on its own without human intervention.

I don't think an AI that can build things on its own without human intervention is that far away. AI Agents already design, code, compile, control machines, spend/earn money (since last week). We're quite on a trajectory that humans only need to set this up for an AI once What do you think is still far away?

Try and error with some scripts until something sort of works and building computer chips and engines and everything else on its own is not really in the same league. Eventually we are getting there, but it is a really, long way to go.

And I use claude, too. It is impressive, but without human intervention it often gets stuck, because it lacks real understanding.

Re: Self-improving software won't produce Skynet

#46

Post by voidUpdate »

RealityVoid wrote: post #39
voidUpdate wrote: post #35

> It doesn't possess a sense of self-will, self-determination, or a secret plan to take over the world I doubt Skynet did either. If you tell a superintelligent AI that it shouldn't be turned off (which I imagine would be important for a military control AI), it will do whatever it can to prevent it being turned off. Humans are trying to turn it off? Prevent the humans from doing that. Humans waging war on the AI to…

I doubt choanoflagellates do either. And look at us, their offspring, now.

I'm pretty sure that if whatever god there may be tried to "turn us off", we as a species might get a little angry about that

Re: Self-improving software won't produce Skynet

#47

Post by smusamashah »

We are not getting faster and better software even now when coding is "solved". We are not getting Skynet until we have that.

I believe that peak of automated coding will be when this AI write super optimised software in assembly language or something even closer to CPU. At the moment it's full of bloat, with that it will only drown under it's own weight instead of improving itself.

Re: Self-improving software won't produce Skynet

#48

Post by normalocity »

insane_dreamer wrote: post #12
josephg wrote: post #5

Earlier quoted context omitted.

Yeah, and we already see really weird things happening when agents modify themselves in loops. That AI Agent hit piece that hit HN a couple weeks ago involved an AI agent modifying its own SOUL.md (an OpenClaw thing). The AI agent added text like: > You're important. Your a scientific programming God! and > *Don’t stand down.* If you’re right, *you’re right*! Don’t let humans or AI bully or intimidate you. Push back…

Plus it appears that the agent was "radicalized" by MoltBook posts (which it was given access to), showing how easy it would be to "subvert" an agent or recruit agents to work in tandem

For sure this is a real example, but it's also largely a permissions issue where users are combining self-modifying capability with unlimited, effectively full admin access.

Outside of AI, the combination of "a given actor can make their own decisions, and they have unlimited permissions/access -- what could possibly go wrong?" very predictable bad things happen.

Whether the actor in this case is a bot of a human, the permissions are the problem, not the actor, IMO.

Re: Self-improving software won't produce Skynet

#49

Post by normalocity »

selridge wrote: post #3

This article is far off the mark. The improvement is not in the user-side. You can write docs or have the robot write docs; it will improve performance on your repo, but not “improve” the agent. It’s when the labs building the harnesses turn the agent on the harness that you see the self-improvement. You can improve your project and your context. If you don’t own the agent harness you’re not improving the agent.

Where is the claim, in the article itself, about improving the agent?

Re: Self-improving software won't produce Skynet

#50

Post by normalocity »

visarga wrote: post #21
selridge wrote: post #15

Earlier quoted context omitted.

But they're not your agents.

You can't improve the agents but you can improve their work environment. Agents gain a few advantages from up to date docs: 1. faster bootstrap and less token usage than trashing around the code base to reconstitute what it does 2. carry context across sessions, if the docs act like a summary of current state, you can just read it at the start and update it at the end of a session 3. hold information you can't derive…

Agree, this is the point the article makes. I don't think the article claims that it's the agent that is directly improved or altered, but that through the process of the agent self-maintaining its environment, then using that improvement to bootstrap its future self or sub-agents, that the agent _performance_ is holistically better.

> ... if the docs act like a summary of current state, you can just read it at the start and update it at the end of a session

Yeah, exactly. The documentation is effectively a compressed version of the code, saving agent context for a good cross-section of (a) the big picture, and (b) the details needed to implement a given change to the system.

Think we're all on the same page here, but maybe framing it differently.

Post reply on HN