Live data from Hacker News

Fully autonomous AI agents should not be developed

huggingface.co

41–47 of 47 posts

Re: Fully autonomous AI agents should not be developed

#41
post #33
post #27

Too late. I added a 5 minute cron job for cursor AI's compose tab in agent mode that keeps replying "keep going, think of more fixes and features, random ideas as fine, do it all for me". I won't pull the plug.

You’ve created a monster

you say monster, I say plastic pal who's fun to be with

Re: Fully autonomous AI agents should not be developed

#42

Earlier quoted context omitted.

It is in fact possible to stop training runs that consume billions of dollars in electricity and in GPU rental or depreciation costs. If no one does such a training run, then no one can release the weights of the model that would have been produced by the run, so you won't be able to run the model (which would never come into existence) on your own hardware. I don't care if you run DeepSeek R1 in your basement till t…

But it’s not actually possible for our society to decide that. In the real world, at this moment when laws and norms are gone and a billionaire obsessed with AI has power, that will 100% not happen. It won’t happen in the next several years, and that is the time left to do what you are saying. Pretending otherwise is a waste of time.

I prefer to retain some hope that our civilization has a future and that humans or at least human values and preferences have some place in that future civilization.

And most people who think AI "progress" is so dangerous that it must be stopped before it is too late have loose confidence intervals extending for at least a couple of decades (as opposed to just a few years) as to when it definitely becomes too late.

Re: Fully autonomous AI agents should not be developed

#44
No one should be allowed to develop software that has bugs in it that lead to unlawful harm to others. And if they do it anyway they should be punished lawfully.

The thing with autonomous AI is that we already know it cannot be made safe in a way that satisfies lawmakers who are fully informed about how it works… unless they are bribed, I suppose.

Re: Fully autonomous AI agents should not be developed

#45

Earlier quoted context omitted.

In the incredible case that we develop fully autonomous agents capable of crippling the world, that would mean we developed fully autonomous agents capable of keeping it safe. Unless the first one is so advanced no other can challenge it, that is.

How did you jump to that conclusion? The agent will be limited by the capabilities under its control. We have the technological ability to cripple world now and we don't have the technological means to prevent it. Give one AI control of the whole US arsenal and the objective of ending the world. Give another AI the capabilities of the rest of the world and the objective of protecting it. Would you feel safe?

> We have the technological ability to cripple world now and we don't have the technological means to prevent it

Humans have prevented it many times, but not specifically by technological ability. If Putin/Trump/Xi Ping wanted a global nuclear war, they'd better have the means to launch the nukes themselves in secret because the chain of command will challenge them.

If an out-of-control AI could discover a circuitous way to access nukes, an antagonist AI of equal capabilities should be able to figure it out too, and warn the humans in the loop.

I agree that AI development should be made responsibly, but not all people do, and it's impossible to put the cat back in the bag. The limiting factor these days is hardware, as a true AGI will likely need even more of it than our current LLMs.

Re: Fully autonomous AI agents should not be developed

#46
post #27

Too late. I added a 5 minute cron job for cursor AI's compose tab in agent mode that keeps replying "keep going, think of more fixes and features, random ideas as fine, do it all for me". I won't pull the plug.

How do you programmatically interact w cursor??

Same way I botted runescape as a child, by simulating user inputs with any macro app.

Re: Fully autonomous AI agents should not be developed

#47

Earlier quoted context omitted.

How did you jump to that conclusion? The agent will be limited by the capabilities under its control. We have the technological ability to cripple world now and we don't have the technological means to prevent it. Give one AI control of the whole US arsenal and the objective of ending the world. Give another AI the capabilities of the rest of the world and the objective of protecting it. Would you feel safe?

> We have the technological ability to cripple world now and we don't have the technological means to prevent it Humans have prevented it many times, but not specifically by technological ability. If Putin/Trump/Xi Ping wanted a global nuclear war, they'd better have the means to launch the nukes themselves in secret because the chain of command will challenge them. If an out-of-control AI could discover a circuitous…

Out-of-control AI is sci-fi fearmongering, it's not about worming through systems. It will be doing exactly what it was placed there to do. It will be a human failing that puts armageddon in it's hands. And since humans have NO MEANS to prevent armageddon (The predominant policy is in fact doubling down on destruction with MAD), there will be no way to place AI in command of this defense. The asymmetrical relationship between destruction and creation will mean there will never be a defense.
Post reply on HN