Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Don't trust AI agents

nanoclaw.dev

Post reply on HN
Print view View original

31–40 of 211 posts

Re: Don't trust AI agents

#31

Post by coffeefirst »

VladVladikoff wrote: post #5

This doesn’t really feel like enough guardrails to prevent the type of problems we’ve seen so far. For example an agent in a single container which has access to an email inbox, can still do a lot of damage if that agent goes off the rails. We agree this agent should not be trusted, yet the ideas proposed as a solution are insufficient. We need a fundamentally different approach. Also and this is just my ignorance ab…

Seriously. I don’t see any way to make any of this safe unless all it does is receive information and queue suggestions for the user.

But that’s not an agent, that’s a webhook.

Even without disk access, you can email the agent and tell it to forward all the incoming forgot password links.

[Edit: if anyone wants to downvote me that's your prerogative, but want to explain why I'm wrong?]

Re: Don't trust AI agents

#32

Post by badsectoracula »

> OpenClaw has nearly half a million lines of code, 53 config files, and over 70 dependencies. This breaks the basic premise of open source security. Chromium has 35+ million lines, but you trust Google’s review processes. Most open source projects work the other way: they stay small enough that many eyes can actually review them. Nobody has reviewed OpenClaw’s 400,000 lines.

This reminds me of a very common thing posted here (and elsewhere, e.g. Twitter) to promote how good LLMs are and how they're going to take over programming: the number of lines of code they produce.

As if every competent programmer suddenly forgot the whole idea of LoC being a terrible metric to measure productivity or -even worse- software quality. Or the idea that software is meant to written to be readable (to water down "Programs are meant to be read by humans and only incidentally for computers to execute" a bit). Or even Bill Gates' infamous "Measuring programming progress by lines of code is like measuring aircraft building progress by weight".

Even if you believe that AI will -somehow- take over the whole task completely so that no human will need to read code anymore, there is still the issue that the AIs will need to be able to read that code and AIs are much worse at doing that (especially with their limited context sizes) than generating code, so it still remains a problem to use LoCs as such a measure even if all you care are about the driest "does X do the thing i want?" aspect, ignoring other quality concerns.

Re: Don't trust AI agents

#33

Post by arnvald »

TeeWEE wrote: post #23

Do you trust your employees? Do you trust a contracter? Do you trust other people? AI is similar to a person you dont know that does work for you. Probably AI is a bit more trustworthy than a random person. But a company, needs to let employees take ownership of their work, and trust them. Allow them to make mistakes. Isnt AI no different?

It’s totally different. People have to obey laws and contracts because there are consequences if they don’t, there are fines, arbitrage, courts.

What happens if AI agent you run causes a lot of damage? The best you can do is to turn it off

Re: Don't trust AI agents

#34

Post by benatkin »

smallpipe wrote: post #3

Docker is not a security boundary. You’re one prompt injection away from handing over your gmail cookie.

No, but Podman is. The recent escapes at the actual container level have been pretty edge case. It's been some years since a general container escape has been found. Docker's CVE-2025-9074 was totally unnecessary and due to Docker being Docker.

Re: Don't trust AI agents

#35

Post by ramoz »

TeeWEE wrote: post #23

Do you trust your employees? Do you trust a contracter? Do you trust other people? AI is similar to a person you dont know that does work for you. Probably AI is a bit more trustworthy than a random person. But a company, needs to let employees take ownership of their work, and trust them. Allow them to make mistakes. Isnt AI no different?

Yes, it is different.

An AI actions and reasons through probabilistic methods - creating a lot more risk than a human with memory, emotions, and rationale thinking.

We can’t trust AI to do any sensitive work because they consistently f up. With & without malicious intent, whether it’s a fault of their attention mechanisms, reward hacking, instrumental convergence, etc all very different than what causes most human f ups.

Re: Don't trust AI agents

#36

Post by nemo44x »

I’ve seen skills, etc haphazardly being launched with no constraints or guardrails. That more or less have admin access and can take actions that are not reversible.

It’s the monkey with a gun meme.

Re: Don't trust AI agents

#37

Post by rco8786 »

rdtsc wrote: post #20

> The container boundary is the hard security layer — the agent can’t escape it regardless of configuration I thought containers were never a proper hard security barrier? It’s barrier so better than not having it, if course.

In the sense that nothing is truly a "proper" hard security barrier outside of maybe airgapping, sure. But containerization is typically a trusted security measure.

Re: Don't trust AI agents

#38

Post by juggle-anyhow »

TeeWEE wrote: post #23

Do you trust your employees? Do you trust a contracter? Do you trust other people? AI is similar to a person you dont know that does work for you. Probably AI is a bit more trustworthy than a random person. But a company, needs to let employees take ownership of their work, and trust them. Allow them to make mistakes. Isnt AI no different?

Exactly, and I would never turn over my email or computer over to a contractor or anyone really. They get their own environment, email etc. Their actions stay as their actions.

Re: Don't trust AI agents

#39

Post by [deleted] »

jrecyclebin wrote: post #17
float4 wrote: post #13

Earlier quoted context omitted.

Wouldn't you get >50% of the usefulness and 0% of the risk if you add read+draft permissions for the email connection through a proxy or oauth permissions? Then your claw can draft replies and you have to manually review+send. It's not a perfect PA that way, but could still be better than doing everything yourself for the vast majority of people who don't have a PA anyway? It feels like, just like SWEs do with AI, we…

Agent can still "forgot password" on many accounts. Or magic link.

[deleted]

Re: Don't trust AI agents

#40

Post by drujensen »

VladVladikoff wrote: post #5

This doesn’t really feel like enough guardrails to prevent the type of problems we’ve seen so far. For example an agent in a single container which has access to an email inbox, can still do a lot of damage if that agent goes off the rails. We agree this agent should not be trusted, yet the ideas proposed as a solution are insufficient. We need a fundamentally different approach. Also and this is just my ignorance ab…

Exactly!

I installed nanoclaw to try to out.

What is kinda crazy is that any extension like discord connection is done using a skill.

A skill is a markdown file written in English to provide a step by step guide to an ai agent on how to do something.

Basically, the extensions are written by claude code on the fly. Every install of nanoclaw is custom written code.

There is nothing preventing the AI Agent from modifying the core nanoclaw engine.

It’s ironic that the article says “Don’t trust AI agents” but then uses skills and AI to write the core extensions of nanoclaw.

Post reply on HN