Live data from Hacker News

Don't trust AI agents

nanoclaw.dev

71–80 of 211 posts

Re: Don't trust AI agents

#71
post #51

Earlier quoted context omitted.

Why not? What changed? It seems like a human factors thing. New people have to get up to speed. Doers become trainers.

Several related reasons working at once. The nature of work changed. The boundary between accidental and incidental complexity shifted (and it’s unclear whether this distinction still exists). Niche specializations within the field emerged. The way to structure and decompose projects changed dramatically (agile and stuff). One pathological example: if you’re running a server-based product, quite often what stands bet…

Hmm interesting, thanks! I was ready to argue but now I have to think, which is even better.

Re: Don't trust AI agents

#72

> OpenClaw has nearly half a million lines of code, 53 config files, and over 70 dependencies. This breaks the basic premise of open source security. Chromium has 35+ million lines, but you trust Google’s review processes. Most open source projects work the other way: they stay small enough that many eyes can actually review them. Nobody has reviewed OpenClaw’s 400,000 lines. This reminds me of a very common thing po…

Really it just continues to demonstrate that "code quality" is not and was not a requirement.

Even with supposedly expert human hand written software powering our products for the last decades, they frequently crash, have outages, and show all sorts of smaller bugs.

There are literally too many examples to count of video games being released with nigh-unplayable amounts of bugs and still selling millions and producing sequels.

Windows 95 and friends were famously buggy and crash prone yet produced one of the most valuable companies in the world.

Re: Don't trust AI agents

#73

This doesn’t really feel like enough guardrails to prevent the type of problems we’ve seen so far. For example an agent in a single container which has access to an email inbox, can still do a lot of damage if that agent goes off the rails. We agree this agent should not be trusted, yet the ideas proposed as a solution are insufficient. We need a fundamentally different approach. Also and this is just my ignorance ab…

Exactly! I installed nanoclaw to try to out. What is kinda crazy is that any extension like discord connection is done using a skill. A skill is a markdown file written in English to provide a step by step guide to an ai agent on how to do something. Basically, the extensions are written by claude code on the fly. Every install of nanoclaw is custom written code. There is nothing preventing the AI Agent from modifyin…

Yeah, the article's claim of having a low number of lines of code are disingenuous. Rather than writing some sort of plugin interface, it has "skills" that are a combination of pre-written typescript and English language instructions for how to modify the codebase to include the feature. I don't see how self-modifying code that uses a RNG to generate changes is going to be better for security than a proper plugin system. And everyone who uses Nanoclaw will have a customized version of it, so any bugs reported on Nanoclaw probably have a high chance of being closed as "can't reproduce". Why would you live this way?

Re: Don't trust AI agents

#74

> OpenClaw has nearly half a million lines of code, 53 config files, and over 70 dependencies. This breaks the basic premise of open source security. Chromium has 35+ million lines, but you trust Google’s review processes. Most open source projects work the other way: they stay small enough that many eyes can actually review them. Nobody has reviewed OpenClaw’s 400,000 lines. This reminds me of a very common thing po…

Respectfully, it feels like your position requires a very low, if not brain-dead level of incompetence on the part of LLM users, in order for your conclusion to be correct.

My personal anecdote: I used an LLM recently to basically vibe code a password manager.

Now, I’ve been a software engineer for 20 years. I’m very familiar with the process of code review and how to dive in to someone else’s code and get a feel for what’s happening, and how to spot issues. So when I say the LLM produced thousands of lines of working code in a very short time (probably at least 10 times faster than I would have done it), you could easily point at me and say “ha, look at ninkendo, he thinks more lines of code equals better!” And walk away feeling smug. Like, in your mind perhaps you think the result is an unmaintainable mess, and that the only thing I’m gushing about is the LOC count.

But here’s the thing: it actually did a good job. I was personally reviewing the code the whole time. And believe me when I say, the resulting product is actually good. The code is readable and obvious, it put clean separation of responsibilities into different crates (I’m using rust) and it wrote tons of tests, which actually validate behavior. It’s very near the quality level of what I would have been able to do. And I’m not half bad. (I’ve been coding in rust in particular, professionally for about 2 years now, on top of the ~20 years of other professional programming experience before that.)

My takeaway is that as a professional engineer, my job is going to be shifting from doing the actual code writing, to managing an LLM as if it’s my pair programming partner and it has the keyboard. I feel sad for the loss of the actual practice of coding, but it’s all over but the mourning at this point. This tech is here to stay.

Re: Don't trust AI agents

#77
Why do people take this article serious? It's just a wall of gibberish trying to make the product look more "secure" then others. It's not. It adds shallow secure looking random junk without tackling the core issues. Which are not solvable obviously.

Re: Don't trust AI agents

#78
post #66
post #14

My take is that agents should only take actions that you can recover from by default. You can gradually give it more permission and build guardrails such as extra LLM auditing, time boxed whitelisted domains etc. That's what I'm experimenting with https://github.com/lobu-ai/lobu 1. Don't let it send emails from your personal account, only let it draft email and share the link with you. 2. Use incremental snapshots an…

> 1. Don't let it send emails from your personal account, only let it draft email and share the link with you. Right now there's no way to have fine-grained draft/read only perms on most email providers or email clients. If it can read your email it can send email. > 3. Don't let your agents see any secret. Swap the placeholder secrets at your gateway and put human in the loop for secrets you care about. harder than…

> Right now there's no way to have fine-grained draft/read only perms on most email providers or email clients. If it can read your email it can send email.

> harder than you might think. openclaw found my browser cookies. (I ran it on a vm so no serious cookies found, but still)

You should never give any secrets to your agents, like your Gmail access tokens. Whenever agents needs to take an action, it should perform the request and your proxy should check if the action is allowed and set the secrets on the fly.

That means agents should not have access to internet without a proxy, which has proper guardrails. Openclaw doesn't have this model unfortunately so I had to build a multi-tenant version of Openclaw with a gateway system to implement these security boundaries.

Re: Don't trust AI agents

#79

Docker is not a security boundary. You’re one prompt injection away from handing over your gmail cookie.

No, but Podman is. The recent escapes at the actual container level have been pretty edge case. It's been some years since a general container escape has been found. Docker's CVE-2025-9074 was totally unnecessary and due to Docker being Docker.

No they have not been. There were at least 16 container escapes last year - at least 8 of them were at the runtime layer.

I personally spent way too much time looking at this in the past month:

https://nanovms.com/blog/last-year-in-container-security

runc: https://www.cve.org/CVERecord?id=CVE-2025-31133

nvidia: https://www.cve.org/CVERecord?id=CVE-2025-23266

runc: https://www.cve.org/CVERecord?id=CVE-2025-52565

youki: https://www.cve.org/CVERecord?id=CVE-2025-54867

Also, last time I checked podman uses runc by default.

Re: Don't trust AI agents

#80

> OpenClaw has nearly half a million lines of code, 53 config files, and over 70 dependencies. This breaks the basic premise of open source security. Chromium has 35+ million lines, but you trust Google’s review processes. Most open source projects work the other way: they stay small enough that many eyes can actually review them. Nobody has reviewed OpenClaw’s 400,000 lines. This reminds me of a very common thing po…

As lines of code become executable line noise, I swear that we need better approaches to developing software - either enforce better test coverage across the board, develop and use languages where it’s exceedingly hard to end up with improper states, or sandbox the frick out of runtimes and permissions.

Just as an example, I should easily be able to give each program an allowlist of network endpoints they’re allowed to use for inbound and outgoing traffic and sandbox them to specific directories and control resource access EASILY. Docker at least gets some of those right, but most desktop OSes feel like the Wild West even when compared to the permissions model of iOS.

Post reply on HN