Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Don't trust AI agents

nanoclaw.dev

Post reply on HN
Print view View original

81–90 of 211 posts

Re: Don't trust AI agents

#81

Post by jimminyx »

drujensen wrote: post #40
VladVladikoff wrote: post #5

This doesn’t really feel like enough guardrails to prevent the type of problems we’ve seen so far. For example an agent in a single container which has access to an email inbox, can still do a lot of damage if that agent goes off the rails. We agree this agent should not be trusted, yet the ideas proposed as a solution are insufficient. We need a fundamentally different approach. Also and this is just my ignorance ab…

Exactly! I installed nanoclaw to try to out. What is kinda crazy is that any extension like discord connection is done using a skill. A skill is a markdown file written in English to provide a step by step guide to an ai agent on how to do something. Basically, the extensions are written by claude code on the fly. Every install of nanoclaw is custom written code. There is nothing preventing the AI Agent from modifyin…

Author and creator of NanoClaw here.

I did my best to communicate this but I guess it was still missed:

NanoClaw is not software that you should run out of the box. It is designed as a sort of framework that gives a solid foundation for you to build your own custom version.

The idea is not that you toggle on a bunch of features and run it. You should customize, review, and make sure that the code does what you want.

So you should not trust the coding agents that they didn't break the security model while adding discord. But after discord is added, you review the code changes and verify that it's correct. And because even after adding discord you still only have 2-3k loc, it's actually something you can realistically do.

Additionally, the skills were originally a bit ad-hoc. Now they are full working, tested and reviewed reference implementations. Code is separate from markdown files. When adding a new integration or messaging channel, the agent uses `git merge` to merge the changes in, rather than rewriting from scratch. Adding the first channel is fully deterministic. The agent only resolves merge conflicts if there are any.

Re: Don't trust AI agents

#82

Post by fvdessen »

VladVladikoff wrote: post #5

This doesn’t really feel like enough guardrails to prevent the type of problems we’ve seen so far. For example an agent in a single container which has access to an email inbox, can still do a lot of damage if that agent goes off the rails. We agree this agent should not be trusted, yet the ideas proposed as a solution are insufficient. We need a fundamentally different approach. Also and this is just my ignorance ab…

I think the best place to put barriers in place is at the mcp / tool layer. The email inbox mcp should have guardrails to prevent damage. Those guardrails could be fine grained permissions, but could also be an adversarial model dedicated to prevent misuse.

Re: Don't trust AI agents

#84

Post by smikhanov »

medi8r wrote: post #71
smikhanov wrote: post #59

Earlier quoted context omitted.

Several related reasons working at once. The nature of work changed. The boundary between accidental and incidental complexity shifted (and it’s unclear whether this distinction still exists). Niche specializations within the field emerged. The way to structure and decompose projects changed dramatically (agile and stuff). One pathological example: if you’re running a server-based product, quite often what stands bet…

Hmm interesting, thanks! I was ready to argue but now I have to think, which is even better.

That’s a lovely comment, thank you. If you’re keen to think about it more, consider the fact that the existing members of the project that’s being late are actually in not as much of an advantage compared to the new joiners, as it’s common to think.

Yes, they know how the feature they work on relates to other features, but actually implementing that feature is very often mostly involves fighting with technology, wrangling the entire stack into the shape you need.

In Brooks’s times the stack was paper-thin, almost nonexistent. In modern times it’s not, and adding someone who knows the technology, but doesn’t have the domain knowledge related to your feature still helps you. It doesn’t slow you down.

One may argue that I’m again pointing to the difference between accidental and incidental complexity, and my argument is essentially “accidental complexity takes over”, but accidental complexity actually does influence your feature too, by defining what’s possible and what’s not.

Some good thoughts (not mine) on the modern boundary between accidental and incidental complexity: https://danluu.com/essential-complexity/

Re: Don't trust AI agents

#85

Post by mihaelm »

jsheard wrote: post #19
scandinavian wrote: post #16

Earlier quoted context omitted.

I did a cloc check on it and it does seem to have 800k lines of typescript. So unless they are vendoring dependencies it's actually as insane as it sounds.

Christ their repo is an absolute nightmare. There's new issues and PRs being posted practically every minute, and I assume 99% of them are from agents given the target demographic. Just full-auto vibeslop from all barrels 24/7. Even if we count the repos whole lifetime, including when it wasn't so active, the averages are still absurd. 96 days / (4,239+9,170) issues = one issue every 10 minutes 96 days / (5,082+10,22…

5000+ open PRs is pretty insane, that's the highest I've seen. How do you even keep track of this? We'll really need trust management systems like vouch (https://github.com/mitchellh/vouch/tree/main) for open source projects in the future to help with reducing noise.

Re: Don't trust AI agents

#86

Post by manoDev »

gyomu wrote: post #41
badsectoracula wrote: post #32

> OpenClaw has nearly half a million lines of code, 53 config files, and over 70 dependencies. This breaks the basic premise of open source security. Chromium has 35+ million lines, but you trust Google’s review processes. Most open source projects work the other way: they stay small enough that many eyes can actually review them. Nobody has reviewed OpenClaw’s 400,000 lines. This reminds me of a very common thing po…

Yeah, it’s pretty wild. Even pg is tweeting stuff like “An experienced programmer told me he's now using AI to generate a thousand lines of code an hour.“ https://x.com/paulg/status/2026739899936944495 Like if you had told pg to his face in (pre AI) office hours “I’m producing a thousand lines of code an hour”, I’m pretty sure he’d have laughed and pointed out how pointless that metric was?

They need to keep the musical chairs going.

Re: Don't trust AI agents

#87

Post by CuriouslyC »

badsectoracula wrote: post #32

> OpenClaw has nearly half a million lines of code, 53 config files, and over 70 dependencies. This breaks the basic premise of open source security. Chromium has 35+ million lines, but you trust Google’s review processes. Most open source projects work the other way: they stay small enough that many eyes can actually review them. Nobody has reviewed OpenClaw’s 400,000 lines. This reminds me of a very common thing po…

The lines of code thing isn't because we think it's a good metric, but because we have literally no good metric and we're trying to communicate a velocity difference. If you invent a new metric that doesn't have LoC's problems while being as easy to use, you'll be a household name in software engineering in short order.

Also, AI is better at reading code than writing it, but the overhead to FIND code is real.

Re: Don't trust AI agents

#88

Post by justonceokay »

I have twice encountered a phone tree AI agent saying my problem could not be solved and then ending the call. One was for PayPal fraud and the other was for closing an unused bank account.

For right now my trick is to say I have a problem that is more recognizable and mundane to the ai (i .e. lie) and then when I finally get the human just say “oh that was a bunch of hooey here’s what I’m trying to do”. For PayPal that involved asking for help with a business tax that did not exist. For my bank it involved asking to /open/ a new account. Obviously th AI wants to help me open an account, even if my intention is to close one.

That will only work for so long but it’s something

Re: Don't trust AI agents

#89

Post by dimitri-vs »

TeeWEE wrote: post #23

Do you trust your employees? Do you trust a contracter? Do you trust other people? AI is similar to a person you dont know that does work for you. Probably AI is a bit more trustworthy than a random person. But a company, needs to let employees take ownership of their work, and trust them. Allow them to make mistakes. Isnt AI no different?

As others have said: accountability

Re: Don't trust AI agents

#90

Post by solfox »

jimminyx wrote: post #81
drujensen wrote: post #40

Earlier quoted context omitted.

Exactly! I installed nanoclaw to try to out. What is kinda crazy is that any extension like discord connection is done using a skill. A skill is a markdown file written in English to provide a step by step guide to an ai agent on how to do something. Basically, the extensions are written by claude code on the fly. Every install of nanoclaw is custom written code. There is nothing preventing the AI Agent from modifyin…

Author and creator of NanoClaw here. I did my best to communicate this but I guess it was still missed: NanoClaw is not software that you should run out of the box. It is designed as a sort of framework that gives a solid foundation for you to build your own custom version. The idea is not that you toggle on a bunch of features and run it. You should customize, review, and make sure that the code does what you want.…

So, nanoclaw requires agents to code extensions on the fly to get to feature parity with openclaw… and you're celebrating nanoclaw having fewer LOC. How's the code smell after nanoclaw gets to feature parity?
Post reply on HN