Don't trust AI agents • Hacker News Board

Re: Don't trust AI agents

#121

Post by bitwize » Sat, Feb 28, 2026, 5:12 PM UTC

drujensen wrote: post #40

VladVladikoff wrote: post #5

This doesn’t really feel like enough guardrails to prevent the type of problems we’ve seen so far. For example an agent in a single container which has access to an email inbox, can still do a lot of damage if that agent goes off the rails. We agree this agent should not be trusted, yet the ideas proposed as a solution are insufficient. We need a fundamentally different approach. Also and this is just my ignorance ab…

Exactly! I installed nanoclaw to try to out. What is kinda crazy is that any extension like discord connection is done using a skill. A skill is a markdown file written in English to provide a step by step guide to an ai agent on how to do something. Basically, the extensions are written by claude code on the fly. Every install of nanoclaw is custom written code. There is nothing preventing the AI Agent from modifyin…

> Basically, the extensions are written by claude code on the fly. Every install of nanoclaw is custom written code.

"Every copy of Nanoclaw is personalized." So if I use it long enough will I see the Wario apparition?

Re: Don't trust AI agents

#122

Post by samiv » Sat, Feb 28, 2026, 5:13 PM UTC

badsectoracula wrote: post #32

> OpenClaw has nearly half a million lines of code, 53 config files, and over 70 dependencies. This breaks the basic premise of open source security. Chromium has 35+ million lines, but you trust Google’s review processes. Most open source projects work the other way: they stay small enough that many eyes can actually review them. Nobody has reviewed OpenClaw’s 400,000 lines. This reminds me of a very common thing po…

That's because they're an additive tool. Everything boils down to "adding" more code. But in the long term its not about how much code you can add but how little you can get away with. But this is an impossible task for the LLMs. How would you train one not to write code? What would the training data look like? Would that be all the lines of code that haven't been written?

Re: Don't trust AI agents

#123

Post by bigstrat2003 » Sat, Feb 28, 2026, 5:16 PM UTC

All this talk about sandboxing and permissions misses the obvious: since you can't trust the agents, don't freaking use them. It is utterly stupid to give an LLM access to run things on your computer, because nothing you do can stop it from hallucinating garbage that harms your system. The whole "agent" craze is the most incredible display of irresponsibility I have ever seen in this industry.

Re: Don't trust AI agents

#124

Post by ldng » Sat, Feb 28, 2026, 5:17 PM UTC

dasil003 wrote: post #114

smikhanov wrote: post #84

Earlier quoted context omitted.

That’s a lovely comment, thank you. If you’re keen to think about it more, consider the fact that the existing members of the project that’s being late are actually in not as much of an advantage compared to the new joiners, as it’s common to think. Yes, they know how the feature they work on relates to other features, but actually implementing that feature is very often mostly involves fighting with technology, wran…

I sort of agree that the surface area and incidental complexity of stacks give more space to plug more developers in than was true in the 70s and 80s. But I disagree strongly this invalidates Brooks Law. Certainly there are cases where adding people helps, especially if they are stronger engineers than the ones that are already there, but I’ve also seen way too many projects devolve into resourcing conversations when…

Plus, the "since there are more resources, let's add features" effect.

Re: Don't trust AI agents

#125

Post by tartoran » Sat, Feb 28, 2026, 5:20 PM UTC

samiv wrote: post #122

badsectoracula wrote: post #32

> OpenClaw has nearly half a million lines of code, 53 config files, and over 70 dependencies. This breaks the basic premise of open source security. Chromium has 35+ million lines, but you trust Google’s review processes. Most open source projects work the other way: they stay small enough that many eyes can actually review them. Nobody has reviewed OpenClaw’s 400,000 lines. This reminds me of a very common thing po…

That's because they're an additive tool. Everything boils down to "adding" more code. But in the long term its not about how much code you can add but how little you can get away with. But this is an impossible task for the LLMs. How would you train one not to write code? What would the training data look like? Would that be all the lines of code that haven't been written?

That’s not an impossible task with LLMs, you just have to mindfully architect the project with that in mind, hence take it slowly to design a good system, don’t outsource all thinking to LLMs.

Re: Don't trust AI agents

#126

Post by skeledrew » Sat, Feb 28, 2026, 5:23 PM UTC

bigstrat2003 wrote: post #123

All this talk about sandboxing and permissions misses the obvious: since you can't trust the agents, don't freaking use them . It is utterly stupid to give an LLM access to run things on your computer, because nothing you do can stop it from hallucinating garbage that harms your system. The whole "agent" craze is the most incredible display of irresponsibility I have ever seen in this industry.

> don't freaking use them

You can't tell people that. People see the obvious benefits of using agents, so the many will always take the leap regardless of what detractors say. Continually iterating on the security model and making it all transparent is the way to go.

Re: Don't trust AI agents

#127

Post by wiseowise » Sat, Feb 28, 2026, 5:30 PM UTC

Zak wrote: post #120

wiseowise wrote: post #54

Earlier quoted context omitted.

It’s all virtual virtue signaling. If you were to say this shit in the office, you’d be walked out pretty fast.

Who is signaling what virtues to whom in this context? When I see PG write something like that, it signals to me that he has embraced AI hype to the point that he is displaying poor taste and embracing a risky technical practice.

PG is signaling his AI bros, idk.

Re: Don't trust AI agents

#128

Post by skeledrew » Sat, Feb 28, 2026, 5:31 PM UTC

samiv wrote: post #122

badsectoracula wrote: post #32

> OpenClaw has nearly half a million lines of code, 53 config files, and over 70 dependencies. This breaks the basic premise of open source security. Chromium has 35+ million lines, but you trust Google’s review processes. Most open source projects work the other way: they stay small enough that many eyes can actually review them. Nobody has reviewed OpenClaw’s 400,000 lines. This reminds me of a very common thing po…

That's because they're an additive tool. Everything boils down to "adding" more code. But in the long term its not about how much code you can add but how little you can get away with. But this is an impossible task for the LLMs. How would you train one not to write code? What would the training data look like? Would that be all the lines of code that haven't been written?

TDD would help here, particularly if a human writes - or at least thoroughly reviews - the tests.

https://martinfowler.com/bliki/TestDrivenDevelopment.html

Re: Don't trust AI agents

#129

Post by andrei_says_ » Sat, Feb 28, 2026, 5:31 PM UTC

wiseowise wrote: post #54

gyomu wrote: post #41

Earlier quoted context omitted.

Yeah, it’s pretty wild. Even pg is tweeting stuff like “An experienced programmer told me he's now using AI to generate a thousand lines of code an hour.“ https://x.com/paulg/status/2026739899936944495 Like if you had told pg to his face in (pre AI) office hours “I’m producing a thousand lines of code an hour”, I’m pretty sure he’d have laughed and pointed out how pointless that metric was?

It’s all virtual virtue signaling. If you were to say this shit in the office, you’d be walked out pretty fast.

Maybe it depends on whose office? C-suite management who salivate after reducing software engineer headcount?

Re: Don't trust AI agents

#130

Post by ramoz » Sat, Feb 28, 2026, 5:35 PM UTC

Sytten wrote: post #56

I am a caveman, I don't understand the need for a personal assistant. What are you guys using it for?

Im terrible with email, so its be genuinely helpful for me there.

Excited to explore more use as time permits. Very optimistic based on email experience.

My next use case is personal notes system.