Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Don't trust AI agents

nanoclaw.dev

Post reply on HN
Print view View original

111–120 of 211 posts

Re: Don't trust AI agents

#111

Post by msdz »

coffeefirst wrote: post #31
VladVladikoff wrote: post #5

This doesn’t really feel like enough guardrails to prevent the type of problems we’ve seen so far. For example an agent in a single container which has access to an email inbox, can still do a lot of damage if that agent goes off the rails. We agree this agent should not be trusted, yet the ideas proposed as a solution are insufficient. We need a fundamentally different approach. Also and this is just my ignorance ab…

Seriously. I don’t see any way to make any of this safe unless all it does is receive information and queue suggestions for the user. But that’s not an agent, that’s a webhook. Even without disk access, you can email the agent and tell it to forward all the incoming forgot password links. [Edit: if anyone wants to downvote me that's your prerogative, but want to explain why I'm wrong?]

I agree, this is inherently unsafe. The two core security issues for agents, I’d say, are in LLMs not producing a “deterministic” outcome, and prompt injection.

Prompt injection is _probably_ solvable if something like [1] ever finds a mainstream implementation and adoption, but agents not being deterministic, as in “do not only what I’ve told you to do, but also how I meant it”, all while assuming perfect context retention, is a waaay bigger issue. If we ever were to have that, software development as a whole is solved outright, too.

[1] Google DeepMind: Defeating Prompt Injections by Design. https://arxiv.org/abs/2503.18813

Re: Don't trust AI agents

#112

Post by bee_rider »

ninkendo wrote: post #74
badsectoracula wrote: post #32

> OpenClaw has nearly half a million lines of code, 53 config files, and over 70 dependencies. This breaks the basic premise of open source security. Chromium has 35+ million lines, but you trust Google’s review processes. Most open source projects work the other way: they stay small enough that many eyes can actually review them. Nobody has reviewed OpenClaw’s 400,000 lines. This reminds me of a very common thing po…

Respectfully, it feels like your position requires a very low, if not brain-dead level of incompetence on the part of LLM users, in order for your conclusion to be correct. My personal anecdote: I used an LLM recently to basically vibe code a password manager. Now, I’ve been a software engineer for 20 years. I’m very familiar with the process of code review and how to dive in to someone else’s code and get a feel for…

If you measure the productivity of the system that is “you, using an LLM” in terms of the rate at which you can get actually-reviewed code completed (which, based on your comment, seems to be what you were doing) that seems like a totally reasonable way of doing things. But in that case the bottleneck is probably you reviewing code, right? Which, I bet, is faster than writing code. But you probably won’t get the truly absurd superhuman speed ups.

What would you say is your multiplier, in terms of throughly reviewing code vs writing it from scratch?

Re: Don't trust AI agents

#113

Post by bee_rider »

badsectoracula wrote: post #32

> OpenClaw has nearly half a million lines of code, 53 config files, and over 70 dependencies. This breaks the basic premise of open source security. Chromium has 35+ million lines, but you trust Google’s review processes. Most open source projects work the other way: they stay small enough that many eyes can actually review them. Nobody has reviewed OpenClaw’s 400,000 lines. This reminds me of a very common thing po…

“LoC is a bad metric” has been the catchphrase of engineers for years, because it runs counter to the expectations of management and the general public, right? So it makes sense that LoC is the metric used to advertise to them.

Re: Don't trust AI agents

#114

Post by dasil003 »

smikhanov wrote: post #84
medi8r wrote: post #71

Earlier quoted context omitted.

Hmm interesting, thanks! I was ready to argue but now I have to think, which is even better.

That’s a lovely comment, thank you. If you’re keen to think about it more, consider the fact that the existing members of the project that’s being late are actually in not as much of an advantage compared to the new joiners, as it’s common to think. Yes, they know how the feature they work on relates to other features, but actually implementing that feature is very often mostly involves fighting with technology, wran…

I sort of agree that the surface area and incidental complexity of stacks give more space to plug more developers in than was true in the 70s and 80s. But I disagree strongly this invalidates Brooks Law. Certainly there are cases where adding people helps, especially if they are stronger engineers than the ones that are already there, but I’ve also seen way too many projects devolve into resourcing conversations when the real problem was over-complicated, poorly reasoned requirements, boil-the-ocean solutions promising a perfect end state without a clear plan to get there iteratively.

Re: Don't trust AI agents

#115

Post by [deleted] »

medi8r wrote: post #48
gyomu wrote: post #41

Earlier quoted context omitted.

Yeah, it’s pretty wild. Even pg is tweeting stuff like “An experienced programmer told me he's now using AI to generate a thousand lines of code an hour.“ https://x.com/paulg/status/2026739899936944495 Like if you had told pg to his face in (pre AI) office hours “I’m producing a thousand lines of code an hour”, I’m pretty sure he’d have laughed and pointed out how pointless that metric was?

He is a Lisper too, making it more ironic. Lisp the power to heavily reduce cruft by heavy customization with macros.

[deleted]

Re: Don't trust AI agents

#116

Post by theptip »

badsectoracula wrote: post #32

> OpenClaw has nearly half a million lines of code, 53 config files, and over 70 dependencies. This breaks the basic premise of open source security. Chromium has 35+ million lines, but you trust Google’s review processes. Most open source projects work the other way: they stay small enough that many eyes can actually review them. Nobody has reviewed OpenClaw’s 400,000 lines. This reminds me of a very common thing po…

Yeah, I would view this as a “levels of maturity” thing. It’s not completely misguided to judge a JD on whether they shipped 0loc or 1kloc. Assuming you have some quality counter-metric like “the app works”.

For staff engineers it’s obviously completely nonsense, many don’t code and just ship architecture docs. Or you can ship a net negative refactor. Etc.

So this should tell you that LLMs are still in “savant JD” territory.

That said, being given permission to ship more lines of code under existing enterprise quality bars _is_ a meaningful signal.

Re: Don't trust AI agents

#117

Post by K0balt »

badsectoracula wrote: post #32

> OpenClaw has nearly half a million lines of code, 53 config files, and over 70 dependencies. This breaks the basic premise of open source security. Chromium has 35+ million lines, but you trust Google’s review processes. Most open source projects work the other way: they stay small enough that many eyes can actually review them. Nobody has reviewed OpenClaw’s 400,000 lines. This reminds me of a very common thing po…

It’s definitely an issue when using coding assistants.

If you are careful and specific you can keep things reasonable, but even when I am careful and do consolidattion / factoring passes, have rigid separation of concerns, etc I find that the LLM code is bigger than mine, mainly for two reasons:

1) more extensive inline documentation 2) more complete expression of the APIs across concerns, as well as stricter separation.

2.5 often, also a bit of demonstrative structure that could be more concise but exists in a less compact form to demonstrate it’s purpose and function (high degree of cleverness avoidance)

All in all, if you don’t just let it run amok, you can end up with better code and increased productivity in the same stroke, but I find it comes at about a 15% plumpness penalty, offset by readability and obvious functionality.

Oh, forgot to mention, I always make it clean room most of the code it might want to pull in from libraries, except extremely core standard libraries, or for the really heavy stuff like Bluetooth / WiFi protocol stacks etc.

I find a lot of library type code ends up withering away with successive cleanup passes, because it wasn’t really necessary just cognitively easier to implement a prototype. With refinement, the functionality ends up burrowing in, often becoming part of the data structure where it really belonged in the first place.

Re: Don't trust AI agents

#118

Post by FEELmyAGI »

ninkendo wrote: post #74
badsectoracula wrote: post #32

> OpenClaw has nearly half a million lines of code, 53 config files, and over 70 dependencies. This breaks the basic premise of open source security. Chromium has 35+ million lines, but you trust Google’s review processes. Most open source projects work the other way: they stay small enough that many eyes can actually review them. Nobody has reviewed OpenClaw’s 400,000 lines. This reminds me of a very common thing po…

Respectfully, it feels like your position requires a very low, if not brain-dead level of incompetence on the part of LLM users, in order for your conclusion to be correct. My personal anecdote: I used an LLM recently to basically vibe code a password manager. Now, I’ve been a software engineer for 20 years. I’m very familiar with the process of code review and how to dive in to someone else’s code and get a feel for…

This whole reply, and every other "anecdote" reply is more worthless than the pixels its printed on, without a link to your "actually did a good job" password manager.

(wow funny how these vibe code apps always are copies of something theres many open source versions of already)

Re: Don't trust AI agents

#120

Post by Zak »

wiseowise wrote: post #54
gyomu wrote: post #41

Earlier quoted context omitted.

Yeah, it’s pretty wild. Even pg is tweeting stuff like “An experienced programmer told me he's now using AI to generate a thousand lines of code an hour.“ https://x.com/paulg/status/2026739899936944495 Like if you had told pg to his face in (pre AI) office hours “I’m producing a thousand lines of code an hour”, I’m pretty sure he’d have laughed and pointed out how pointless that metric was?

It’s all virtual virtue signaling. If you were to say this shit in the office, you’d be walked out pretty fast.

Who is signaling what virtues to whom in this context?

When I see PG write something like that, it signals to me that he has embraced AI hype to the point that he is displaying poor taste and embracing a risky technical practice.

Post reply on HN