Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Don't trust AI agents

nanoclaw.dev

Post reply on HN
Print view View original

201–210 of 211 posts

Re: Don't trust AI agents

#201

Post by ninkendo »

FEELmyAGI wrote: post #200
ninkendo wrote: post #198

Earlier quoted context omitted.

So if you’re just going to complain about me reinventing the wheel (even though I already explained this is literally an exercise to see what an LLM can do ), can we at least recognize the goalpost shift here? I’m just going to assume that you’ve completely conceded your original point then, since you have absolutely zero to say about whether you agree the LLM did good work or not. Since the moment I showed you its w…

If we're talking about goalposts, lets also recognize the Motte and Bailey of initial claims of "20 years professional software engineer actually good password manager" to "hee hee toy project testing how LLms work not complete not even for personal not recmmond for others" I don't think your password manager is good, and I don't think you think it's good either or you'd be using it. I'm not trolling. It's way cheape…

My goal is to mess around with an LLM, not just to mess around with a password manager. If you read literally any of my posts in this thread without the intention of throwing shade, maybe you would have gotten that point by now. I’ve certainly repeated it enough times.

> I don't think your password manager is good, and I don't think you think it's good either or you'd be using it.

Lol I am using it now though. In the time from yesterday’s post to now I have an iOS app, an iOS Password Autofill extension, a Mac app and the existing Linux CLI and Firefox browser extension. Automatically syncs conflict-free between everything too, using a simple web server for sync. It now covers every use case 1Password did for me, and no, none of the “rust password manager site:github.com” results do any of this.

It was an experiment to see if the effort of vibe coding a password manager would be easy enough that it would be worth doing, and guess what: resounding success. Cope more.

Re: Don't trust AI agents

#202

Post by RevEng »

aerhardt wrote: post #156

A question I've been asking myself and which I honestly want to put out there - and I apologize in advance, because you will see me repeat it in other threads, out of genuine curiosity: Does your life have so much friction that you need a digital agent to act on your behalf? Some of the use cases I saw on the OpenClaw website, like "checking me into a flight", are non-issues for me. I work in business automation, but…

I don't get it either. I even sat down to try it out and see what it's all about, but I can't think of a single thing in my life that I want an agent to automate.

Summarizing news? No, I'd rather just read it. Besides, it's hard to say what will or won't be interesting at any given moment.

Reply to emails? No, I want to make sure I say what I mean to say and I don't see why I would tell the whole story to an LLM just so it could rephrase it all.

Trade stocks? Dear God no! That's a good way to lose my life savings, and if the solution is to just put in a little then what's the point?

Every video and post I see talking about all the things they have automated with agents are things I would never want. Most of them describe content farms - look at what's hot on Twitter, generate a video, post to Reddit, etc. Others like preparing a morning summary are neat and all but so what? If my life was so hectic that I needed a personal assistant to take my calls and book my meetings, I'd hire one.

Seriously, what is the thing that agents can do that every ordinary person just can't live without?

Re: Don't trust AI agents

#203

Post by RevEng »

medi8r wrote: post #58
nkzd wrote: post #43

As someone who only coding agents at work, can someone describe their use case for claw type agent? What do you do with it?

I want to try one to be a bit of a personal coach. Remind me to do things and check in on goals. The memory / schedule / chat thing is enough and it wont need emails or anything more dangerous.

Wouldn't a weekly reminder if your calendar do this? I don't see how an agent improves this.

Re: Don't trust AI agents

#204

Post by RevEng »

jswelker wrote: post #175

As a fun thought experiment, when people complain about LLMs, I substitute the word "human" or "employee" into the sentence and see if it is equally true. "You can never really trust an LLM!" -> "You can never really trust an employee!" (Every IT department ever.) "LLMs make shit up." -> "Humans make shit up." (Wow very profound insight.)

While that will always be true, LLMs do it a lot more often and do so with confidence and poise. We have evolved ways to tell if someone is making shit up (which usually works); LLMs subvert this. We are also being sold the idea that these LLMs are some kind of super intelligence which isn't helping matters.

Re: Don't trust AI agents

#205

Post by FEELmyAGI »

ninkendo wrote: post #201
FEELmyAGI wrote: post #200

Earlier quoted context omitted.

If we're talking about goalposts, lets also recognize the Motte and Bailey of initial claims of "20 years professional software engineer actually good password manager" to "hee hee toy project testing how LLms work not complete not even for personal not recmmond for others" I don't think your password manager is good, and I don't think you think it's good either or you'd be using it. I'm not trolling. It's way cheape…

My goal is to mess around with an LLM, not just to mess around with a password manager. If you read literally any of my posts in this thread without the intention of throwing shade, maybe you would have gotten that point by now. I’ve certainly repeated it enough times. > I don't think your password manager is good, and I don't think you think it's good either or you'd be using it. Lol I am using it now though. In the…

> maybe you would have gotten that point by now.

I don't deny it. When I said "mess around with making a password manager suit your taste" feel free to append " while using and getting experience with an LLM". nothing changes.

im glad you're dogfooding your pw manager now. I'm glad vibe coding met your own standard of quality.

I will leave it as an exercise to you to think about all the edge cases and usability issues that have been solved in mainstream pw managers that you've never thought about. I hope you keep your LLM subscription active so you can fix them as they come up. And to keep up with updates from IOS and firefox as they come out.

Re: Don't trust AI agents

#206

Post by mrothroc »

ninkendo wrote: post #74
badsectoracula wrote: post #32

> OpenClaw has nearly half a million lines of code, 53 config files, and over 70 dependencies. This breaks the basic premise of open source security. Chromium has 35+ million lines, but you trust Google’s review processes. Most open source projects work the other way: they stay small enough that many eyes can actually review them. Nobody has reviewed OpenClaw’s 400,000 lines. This reminds me of a very common thing po…

Respectfully, it feels like your position requires a very low, if not brain-dead level of incompetence on the part of LLM users, in order for your conclusion to be correct. My personal anecdote: I used an LLM recently to basically vibe code a password manager. Now, I’ve been a software engineer for 20 years. I’m very familiar with the process of code review and how to dive in to someone else’s code and get a feel for…

My experience is similar to yours: LLMs can write excellent code, though you really have to drive them the right way. I use a harness to drive long-run autonomous agents to create production code. (Not open source, but it is an actual product used by companies.) The key is understanding how they fail, then driving them in a way that sidesteps this. If you let them run too long, they become self-contradictory. However, if you break long work into discrete chunks, then they can still fail, but it changes: they forget things. But that is a much easier thing to catch, because you can use things like lint or even a simple regex for "// TODO" to find them. Once you set up your pipeline to orchestrate the agents so the errors become easily detectable, and you have gates that check for those errors, the quality goes way, way up.

Re: Don't trust AI agents

#207

Post by mrothroc »

KronisLV wrote: post #80
badsectoracula wrote: post #32

> OpenClaw has nearly half a million lines of code, 53 config files, and over 70 dependencies. This breaks the basic premise of open source security. Chromium has 35+ million lines, but you trust Google’s review processes. Most open source projects work the other way: they stay small enough that many eyes can actually review them. Nobody has reviewed OpenClaw’s 400,000 lines. This reminds me of a very common thing po…

As lines of code become executable line noise, I swear that we need better approaches to developing software - either enforce better test coverage across the board, develop and use languages where it’s exceedingly hard to end up with improper states, or sandbox the frick out of runtimes and permissions. Just as an example, I should easily be able to give each program an allowlist of network endpoints they’re allowed…

I'm old enough to remember that engineers researching distributed systems had the same challenge. Everyone was trying to build 100% reliable nodes, which is impossible. Then Lamport came along and showed you could actually achieve your goal at the protocol/system level. What you're describing here is a workflow or pipeline, which is the analogy. As the LLMs produce artifacts, you have gates that verify the output deterministically. If the LLM breaks a rule, you either throw it out and reroll or you give it the feedback and let it revise. I do this in my own tooling and I get great results. One thing from the data: they are often pretty crap at revising, spending ridiculous time/tokens in a revision loop. I'm trying to find the right balance of reroll/revise myself.

Re: Don't trust AI agents

#208

Post by Noyra-X »

The trust problem is real, and I think the framing of "trust" vs "don't trust" misses the more useful question: trust for what, exactly? In our experience with small business automation, agents work well on bounded, well-defined tasks with clear success criteria and human checkpoints. The failures happen when people deploy agents on ambiguous, open-ended decisions and then walk away. The answer isn't less automation – it's better handoff design between agent and human. What specific failure modes have you seen that you think no amount of prompt engineering or guardrails can fix?

Re: Don't trust AI agents

#209

Post by hkonte »

K0balt wrote: post #117
badsectoracula wrote: post #32

> OpenClaw has nearly half a million lines of code, 53 config files, and over 70 dependencies. This breaks the basic premise of open source security. Chromium has 35+ million lines, but you trust Google’s review processes. Most open source projects work the other way: they stay small enough that many eyes can actually review them. Nobody has reviewed OpenClaw’s 400,000 lines. This reminds me of a very common thing po…

It’s definitely an issue when using coding assistants. If you are careful and specific you can keep things reasonable, but even when I am careful and do consolidattion / factoring passes, have rigid separation of concerns, etc I find that the LLM code is bigger than mine, mainly for two reasons: 1) more extensive inline documentation 2) more complete expression of the APIs across concerns, as well as stricter separat…

[dead]

Re: Don't trust AI agents

#210

Post by felix9527 »

The core problem this article surfaces is forensic: once the agent session ends, the evidence is whatever the vendor chose to log. Terminal scrollback is lossy, session logs are vendor-controlled, and "undo" only works if you catch it in time.

Certificate Transparency (RFC 6962) solved a structurally identical problem for TLS certificates after the DigiNotar incident. The insight: commit every action to an append-only Merkle tree where any third party can verify inclusion proofs — without trusting the log operator.

Applied to agents: - Inclusion proof: "this specific action was recorded at position N, and the log hasn't been rewritten" - Consistency proof: "between checkpoint A and B, the log only grew — nothing was removed or altered"

This gives you verification-based accountability, not trust-based logging. The difference matters: a signed receipt proves the signer said something happened. A Merkle inclusion proof proves it's part of a complete, append-only sequence — deletions are structurally detectable.

The AI agent ecosystem is having its DigiNotar moment weekly. We have the cryptographic tools to fix it. The question is whether we'll wait for a catastrophic incident to force adoption, like we did with CT.

Post reply on HN