Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Don't trust AI agents

nanoclaw.dev

Post reply on HN
Print view View original

131–140 of 211 posts

Re: Don't trust AI agents

#131

Post by andrei_says_ »

hirako2000 wrote: post #62
badsectoracula wrote: post #32

> OpenClaw has nearly half a million lines of code, 53 config files, and over 70 dependencies. This breaks the basic premise of open source security. Chromium has 35+ million lines, but you trust Google’s review processes. Most open source projects work the other way: they stay small enough that many eyes can actually review them. Nobody has reviewed OpenClaw’s 400,000 lines. This reminds me of a very common thing po…

More people believe a software developer job and value is in the lines of code produced. Perhaps over half of engineering managers unconsciously or admittedly take the amount of PR and code additions as a rough but valid measure of productivity. I recall a role in architecture, senior director asking me how come a principal engineer didn't commit any code in 2 weeks, that we pay principals a fortune. I asked that bri…

> Leaders haven't all gone insane, they answer to difficult questions with the narrative of least resistance.

Brilliantly said. I’d like to add - a distorted narrative actively, intentionally established and maintained by the entities profiting from the technology. Quite similar to the crypto scam hype cycle.

Re: Don't trust AI agents

#132

Post by Eggpants »

I’m using this but using gpt-oss-120B instead of a cloud service. It has been eye opening when I realized the LLM is beings used as a compiler. I asked it to add apple iMessage and apple notes support as I I rather have long responses, like write me a program ideas, not fill my iMessage history. The local LLM, which I believe has limited bash training data, does pretty well.

For example: I enjoy industrial music and asked it for the tour data of the band KMFDM which returned they will be in Las Vegas in April for a festival(Sick new world). This festival has something like 20 bands most of which I never heard of. I asked nanoclaw to search all of the band list and generate a listing grouped by the type of music they play: Industrial, rap, etc. It did a good job based on bands I do know.

I was pleased as I certainly did not want to do 20 band web searches by hand. It’s still at a bar trick level. It gives me hope that an upgraded agent based Siri-like OS component could actually be useful from time to time.

Re: Don't trust AI agents

#133

Post by simgt »

samiv wrote: post #122
badsectoracula wrote: post #32

> OpenClaw has nearly half a million lines of code, 53 config files, and over 70 dependencies. This breaks the basic premise of open source security. Chromium has 35+ million lines, but you trust Google’s review processes. Most open source projects work the other way: they stay small enough that many eyes can actually review them. Nobody has reviewed OpenClaw’s 400,000 lines. This reminds me of a very common thing po…

That's because they're an additive tool. Everything boils down to "adding" more code. But in the long term its not about how much code you can add but how little you can get away with. But this is an impossible task for the LLMs. How would you train one not to write code? What would the training data look like? Would that be all the lines of code that haven't been written?

Well they will train on my Claude Code sessions for a start. I spend a lot of time asking it to remove unnecessary code that was produced, I'm not the only one.

Re: Don't trust AI agents

#134

Post by ninkendo »

FEELmyAGI wrote: post #118
ninkendo wrote: post #74

Earlier quoted context omitted.

Respectfully, it feels like your position requires a very low, if not brain-dead level of incompetence on the part of LLM users, in order for your conclusion to be correct. My personal anecdote: I used an LLM recently to basically vibe code a password manager. Now, I’ve been a software engineer for 20 years. I’m very familiar with the process of code review and how to dive in to someone else’s code and get a feel for…

This whole reply, and every other "anecdote" reply is more worthless than the pixels its printed on, without a link to your "actually did a good job" password manager. (wow funny how these vibe code apps always are copies of something theres many open source versions of already)

Ugh, you made me spend the 20 minutes it takes to spin up a new github account to share this (my existing one uses my real name and I don't really want to doxx myself that much. Not that it's a huge deal, my real identity and the "ninkendo" handle have been intertwined a lot in the past.)

https://github.com/ninkendo84/kenpass

I'm not saying it's perfect, there's some things I would've done differently in the code. It's also not even close to done/complete, but it has:

- A background agent that keeps the unsealed vault in-memory

- A CLI for basic CRUD

- Encryption for the on-disk layout that uses reasonably good standards (pbkdf2 with 600,000 iterations, etc)

- Sync with any server that supports webdav+etags+mTLS auth (I just take care of this out of band, I had the LLM whip up the nginx config though)

- A very basic firefox extension that will fill passwords (I only did 2 or 3 rounds of prompting for that one, I'm going to add more later)

Every commit that was vibe-coded contains the prompt I gave to Codex, so you can reproduce the entire development yourself if you want... A few of the prompts were actually constructed by ChatGPT 5.2. (It started out as a conversation with ChatGPT about what the sync protocol would look like for a password manager in a way that is conflict-free, and eventually I just said "ok give me a prompt I can give to codex to get a basic repo going" and then I just kept building from there.)

Also full disclosure, it had originally put all the code for each crate in a single lib.rs, so I had it split the crates into more modules for readability, before I published but after I made the initial comment in this thread.

I haven't decided if I want to take this all the way to something I actually use full time, yet. I just saw the 1password subscription increase and decided "wait what if I just vibe-coded my own?" (I also don't think it's even close to worthy of a "Show HN", because literally anybody could have done this.)

Re: Don't trust AI agents

#135

Post by ruszki »

gyomu wrote: post #41
badsectoracula wrote: post #32

> OpenClaw has nearly half a million lines of code, 53 config files, and over 70 dependencies. This breaks the basic premise of open source security. Chromium has 35+ million lines, but you trust Google’s review processes. Most open source projects work the other way: they stay small enough that many eyes can actually review them. Nobody has reviewed OpenClaw’s 400,000 lines. This reminds me of a very common thing po…

Yeah, it’s pretty wild. Even pg is tweeting stuff like “An experienced programmer told me he's now using AI to generate a thousand lines of code an hour.“ https://x.com/paulg/status/2026739899936944495 Like if you had told pg to his face in (pre AI) office hours “I’m producing a thousand lines of code an hour”, I’m pretty sure he’d have laughed and pointed out how pointless that metric was?

I don't understand how some people decide here, who the good programmers are. A lot of people reminded me a guy from West Palm Beach, who votes on elections solely on the principle of who has more "fame". Paul Graham is famous for sure (at least in HN circles), but I never considered him an exceptional or good programmer at all. So I always interpreted his words with a hefty amount of grain of salt. And sometimes some comments have a list of "good" coders, then half of them is like these famous, but not good ones.

Re: Don't trust AI agents

#136

Post by swid »

Doublon wrote: post #92
buremba wrote: post #14

My take is that agents should only take actions that you can recover from by default. You can gradually give it more permission and build guardrails such as extra LLM auditing, time boxed whitelisted domains etc. That's what I'm experimenting with https://github.com/lobu-ai/lobu 1. Don't let it send emails from your personal account, only let it draft email and share the link with you. 2. Use incremental snapshots an…

I'd like to try a pattern where agents only have access to read-only tools. They can read you emails, read your notes, read your texts, maybe even browse the internet with only GET requests... But any action with side-effects ends up in a Tasks list, completely isolated. The agent can't send an email, they don't have such a tool. But they can prepare a reply and put it in the tasks list. Then I proof-read and approve…

There is no real such thing as a read only GET request if we are talking about security issues here. Payloads with secrets can still be exfiltrated, and a server you don’t control can do what it wants when it gets the request.

Re: Don't trust AI agents

#137

Post by TacticalCoder »

ruszki wrote: post #135
gyomu wrote: post #41

Earlier quoted context omitted.

Yeah, it’s pretty wild. Even pg is tweeting stuff like “An experienced programmer told me he's now using AI to generate a thousand lines of code an hour.“ https://x.com/paulg/status/2026739899936944495 Like if you had told pg to his face in (pre AI) office hours “I’m producing a thousand lines of code an hour”, I’m pretty sure he’d have laughed and pointed out how pointless that metric was?

I don't understand how some people decide here, who the good programmers are. A lot of people reminded me a guy from West Palm Beach, who votes on elections solely on the principle of who has more "fame". Paul Graham is famous for sure (at least in HN circles), but I never considered him an exceptional or good programmer at all. So I always interpreted his words with a hefty amount of grain of salt. And sometimes som…

> Paul Graham is famous for sure (at least in HN circles), but I never considered him an exceptional or good programmer at all.

pg wrote a Lisp dialect, Arc, with Morris. The Morris from "the Morris worm". These people are at the very least hackers and they definitely know how to code.

I don't think a "not good programmer" can write a Lisp dialect. At least of all the "not good" programmers I met in my life, 0% of them could have written a Lisp dialect.

It's not because Arc didn't reach the level of fame of Linux or Quake or Kubernetes or whatever that pg is not a good programmer.

Re: Don't trust AI agents

#138

Post by jorvi »

supriyo-biswas wrote: post #69
badsectoracula wrote: post #32

> OpenClaw has nearly half a million lines of code, 53 config files, and over 70 dependencies. This breaks the basic premise of open source security. Chromium has 35+ million lines, but you trust Google’s review processes. Most open source projects work the other way: they stay small enough that many eyes can actually review them. Nobody has reviewed OpenClaw’s 400,000 lines. This reminds me of a very common thing po…

Somehow, this narrative has taken hold at multiple levels of management, especially amongst non-technical management, that "typing" was somehow the bottleneck of software engineering, reality is however more complex. The act of "typing" code was technically mixed in with researching solutions, which means that code often took a different shape or design based on the outcome of that activity. However, this nuance has…

AI is good at the first 80% but terrible at the last 20% of producing good code. And you need to through that first 80% to really understand what the code is scaffolded to do, which writing it yourself will vastly improve. And typing speed has never been the bottleneck for coding.

Even worse, whole generation of devs are being trained to not care of learn about that last 20% because the AI does it """all""" for them. That last bit is an unknown unknown for the neo developer nee prompter.

Re: Don't trust AI agents

#140

Post by raffael_de »

> OpenClaw has nearly half a million lines of code, 53 config files, and over 70 dependencies.

Isn't OpenClaw just ...

  while(true) {
    in = read_input();
    if(in) {
      async relay_2_llm(in);
    }
    sleep(1.0);
  }
... and then some?
Post reply on HN