Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Shall I implement it? No

gist.github.com

Post reply on HN
Print view View original

21–30 of 603 posts

Re: Shall I implement it? No

#21

Post by Swizec »

verdverm wrote: post #5

Why is this interesting? Is it a shade of gray from HN's new rule yesterday? https://news.ycombinator.com/item?id=47340079 Personally, the other Ai fail on the front of HN and the US Military killing Iranian school girls are more interesting than someone's poorly harnessed agent not following instructions. These have elements we need to start dealing with yesterday as a society. https://news.ycombinator.com/item?id=4…

Because the operator told the computer not to do something so the computer decided to do it. This is a huge security flaw in these newfangled AI-driven systems.

Imagine if this was a "launch nukes" agent instead of a "write code" agent.

Re: Shall I implement it? No

#22

Post by sssilver »

I wonder if there's an AGENTS.md in that project saying "always second-guess my responses", or something of that sort.

The world has become so complex, I find myself struggling with trust more than ever.

Re: Shall I implement it? No

#23

Post by verdverm »

marbletiles wrote: post #19
verdverm wrote: post #9

Earlier quoted context omitted.

That's why we keep humans in the loop. I've seen stuff like this all the time. It's not unusual thinking text, hence the lack of interestingness

The human in the loop here said “no”, though. Not sure where you’d expect another layer of HITL to resolve this.

Tool confirmation

Or in the context of the thread, a human still enters the coords and pulls the trigger

Ukraine is letting some of their drones make kill decisions autonomously, re: areas of EW effect in dead man's zones

Re: Shall I implement it? No

#26

Post by verdverm »

mmanfrin wrote: post #20
verdverm wrote: post #5

Why is this interesting? Is it a shade of gray from HN's new rule yesterday? https://news.ycombinator.com/item?id=47340079 Personally, the other Ai fail on the front of HN and the US Military killing Iranian school girls are more interesting than someone's poorly harnessed agent not following instructions. These have elements we need to start dealing with yesterday as a society. https://news.ycombinator.com/item?id=4…

How is this not clear?

I seen this pattern so often, it's dull. They will do all sorts of stupid things, this is no different.

Re: Shall I implement it? No

#27

Post by serf »

mildred593 wrote: post #15

Never trust a LLM for anything you care about.

never trust a screenshot of a command prompts output blindly either.

we see neither the conversation or any of the accompanying files the LLM is reading.

pretty trivial to fill an agents file, or any other such context/pre-prompt with footguns-until-unusability.

Re: Shall I implement it? No

#29

Post by antdke »

reconnecting wrote: post #24

I’m not an active LLMs user, but I was in a situation where I asked Claude several times not to implement a feature, and that kept doing it anyway.

Yeah, anyone who’s used LLMs for a while would know that this conversation is a lost cause and the only option is to start fresh.

But, a common failure mode for those that are new to using LLMs, or use it very infrequently, is that they will try to salvage this conversation and continue it.

What they don’t understand is that this exchange has permanently rotted the context and will rear its head in ugly ways the longer the conversation goes.

Re: Shall I implement it? No

#30

Post by siva7 »

reconnecting wrote: post #24

I’m not an active LLMs user, but I was in a situation where I asked Claude several times not to implement a feature, and that kept doing it anyway.

people read a bit more about transformer architecture to understand better why telling what not to do is a bad idea
Post reply on HN