31–40 of 603 posts
Claude is quite bad at following instructions compared to other SOTA models. As in, you tell it "only answer with a number", then it proceeds to tell you "13, I chose that number because..."
I wonder if there's an AGENTS.md in that project saying "always second-guess my responses", or something of that sort. The world has become so complex, I find myself struggling with trust more than ever.
Why is this interesting? Is it a shade of gray from HN's new rule yesterday? https://news.ycombinator.com/item?id=47340079 Personally, the other Ai fail on the front of HN and the US Military killing Iranian school girls are more interesting than someone's poorly harnessed agent not following instructions. These have elements we need to start dealing with yesterday as a society. https://news.ycombinator.com/item?id=4…
Because the operator told the computer not to do something so the computer decided to do it. This is a huge security flaw in these newfangled AI-driven systems. Imagine if this was a "launch nukes" agent instead of a "write code" agent.
They aren't smart, they aren't rationale, they cannot reliably follow instructions, which is why we add more turtles to the stack. Sharing and reading agent thinking text is boring.
I had one go off on e one time, worse than the clawd bot who wrote that nasty blog after being rejected on GitHub. Did I share that session? No, because it's boring. I have 100s of these failed sessions, they are only interesting in aggregate for evals, which is why is save them.
Seems like they skipped training of the me too movement
I've always wondered what these flagship AI companies are doing behind the scenes to setup guardrails. Golden Gate Claude[1] was a really interesting... I haven't seen much additional research on the subject, at the least open-facing.
I’m not an active LLMs user, but I was in a situation where I asked Claude several times not to implement a feature, and that kept doing it anyway.
Why is this interesting? Is it a shade of gray from HN's new rule yesterday? https://news.ycombinator.com/item?id=47340079 Personally, the other Ai fail on the front of HN and the US Military killing Iranian school girls are more interesting than someone's poorly harnessed agent not following instructions. These have elements we need to start dealing with yesterday as a society. https://news.ycombinator.com/item?id=4…