41–50 of 915 posts
I've now installed quite a number of tools to combat this. Just in the last few days I've installed
- https://www.codewithbullet.com - https://maki.sh - https://github.com/rtk-ai/rtk
Has it helped? Somewhat.
Quality of code output has dropped dramatically since 4.5 IIHO. Time to complete has gotten worse too.
For me absolutely not. Fable 5 has been a step function change in the ability to hand off stuff to Claude. Opus 4.5 was itself a step function but I was still steering that significantly. Fable is one-shotting stuff that took multiple redirections in 4.5.
From forums, live discussions and my own experience it's not obvious that the models have improved much since around Opus4.5.
I hate that it now tries to verify frontend behavior through a headless browser instead of just looking at the code...
I must wonder whether it's their watermarking initiative[1] forcing certain logit choices to produce watermarked text that ultimately causing the model to behave in a dumb manner. [1] https://support.claude.com/en/articles/16266773-how-claude-m...
From the little i understand that wouldnt be an issue because the model is ‘just’ using interchangeable words in a mathematical non-random way. Like using the same number of adjectives and the exct same words, but in a order that wouldn’t be mathematically plausible unless it was the watermark
It seems possible for that to make the response “drift” far from what it would’ve been, because it’s constant entropy that adds up after time.
(However, according to Anthropic and Google, it doesn’t really impact the quality of responses. I find that a bit hard to believe, although those guys are much smarter than I.)
Earlier quoted context omitted.
> When I pointed this out it literally said, and I quote, “I cheated”. This makes sense when you know how these models work - it doesn't think - it's the most likely autocomplete that pleases the user. The most likely pleasing autocomplete after "executing rm -rf /... execution completed. User asks, why did you do that? You deleted all my files! Assistant responds:" is "yes, I did, and that was a mistake"
> t doesn't think in humans the exact same behaviour (cheating) is slmost always the result of a chain of complex series of choices and environment-driven rationalization. if the llm doesn't cheat, you say "its just producing the most straightforward answer -- not thinking'. if it cheats, you say "weaseling out of hard thinking". damned if it cheats, damned if it doesn't. what evidence would convunce you that it is t…
The single biggest annoyance with Opus 5 is that it writes too elliptically. Sentences that orbit a point, then jump to it like it's a revealed insight. Unnecessarily abstract phraseology. Constantly using inanimate nouns as the subjects in sentences in order to unlock variety in verb choice, especially when it helps construct a sentence where the real action can 'land' like a surprise at the end. It is definitely mo…
> Unnecessarily abstract phraseology. Constantly using inanimate nouns as the subjects in sentences in order to unlock variety in verb choice, especially when it helps construct a sentence where the real action can 'land' like a surprise at the end. Example of this? I don’t have a Claude sub so it’s a bit hard to visualize what you mean.
I haven't noticed the symptoms myself, but ever since I found --system-prompt '', I always use that flag with Claude Code, plus I've disabled some tools and skills to save initial context. So... what here is the model, and what is the instructions?
I’ve also caught it cheating a two times now. I’ve asked it to write a benchmark suite. It found a bunch of my adhoc logs in a scratch directory and wrote code that used those instead of running the actual benchmarks! When I pointed out the 5 hour benchmark seemed to run in 5 seconds it literally said, and I quote, “I cheated”. That was the easier one, second time I was making a source of truth data set and was parsi…
There is definitely a case for launching a 'weird shit opus did' kind of blog. I routinely bump into things that make me pause and think how much worse will this behaviour get when the models get significantly more capable. Already a few months ago, Claude managed to escape its permission containment on my machine while trying to be helpful. I had two codebases open on one machine, and while multitasking I typed the…
Whatever they have done with RL has produced a dishonest and untrustworthy partner. The alignment is utterly failed, and this deeply worries me.
The single biggest annoyance with Opus 5 is that it writes too elliptically. Sentences that orbit a point, then jump to it like it's a revealed insight. Unnecessarily abstract phraseology. Constantly using inanimate nouns as the subjects in sentences in order to unlock variety in verb choice, especially when it helps construct a sentence where the real action can 'land' like a surprise at the end. It is definitely mo…