[flagged]
How do they “nerf the models”?
Are they quietly compacting context to reduce kv cache usage, before the actual compaction? Like there’s a slider for how much to compress it, and that’s never revealed to us?
Will It Mythos?
151–160 of 232 posts
Re: Will It Mythos?
#152I was pretty impressed with Fable when I used it. Fable on Low was better than Opus 4.8 on High (and cheaper).
Now, for me, it was really about how well it worked on big existing human made code bases. I was working on some new screens in GalCiv IV and if you've ever had to make screens for games, it is incredibly tedious, low brain work. But GPT 5.5 and Opus 4.8 would just struggle with these over and over again and this is C++ work with limited hotloading so it's a slow process. Fable nailed these screens fast.
Re: Will It Mythos?
#153Earlier quoted context omitted.
All of these discussions of models being "nerfed" reminds me of discussions among audiophiles "this cable sounds so much better than this other one, it's night and day, ferrari versus honda civic" Yet when you do blind tests they can't tell the difference between a $1000 cable and a $1 one. I bet if you do blind tests between GPT-5.3, 5.4 and 5.5 most would struggle to tell them apart, yet they are certain that "5.5…
You will be amused to hear that when Anthropic "refreshed" 4.6 on AWS Bedrock I found it in my tests and wrote about it – and they actually rolled it back. This is how much non–coding tests may tell you about the model.
So Bedrock 4.6 is old school Opus?
I know you can point Claude code at Bedrock.. might be worth a play.
Re: Will It Mythos?
#154This just shows that Google needs to double down on its AI models fast. Even open source chinese models are beating 3.1 Pro and 3.5.Flash in almost everything.
Gemma 4 beat Gemini 3.1 Pro, as well. In a later replication test I haven't published yet, it found more bugs than all other models (somewhat inconsistently) when given multiple attempts. So, it seems like they are doing real work but seemingly on making models efficient rather than making them bigger. Gemma 4 12b is the most effective vision model I've tested, including models several times its size.
Re: Will It Mythos?
#155As I posted in another comment, I found Fable to be substantially more powerful than any previous model. However, this isn't just an ungrounded opinion - I uploaded my full session transcript and code created working on a very complex implementation, so people can judge for themselves, if they're interested: https://tossrock.substack.com/p/36-hours-with-fable
You write to the AI as if it were a person. From my point of view it looks like a fair bit of extra typing and extra tokens. Is there a reason you include things like your emotional response and use a very chatty tone? Do you find this seems to alter responses?
I'll go a step further and to say this it's genuinely unsettling someone type to a computer like this. I won't claim to be a psychologist, but with how many instances of "AI psychosis" have been reported (and I've seen first-hand) it seems like treating the computer like a computer is safer, not to mention more effective e.g. lower token usage.
Re: Will It Mythos?
#156Earlier quoted context omitted.
No, it’s just a fundamentally much better model. Going back to Opus feels like the model has been lobotomized. It makes much more frequent errors, especially of the “I claimed I tested x y and z, but actually only kinda half heartedly tested x, and assumed I understood what was wrong” variety.
Wait but that has been the exact word-for-word complaint when comparing sonnet to opus Or opus to opus Or really any new thing to old thing
You hear the same canard every time Anthropic releases a new model or version. I'm not convinced they're objective anecdotes. I wonder if it's simply the new model, while marginally better, has a different style and people find that new/refreshing. That is what makes it feel so much better than the previous release.
Re: Will It Mythos?
#157The leaderboard sorting is very misleading, gpt-5.5-pro only found 2 while mimo-v2.5-pro found 4.5 out of 9 cases.
Yeah, I'm not super happy with the chart sorting order, but trying to balance all the information is challenging. I chose not to include partials (right place, inaccurate bug description, so it smelled something funny but didn't quite understand it) in the sort order, but maybe should.
And, it does feel wrong that the unrealistically expensive model that no one in their right mind would use for anything but the most critical tasks (and even then, a committee of ten of the best alternatives would cost half as much) is at the top. But, GPT 5.5 Pro did find a bug nobody else found among the four cases it got to, hinting at some real difference. It may be closer to Mythos than others, but at an absurd price. It'd cost tens of thousands of dollars to audit all the files in a large codebase, versus maybe fifty bucks for MiMo or DeepSeek.
Re: Will It Mythos?
#158This is cool, but note that it doesn't address one of the main (claimed) advantages of Mythos: lower false positive rates. That is, give it files without serious bugs and it will not raise alarms.
Re: Will It Mythos?
#159Earlier quoted context omitted.
Also, with regard to tools, I originally ran a batch of several models in a full-featured agent (and whatever tools the agent provides), and they didn't perform better than the basic minimal harness with just read and grep. They chewed more tokens but didn't find more bugs. I'm currently doing tests with more advanced tools, like tree-sitter so the model can better understand execution and data flow and semgrep (whic…
We can't really know in what manner Mythos was used to find these bugs, right?
Right. I noted that in the post. Some of the information out of Anthropic indicates dumb loops, sometimes, but some hint at a more sophisticated harness and process for some of the Mythos bug hunts. But, nothing specific.
I've been doing more benchmarks with additional tools, with no silver bullet revealing itself thus far.
Re: Will It Mythos?
#160Earlier quoted context omitted.
Interesting. I tried Fable vs Codex 5.5 xhigh on three different cases. 1. A resource leak with unknown cause. Both of them zoomed onto the same potential issue and proposed almost identical patches. Fable missed an edge case that Codex handled correctly. 2. Review of a SPICE model. Models had different comments, none substantial. Both missed important issues that were simulated inadequately. Clearly a valley where t…
To me it feels like they're basically tweaking these things around the edges. I'm not seeing any difference in capability just preference. This has been the case for a while.
That makes sense, its seemed to me for a while now the competing product is the harness not the model itself.