Frankly after testing out Fable last week, it was just a bigger sink of tokens than anything else. The amount of tokens consumed by it wasn't worth the steps it saved me compared to using opus 4.8.
Will It Mythos?
71–80 of 232 posts
Re: Will It Mythos?
#72Earlier quoted context omitted.
Hard disagree. Opus reports to me like a student. Fable reported to me like a colleague (researcher). It genuinely seemed to pick up on nuance that the other models just don't, even when I tell them explicitly. It's been really frustrating that neither Codex nor Opus can make targetted edits to Fable's code without screwing something subtle up. For context, this is for computational geometry work, so your mileage may…
> It's been really frustrating that neither Codex nor Opus can make targetted edits to Fable's code without screwing something subtle up. Reminds me of the old adage: don't try to be too smart when writing code. Otherwise, dumber people - including your future self - will have trouble working with it.
For reference:
if you write the code as cleverly as possible, you are, by definition, not smart enough to debug it
Re: Will It Mythos?
#73I don't understand the article. "I’d say this benchmark answers with a resounding, “Maybe.” Mythos maybe really is better than the other current models at finding security bugs" Yet in the results, I don't see Mythos? It seems like a really well researched article with lots of results for other models, yet the title seems to be clickbait because the results don't contain Mythos, do they?
Bugs the other models were benchmarked on are from the corpus that Mythos found. So Mythos might have 100% in this benchmark. Although the benchmark had 100$ budget cap and rudimentary tooling so probably a bit less than 100%. GPT-5.5-pro attemted only 4 problems out of 9 before the budget ran out and got 2 of them right. It's a shame that the author didn't try GPT-5.5-pro on all 9 just for completeness, pehaps on su…
At the time a GPT subscription didn't include Pro usage in the rolling limits. It was billed at API rates. Does it now?
If anyone wants to fund the other five cases (~$125), I'll run them. I find that an unrealistic cost, though...simply not useful data. I'm certainly not going to spend $23 per file to audit a project with hundreds or thousands of files. I don't know anyone who would.
Also note that it was $100 cap per model, and the next most expensive model was GPT 5.5 at a 20th the price per case, about ten bucks for the whole batch.
Re: Will It Mythos?
#74From all the things I read I'm pretty convinced that Mythos is just standard LLM with safety features turned off. If current models weren't reluctant to search for vulnerabilities, they might perform as good as Mythos.
Early on, I had a vague suspicion that the reason some of the Chinese models, including quite small ones, perform so well on this task, especially relative to their size and cost, is because they don't have the same safety guardrails baked in regarding software security that US models seem to have. Gemini 3.1 Pro doing so poorly sort of reinforced that gut feeling. But, then Gemma 4 proved to be extraordinarily good…
Can you elaborate on the "software security that US models" seem to have? According to blog posts I read, the code generated had security problems and naive ones at that. Perhaps it got better now or people have learned not to blindly vibe code applications that are to be used publicly but it certainly didn't feel like there were security guardrails.
Re: Will It Mythos?
#75I don't understand the article. "I’d say this benchmark answers with a resounding, “Maybe.” Mythos maybe really is better than the other current models at finding security bugs" Yet in the results, I don't see Mythos? It seems like a really well researched article with lots of results for other models, yet the title seems to be clickbait because the results don't contain Mythos, do they?
Bugs the other models were benchmarked on are from the corpus that Mythos found. So Mythos might have 100% in this benchmark. Although the benchmark had 100$ budget cap and rudimentary tooling so probably a bit less than 100%. GPT-5.5-pro attemted only 4 problems out of 9 before the budget ran out and got 2 of them right. It's a shame that the author didn't try GPT-5.5-pro on all 9 just for completeness, pehaps on su…
Also, with regard to tools, I originally ran a batch of several models in a full-featured agent (and whatever tools the agent provides), and they didn't perform better than the basic minimal harness with just read and grep. They chewed more tokens but didn't find more bugs. I'm currently doing tests with more advanced tools, like tree-sitter so the model can better understand execution and data flow and semgrep (which is almost cheating, since it finds bugs on its own, but worth a try since models can still be useful in helping rule out false positives and suggest mitigations). When I've got time for it, I'll also give them a full dev environment with compiler, debugger, and maybe fuzzer, and a loop that iterates through a security bug hunting checklist (since a single prompt and context window can't handle that much complexity at once).
Re: Will It Mythos?
#76From all the things I read I'm pretty convinced that Mythos is just standard LLM with safety features turned off. If current models weren't reluctant to search for vulnerabilities, they might perform as good as Mythos.
Early on, I had a vague suspicion that the reason some of the Chinese models, including quite small ones, perform so well on this task, especially relative to their size and cost, is because they don't have the same safety guardrails baked in regarding software security that US models seem to have. Gemini 3.1 Pro doing so poorly sort of reinforced that gut feeling. But, then Gemma 4 proved to be extraordinarily good…
>But, then Gemma 4 proved to be extraordinarily good for its size (better than Qwen), and kinda disproved that US models are any weaker at small sizes.
Did it "disprove" it retroactively or just changed what the situation is, given that until then they were indeed weaker at small sizes?
Re: Will It Mythos?
#77As I posted in another comment, I found Fable to be substantially more powerful than any previous model. However, this isn't just an ungrounded opinion - I uploaded my full session transcript and code created working on a very complex implementation, so people can judge for themselves, if they're interested: https://tossrock.substack.com/p/36-hours-with-fable
> code created working on a very complex implementation
I always find it amusing when people claim "a very complex implementation". Sometimes it's a hard problem, other times an easy one. Either way that's not for you to judge.
And the implementation being complex... is that a good thing? Wouldn't a simple implementation be better? It reminded me of the parable of two programmers.
Re: Will It Mythos?
#78Earlier quoted context omitted.
Early on, I had a vague suspicion that the reason some of the Chinese models, including quite small ones, perform so well on this task, especially relative to their size and cost, is because they don't have the same safety guardrails baked in regarding software security that US models seem to have. Gemini 3.1 Pro doing so poorly sort of reinforced that gut feeling. But, then Gemma 4 proved to be extraordinarily good…
Can you elaborate on the "software security that US models" seem to have? According to blog posts I read, the code generated had security problems and naive ones at that. Perhaps it got better now or people have learned not to blindly vibe code applications that are to be used publicly but it certainly didn't feel like there were security guardrails.
I'm talking about guardrails that prevent finding exploits, which is only peripherally related to writing secure code.
This benchmark is about finding security bugs, not writing secure code. I don't believe the models have guardrails that prevent writing safe code, but they're also not intelligent and have a bunch of insecure code in their training data, so they definitely write insecure code sometimes.
Re: Will It Mythos?
#79As I posted in another comment, I found Fable to be substantially more powerful than any previous model. However, this isn't just an ungrounded opinion - I uploaded my full session transcript and code created working on a very complex implementation, so people can judge for themselves, if they're interested: https://tossrock.substack.com/p/36-hours-with-fable
You guys are getting Fable?
Re: Will It Mythos?
#80For malware detection, many models are biased for or against detecting a threat (likely a thing that can be adjusted with a prompt).
I suggest tasks cannot be guessed (find, not tell). And 2d charts, both for ROC and pricing, vide https://quesma.com/benchmarks/binaryaudit/