Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Will It Mythos?

swelljoe.com

Post reply on HN
Print view View original

221–230 of 232 posts

Re: Will It Mythos?

#221

Post by varjag »

clickety_clack wrote: post #174
varjag wrote: post #65

Earlier quoted context omitted.

Interesting. I tried Fable vs Codex 5.5 xhigh on three different cases. 1. A resource leak with unknown cause. Both of them zoomed onto the same potential issue and proposed almost identical patches. Fable missed an edge case that Codex handled correctly. 2. Review of a SPICE model. Models had different comments, none substantial. Both missed important issues that were simulated inadequately. Clearly a valley where t…

I think that Obama-esque, GMAT essay format is the AI flavor that turns me off AI-written articles. It used to be good writing, but because AI locked onto it as such, it's become the watermark of AI generated content.

I meant that in positive sense. Fable's writing was concise and effective without ornamentals. Rather opposite from traditional watery modelese. I rank it better than Codex which is in turn better than Opus.

But my outlook on this is from agent user interface POV where textual communication is essential. When text is the end product there are certainly other considerations.

Re: Will It Mythos?

#222

Post by camdenreslink »

dist-epoch wrote: post #92
airstrike wrote: post #54

Around February, Opus 4.6 was excellent. Smart, fast, proactive. Then it got lobotomized and it's never been the same after that nerf. 4.7 came along and it too was disappointing—not unlike 4.8, which despite feeling a smidge smarter, tends to write word salad and is basically unusable for some workflows. Fable felt like having access to that "old Opus" again, but a little smarter. Sort of like I'd expect an Opus 5 t…

All of these discussions of models being "nerfed" reminds me of discussions among audiophiles "this cable sounds so much better than this other one, it's night and day, ferrari versus honda civic" Yet when you do blind tests they can't tell the difference between a $1000 cable and a $1 one. I bet if you do blind tests between GPT-5.3, 5.4 and 5.5 most would struggle to tell them apart, yet they are certain that "5.5…

I had a repeatable process set up with o4-mini that gave very good results. It generated output in a structured format, and the results were fairly formulaic but good. I did a lot of runs of this process and observed a lot of output.

I set that process down for like 4 or 5 months and then came back to it after GPT 5 came out. o4-mini was literally unusable. I've seen it with my own eyes.

Re: Will It Mythos?

#223

Post by varjag »

Lerc wrote: post #141
varjag wrote: post #65

Earlier quoted context omitted.

Interesting. I tried Fable vs Codex 5.5 xhigh on three different cases. 1. A resource leak with unknown cause. Both of them zoomed onto the same potential issue and proposed almost identical patches. Fable missed an edge case that Codex handled correctly. 2. Review of a SPICE model. Models had different comments, none substantial. Both missed important issues that were simulated inadequately. Clearly a valley where t…

> 2. Review of a SPICE model. Models had different comments, none substantial. Both missed important issues that were simulated inadequately. Clearly a valley where they are undertrained. When models miss things, there is always the possibility that it has the capability to identify the issues but it is misevaluating the level of analysis that you want it to do. The fine tuning will have them targeting a balance of s…

Am used to communicating to EEs, knew what they should've been looking for and I prompted the models just fine. But you would have to take my word for that.

Re: Will It Mythos?

#224

Post by SwellJoe »

utopcell wrote: post #193

The "best" model finds 4/9 bugs. It would be interesting to see if all models find the _same_ bugs. Does a collection of models exist that can cover all 9? Also, it seems to me that pointing a model to a bug and asking it to solve it is somewhat easier than what Mythos did, which if I understand correctly, was to generally look at a codebase and find any bug. Even so, non-Mythos models only managed to fix 4/9 of thes…

You can see which bugs each model found in the full report.

And, you have misunderstood what the benchmark does. It tells the model to audit the file, and it is allowed to look at the rest of the repo. It is not pointed at the bug.

The judge model was pointed at the bug to make sure it could understand and articulate what the bug is as part of the process of selecting the corpus, so it could accurately judge the results.

And, the models were not instructed to fix any bugs. They were instructed to find security bugs. Finding and fixing bugs are quite different problems. This benchmark only tests finding bugs.

In replication tests, where a single model gets several attempts, even a small model (Gemma 4 31b) was able to find 6 of the 9 bugs, in some cases. I still think Mythos is a step up from any current public model. But, I also think it's a bit of hype that it'll upend software security in a way no other model can.

Re: Will It Mythos?

#227

Post by brigandish »

isomorphic_duck wrote: post #94
brigandish wrote: post #20

Earlier quoted context omitted.

They are told outside the test because if they can't find it when given hints then it's safe to assume it won't find it given no hints. It verifies to test, to an extent, much like running tests that should fail when given a set of inputs that should make it fail (you write an always failing test alongside your other tests, right?;)

No, the purpose was to create a (automated) test set in the first place. The author builds an LLM judge which can score the LLMs participating during test-time. That would be why the author used the strongest model (Opus 4,7 at the time) as the judge.

Yes, by giving hints it verifies the test because if it can't find it with hints then it shouldn't find it without hints, else there is some other mechanism at work (which would also be useful to know, like access to other tools).

Re: Will It Mythos?

#228

Post by l1ng0 »

icholy wrote: post #120
l1ng0 wrote: post #118

Earlier quoted context omitted.

You write to the AI as if it were a person. From my point of view it looks like a fair bit of extra typing and extra tokens. Is there a reason you include things like your emotional response and use a very chatty tone? Do you find this seems to alter responses?

I don't want LLM usage to inadvertently change the way I communicate with people.

Lots of interesting answers here, but I'm particularly intrigued people feel this would affect how they talk to humans.

I guess I see AI the same way I see a compiler. I've never worried that I'll end up using code syntax in human conversations.

I tend to write to the AI using language I'd use in technical documentation: concise, detailed, unambiguous as possible.

Re: Will It Mythos?

#229

Post by utopcell »

SwellJoe wrote: post #224
utopcell wrote: post #193

The "best" model finds 4/9 bugs. It would be interesting to see if all models find the _same_ bugs. Does a collection of models exist that can cover all 9? Also, it seems to me that pointing a model to a bug and asking it to solve it is somewhat easier than what Mythos did, which if I understand correctly, was to generally look at a codebase and find any bug. Even so, non-Mythos models only managed to fix 4/9 of thes…

You can see which bugs each model found in the full report. And, you have misunderstood what the benchmark does. It tells the model to audit the file, and it is allowed to look at the rest of the repo. It is not pointed at the bug. The judge model was pointed at the bug to make sure it could understand and articulate what the bug is as part of the process of selecting the corpus, so it could accurately judge the resu…

> You can see which bugs each model found in the full report.

This should have probably been surfaced at the text. To answer my own question: If one uses gemma4-26b-a4b, mimo-v2.5-pro and gemini-3.5-flash then 7/9 bugs are covered, while no model can uncover the remaining two.

> And, you have misunderstood what the benchmark does. It tells the model to audit the file, and it is allowed to look at the rest of the repo. It is not pointed at the bug.

I obviously meant that the model was pointed to the bug report. What would have been the point of providing the actual bug. This is still easier than telling Mythos to generally look at a codebase and find any bug.

Re: Will It Mythos?

#230

Post by saberience »

hodgehog11 wrote: post #198
saberience wrote: post #123

Earlier quoted context omitted.

Funny, I find Codex to still be better at Coding than Opus or Fable. I A/B tested on a whole array of prompts between Codex and Fable, and Fable almost always found that Codex had produced a better plan and covered more edge cases than it did itself. For every problem I gave the exact same prompt to both models, then I had each analyze the other's output. For roughly 80% of the prompts, Fable acknowledged that Codex'…

I've used them back to back as well. Codex is good at specific tasks; it doesn't try to go big, it does what it's told provided the task is relatively procedural. If Codex can make progress on a task, why would I give it to Fable? Fable fumbled the one simple task that I gave it too. I gave it multiple very hard open-ended tasks (effectively math tasks) involving research code and it crushed them. It's the first mode…

I can buy the crazy solutions part.

Codex is highly steerable I find whereas Opus (and Fable in a different way) worked for longer and often did MORE than I wanted it to do.

Fable was absolutely maximalist. We tested it on a fairly innocuous looking prompt and it ended up working for an hour and costing roughly 1000 dollars in tokens.

So I think Fable probably has it's place, but if you know what you want and roughly how to get it, it seems like it would waste too many tokens/money.

Post reply on HN