Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Will It Mythos?

swelljoe.com

Post reply on HN
Print view View original

141–150 of 232 posts

Re: Will It Mythos?

#141

Post by Lerc »

varjag wrote: post #65
Tossrock wrote: post #39

As I posted in another comment, I found Fable to be substantially more powerful than any previous model. However, this isn't just an ungrounded opinion - I uploaded my full session transcript and code created working on a very complex implementation, so people can judge for themselves, if they're interested: https://tossrock.substack.com/p/36-hours-with-fable

Interesting. I tried Fable vs Codex 5.5 xhigh on three different cases. 1. A resource leak with unknown cause. Both of them zoomed onto the same potential issue and proposed almost identical patches. Fable missed an edge case that Codex handled correctly. 2. Review of a SPICE model. Models had different comments, none substantial. Both missed important issues that were simulated inadequately. Clearly a valley where t…

>2. Review of a SPICE model. Models had different comments, none substantial. Both missed important issues that were simulated inadequately. Clearly a valley where they are undertrained.

When models miss things, there is always the possibility that it has the capability to identify the issues but it is misevaluating the level of analysis that you want it to do. The fine tuning will have them targeting a balance of subjective opinions of what is appropriate. To go beyond broad demographic guessing the model really needs to 'get to know you' to know what it means when you specifically request an action. Without that information about you it has to weigh your words against the level of sophistication it expects a standard user is able to express.

Re: Will It Mythos?

#142

Post by kingofthehill98 »

hosel wrote: post #134
sfjailbird wrote: post #130

[flagged]

Is there any evidence that they nerf models? Anthropic is set to mark a profit Q2 2026 (which is actually not ideal), but there is profit.

Only if you buy their math, which basically is "hey, if we don't do any training we can actually make a profit".

The problem with that math is that if they don't do any training they would be out of the market in 12 months, they're only relevant ("profitable") precisely because they trained the current reference SOTA model.

They can't just release Mythos and sit on top of it forever, competition is catching up fast and people expect a new more powerful model every 6 months.

Re: Will It Mythos?

#143

Post by jrochkind1 »

SwellJoe wrote: post #36
jrochkind1 wrote: post #2

> And, all of the bugs can be identified by several models if they are pointed directly at it and told what to look for. This made me think, well, sure, if you tell them what to look for... but then: > The models can look at the whole repo, and follow logic across file boundaries, but they’re not told what to look for. So okay, the first one was an accidental mis-statement?

You're mixing up corpus selection and the benchmark. I possibly could have explained better. In the benchmark the models were told to look at the file and were allowed to look at the rest of the repo, with no clues about what to look for. During selection of which mythos bugs to include, I needed judge models to be able to determine if contestants found the right bug, since I couldn't realistically judge hundreds of…

I see now, thank you!

Re: Will It Mythos?

#144

Post by [deleted] »

Lerc wrote: post #141
varjag wrote: post #65

Earlier quoted context omitted.

Interesting. I tried Fable vs Codex 5.5 xhigh on three different cases. 1. A resource leak with unknown cause. Both of them zoomed onto the same potential issue and proposed almost identical patches. Fable missed an edge case that Codex handled correctly. 2. Review of a SPICE model. Models had different comments, none substantial. Both missed important issues that were simulated inadequately. Clearly a valley where t…

> 2. Review of a SPICE model. Models had different comments, none substantial. Both missed important issues that were simulated inadequately. Clearly a valley where they are undertrained. When models miss things, there is always the possibility that it has the capability to identify the issues but it is misevaluating the level of analysis that you want it to do. The fine tuning will have them targeting a balance of s…

[deleted]

Re: Will It Mythos?

#145

Post by kgwgk »

Lerc wrote: post #141
varjag wrote: post #65

Earlier quoted context omitted.

Interesting. I tried Fable vs Codex 5.5 xhigh on three different cases. 1. A resource leak with unknown cause. Both of them zoomed onto the same potential issue and proposed almost identical patches. Fable missed an edge case that Codex handled correctly. 2. Review of a SPICE model. Models had different comments, none substantial. Both missed important issues that were simulated inadequately. Clearly a valley where t…

> 2. Review of a SPICE model. Models had different comments, none substantial. Both missed important issues that were simulated inadequately. Clearly a valley where they are undertrained. When models miss things, there is always the possibility that it has the capability to identify the issues but it is misevaluating the level of analysis that you want it to do. The fine tuning will have them targeting a balance of s…

> has the capability to identify the issues but it is misevaluating the level of analysis that you want it to do.

I guess OP should have told it more explicitly to “find all errors without missing anything.”

Re: Will It Mythos?

#146

Post by flatline »

Tossrock wrote: post #39

As I posted in another comment, I found Fable to be substantially more powerful than any previous model. However, this isn't just an ungrounded opinion - I uploaded my full session transcript and code created working on a very complex implementation, so people can judge for themselves, if they're interested: https://tossrock.substack.com/p/36-hours-with-fable

A nit: did you go from Opus 4.5 to Fable? One of the big questions in my mind is how much of a real change Fable is over the existing models. Opus 4.5 -> 4.8 was also a major capability increase.

Re: Will It Mythos?

#147

Post by BlobberSnobber »

kgwgk wrote: post #145
Lerc wrote: post #141

Earlier quoted context omitted.

> 2. Review of a SPICE model. Models had different comments, none substantial. Both missed important issues that were simulated inadequately. Clearly a valley where they are undertrained. When models miss things, there is always the possibility that it has the capability to identify the issues but it is misevaluating the level of analysis that you want it to do. The fine tuning will have them targeting a balance of s…

> has the capability to identify the issues but it is misevaluating the level of analysis that you want it to do. I guess OP should have told it more explicitly to “find all errors without missing anything.”

> Thinking. I know this user well, they don't actually want me to find all errors.

> Thinking.. But I found a smoking gun of an error with this SPICE model, maybe I should inform the user.

> Thinking... Hm, but again, I know this human well, they likely don't care about this error. That's absolutely right - it's not an assistant's job to decide this, it's the user's.

Re: Will It Mythos?

#148

Post by imxyy_soope_ »

hosel wrote: post #134
sfjailbird wrote: post #130

[flagged]

Is there any evidence that they nerf models? Anthropic is set to mark a profit Q2 2026 (which is actually not ideal), but there is profit.

There are LLM performance trackers in the wild, for instance https://marginlab.ai

You may notice that the performance of the old model tends to decline before each new model release.

Re: Will It Mythos?

#149

Post by dhagz »

l1ng0 wrote: post #118
Tossrock wrote: post #39

As I posted in another comment, I found Fable to be substantially more powerful than any previous model. However, this isn't just an ungrounded opinion - I uploaded my full session transcript and code created working on a very complex implementation, so people can judge for themselves, if they're interested: https://tossrock.substack.com/p/36-hours-with-fable

You write to the AI as if it were a person. From my point of view it looks like a fair bit of extra typing and extra tokens. Is there a reason you include things like your emotional response and use a very chatty tone? Do you find this seems to alter responses?

I've found it to lead to an overall better experience, yes. I don't see any reason to not do so - I don't think the token spend is enough to really make an impact, and who cares about typing more? If I get tired of typing I can switch to dictation.

Re: Will It Mythos?

#150

Post by whatisthiseven »

l1ng0 wrote: post #118
Tossrock wrote: post #39

As I posted in another comment, I found Fable to be substantially more powerful than any previous model. However, this isn't just an ungrounded opinion - I uploaded my full session transcript and code created working on a very complex implementation, so people can judge for themselves, if they're interested: https://tossrock.substack.com/p/36-hours-with-fable

You write to the AI as if it were a person. From my point of view it looks like a fair bit of extra typing and extra tokens. Is there a reason you include things like your emotional response and use a very chatty tone? Do you find this seems to alter responses?

LLMs lack context, and I found the more information I provided the better. At some point it was better to just talk to the LLM like I would anyone else. For that matter, LLMs were trained on human speech anyway. It isn't like it was trained on if-else blocks like an Alexa speaker that tries to string together recognized tokens into a pre-configured execution flow.

And finally, LLMs also lack the emotional or human context for why I am doing the specific thing I am doing. Otherwise it will revert to the mode/mean in everything it does. This is obvious, btw: LLMs are generative but they are trained on and largely produce median results if given median inputs. To get results that are "outside the mean/median/average/mode", you need to provide it sufficient context, tokens and input to guide it towards a path that generates higher quality output.

Once you stop approaching LLMs like a machine, and view them more like pseudo-random walks across the compressed set of human written knowledge, it is a little clearer (or at least was to me) how to better write to them.

Post reply on HN