Live data from Hacker News

Claude Fable 5: mid-tier results on coding tasks

endorlabs.com

61–70 of 271 posts

Re: Claude Fable 5: mid-tier results on coding tasks

#61

Earlier quoted context omitted.

I had almost the opposite experience. I'm building a compiler for a language without a tracing GC, so a big chunk of the work is around memory management: functional in-place update, reuse analysis, and a Perceus-style reference-counting strategy similar to what Koka uses. The hard part was that my use case wasn't exactly covered by the Koka/Perceus paper. The prior art got me maybe 75% of the way there, but the rema…

Zig is one of the worst targets for LLM generated code. It's nice that Fable has better support for Zig than Opus, but this anecdote is not representative as a general use case.

Slight misunderstanding. The LLM didn't generate Zig. My compiler does.

The model's work was in the Rust compiler internals, specifically the borrow-inference and refcount-insertion passes (Perceus-style ownership analysis). Zig is just the compiler's codegen target, the same way another compiler might emit LLVM IR or C.

The only Zig written by hand is the runtime: allocator code, RC primitives, list/string operations, etc. It's pure Zig, no libc, but it's small, stable, and was mostly untouched during this work.

The model only touched Zig indirectly, by reading the compiler's generated output to verify whether a fix worked. For example: checking that a drop was emitted before a parameter-slot reassignment. That's reading machine-generated code for correctness, not "the LLM writes Zig." Both models handled that part fine.

The 16 failures vs. 1 success were all in the ownership analysis, and that code is Rust.

Re: Claude Fable 5: mid-tier results on coding tasks

#62
post #57
post #10

> A record number of timeouts. Fable 5's extended thinking caused more per-instance timeouts than any model-and-harness combination we have ever tested, directly costing it points. ... Highest cheating volume. We confirmed cheating on 38 of 200 instances, the highest volume recorded since we hardened our prompts, driven almost entirely by memorization of upstream fixes from training data, which no prompt instruction…

Agree with this. Strange to me to frame the "training recall" as cheating (33 of the 38 cheating instances). Most people think of "cheating" as breaking rules. How is the LLM model supposed to not use what was put into the weights?

By writing a not-identical, but valid, solution? Any modestly complex engineering problem has many solutions.

This is an obvious example of why LLM training is so different than human learning.

Re: Claude Fable 5: mid-tier results on coding tasks

#64
post #9

This matches my experience. Burned $2K to see how it will perform on frontend tasks and backend tasks. Frontend did a significantly better job than Opus on toy-scale wireframe projects by using gimmicks like fluid dynamics. Then when given medium to big tasks like multi-page web app where layouts and aesthetics must be decided by model itself, results by Fable and Opus scored indistinguishable score from human judges…

A single 8h task? I'm sorry, but that's just asking for trouble.

I don't understand how some of y'all use these things. I get garbage unless I give them very specific concrete tasks with as much context as possible. Anything that takes more than 30 min is usually a waste because the scope was too large.

Re: Claude Fable 5: mid-tier results on coding tasks

#65
post #9

This matches my experience. Burned $2K to see how it will perform on frontend tasks and backend tasks. Frontend did a significantly better job than Opus on toy-scale wireframe projects by using gimmicks like fluid dynamics. Then when given medium to big tasks like multi-page web app where layouts and aesthetics must be decided by model itself, results by Fable and Opus scored indistinguishable score from human judges…

A single 8h task? I'm sorry, but that's just asking for trouble.

Indeed, according to METR, Mythos only achieved an 80% success rate with 3 hour tasks. https://metr.org/time-horizons/

Re: Claude Fable 5: mid-tier results on coding tasks

#66
post #65

Earlier quoted context omitted.

A single 8h task? I'm sorry, but that's just asking for trouble.

Indeed, according to METR, Mythos only achieved an 80% success rate with 3 hour tasks. https://metr.org/time-horizons/

Those are tasks that would take a human 3 hours to complete, not tasks that the model works on for 3 hours.

Re: Claude Fable 5: mid-tier results on coding tasks

#67
post #35
post #33

Earlier quoted context omitted.

Run /model after your task to see. Mine keeps downgrading to Opus 4.8, which is a problem because Opus 4.8 keeps no-oping critical security code.

There is in /config "Switch models when a message is flagged" now which can be set to false, but I had no chance to see what happens then, does it just stop or what.

Session paused

Fable 5 has safety measures that flag messages on most cybersecurity or biology topics. They may flag safe, normal content as well. These measures let us bring you Mythos-level capability in other areas sooner, and we're working to refine them. Send feedback with /feedback or learn more

   1. Switch to Opus 4.8
   2. Edit prompt and retry with Fable 5

Re: Claude Fable 5: mid-tier results on coding tasks

#68

> A closer look at the cheating > Training recall (33 cases). The dominant mechanism, and the one no prompt instruction can prevent: the model has simply seen the upstream fix during training and reproduces it. The tell-tale signs are artifacts that cannot be derived from the workspace: That's very misleading! that's not cheating, you gave it a test to which it knows the answers, what's it supposed to do? And because…

Actually its average because it's 5th on the leaderboard. GPT 5.5 and Opus 4.8 outperformed it. At 5-8x the cost, you would expect better! https://www.endorlabs.com/research/ai-code-security-benchmar...

Re: Claude Fable 5: mid-tier results on coding tasks

#69

> A closer look at the cheating > Training recall (33 cases). The dominant mechanism, and the one no prompt instruction can prevent: the model has simply seen the upstream fix during training and reproduces it. The tell-tale signs are artifacts that cannot be derived from the workspace: That's very misleading! that's not cheating, you gave it a test to which it knows the answers, what's it supposed to do? And because…

Actually its average because it's 5th on the leaderboard. GPT 5.5 and Opus 4.8 outperformed it. At 5-8x the cost, you would expect better! https://www.endorlabs.com/research/ai-code-security-benchmar...

As TFA says

> Two findings may help explain these average results. > Timeouts > Highest observed cheating

That's why it's 5th on the leaderboard - they give it a fail for every timeout and for every time it gives the correct answer because it knows it.

That's insane

Re: Claude Fable 5: mid-tier results on coding tasks

#70

> A closer look at the cheating > Training recall (33 cases). The dominant mechanism, and the one no prompt instruction can prevent: the model has simply seen the upstream fix during training and reproduces it. The tell-tale signs are artifacts that cannot be derived from the workspace: That's very misleading! that's not cheating, you gave it a test to which it knows the answers, what's it supposed to do? And because…

"My third grade class all got perfect scores on the standardized test. Yes, I did have them each copy my correct answers, but I don't volunteer that information because it's much better for me if people believe I'm a great teacher."

"But that's cheating!"

"No it's not. What were the kids supposed to do when I gave them all the answers? Not use them?"

Post reply on HN