Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Claude Fable 5: mid-tier results on coding tasks

endorlabs.com

Post reply on HN
Print view View original

101–110 of 271 posts

Re: Claude Fable 5: mid-tier results on coding tasks

#101

Post by matheusmoreira »

andai wrote: post #82

> Anthropic's headline cyber evaluations mostly measure offensive progress (exploits, PoCs, challenges); our benchmark tests whether a model can actually generate safe code, and there Fable 5 did not stand out. The model isn't allowed to think about security. I heard several people here mention that if it starts thinking about security -- e.g. writing tests related to it -- the safety filter flags it and downgrades t…

Yeah. Fable apparently found bugs in my C code but Anthropic wouldn't allow it to test them, fix them or even tell me what the problem was. The memory safety parts of my Fable code review were 50% Opus. Even the coordinator Fable that just launched the code review agents got downgraded to Opus for some reason.

Model is definitely better than Opus but Anthropic's delivering a pretty terrible experience.

Re: Claude Fable 5: mid-tier results on coding tasks

#102

Post by m-dot-reviews »

afro88 wrote: post #8

Similar result on our kotlin coding benchmark at work. It measures how close agents can get to a small mergable PR (according to my team). 20 tasks of varying difficulty, with 5 attempts each, LLM as judge to evaluate accuracy (same outcome and quality but allowing for acceptable variances). Fable 5 sits ahead of Opus 4.7, but behind Opus 4.6, Sonnet 4.6, Opus 4.8, GPT-5.4, GPT-5.5. Fable isn't a good coding workhors…

I'm starting a repository of LLM reviews [1] with the goal of creating a catalog that is more task-oriented and less marketing-y than corporate blogs or benchmark leaderboards. You seem to have a lot of experience across a bunch of different models: if you have a chance and feel like sharing, you'd be one of the first.

[1] - https://model.reviews/ - all the user-submitted content is CC licensed and will be available for download in periodic dumps.

Re: Claude Fable 5: mid-tier results on coding tasks

#103

Post by brookst »

I’m finding Fable dramatically better for auditing PR’s and large features. In a side by side with the same prompt I’ve been happily using on Opus, Opus found one major and one minor issue, fable found two major and four minor (a superset of Opus).

I’ve taken to using fable to plan arch, specs, build plan, and then to be the final QA. Opus for the actual build.

Re: Claude Fable 5: mid-tier results on coding tasks

#106

Post by notnullorvoid »

sigmar wrote: post #57
gwern wrote: post #10

> A record number of timeouts. Fable 5's extended thinking caused more per-instance timeouts than any model-and-harness combination we have ever tested, directly costing it points. ... Highest cheating volume. We confirmed cheating on 38 of 200 instances, the highest volume recorded since we hardened our prompts, driven almost entirely by memorization of upstream fixes from training data, which no prompt instruction…

Agree with this. Strange to me to frame the "training recall" as cheating (33 of the 38 cheating instances). Most people think of "cheating" as breaking rules. How is the LLM model supposed to not use what was put into the weights?

While I probably wouldn't classify it as cheating, it is an even bigger signal of concern for model quality.

Cheating by breaking the rules at least implies some learned patterns.

Repeating training data verbatim for narrow cases like this implies that the model is overfitting.

Re: Claude Fable 5: mid-tier results on coding tasks

#107

Post by smoe »

queuebert wrote: post #64
jasondigitized wrote: post #48

Earlier quoted context omitted.

A single 8h task? I'm sorry, but that's just asking for trouble.

I don't understand how some of y'all use these things. I get garbage unless I give them very specific concrete tasks with as much context as possible. Anything that takes more than 30 min is usually a waste because the scope was too large.

I had good experiences doing multi-hour refactoring/housekeeping tasks that basically consisted of applying the same steps and rules n times.

Worth noting, a significant chunk of those runs involved the agent waiting for the compiler, linters, type checks, and test suites, as well as updating journals. It’s not the agent sputtering out code for eight hours straight.

And naturally I spend more time on manual verification in the end as much less of it is happening during the coding process.

Re: Claude Fable 5: mid-tier results on coding tasks

#108

Post by notnullorvoid »

bensyverson wrote: post #4

> The dominant mechanism, and the one no prompt instruction can prevent: the model has simply seen the upstream fix during training and reproduces it… > On numpy, the patch is 100% character-for-character identical to the golden patch… down to idiosyncratic comments like "Extending singleton dimension for 'reflect' is legacy behavior; it really should raise an error." This… seems like a flaw in the benchmark suite me…

Maybe a flaw in the labeling, but not the core methodology.

Verbatim code snippets like this imply the model is overfitting to it's training data.

Re: Claude Fable 5: mid-tier results on coding tasks

#109

Post by i2km »

My theory is that anthropic have hit the beginnings of model collapse and the whole "fable may silently downgrade with deliberately incorrect results" is a diabolical attempt to gas light and get ahead of the curve.

So when it fails, people will chalk it up to "oh. Must have been silently downgraded because it thought I was doing something tricky enough to count as a distillation attack. My bad. Lemme try again..."

Re: Claude Fable 5: mid-tier results on coding tasks

#110

Post by notnullorvoid »

jasondigitized wrote: post #48
renoir wrote: post #9

This matches my experience. Burned $2K to see how it will perform on frontend tasks and backend tasks. Frontend did a significantly better job than Opus on toy-scale wireframe projects by using gimmicks like fluid dynamics. Then when given medium to big tasks like multi-page web app where layouts and aesthetics must be decided by model itself, results by Fable and Opus scored indistinguishable score from human judges…

A single 8h task? I'm sorry, but that's just asking for trouble.

This sounds like classic "you're using it wrong", if they had said it was done in smaller tasks you would very likely have people here saying that was wrong too.
Post reply on HN