Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Claude Fable 5: mid-tier results on coding tasks

endorlabs.com

Post reply on HN
Print view View original

71–80 of 271 posts

Re: Claude Fable 5: mid-tier results on coding tasks

#71

Post by standardUser »

renoir wrote: post #9

This matches my experience. Burned $2K to see how it will perform on frontend tasks and backend tasks. Frontend did a significantly better job than Opus on toy-scale wireframe projects by using gimmicks like fluid dynamics. Then when given medium to big tasks like multi-page web app where layouts and aesthetics must be decided by model itself, results by Fable and Opus scored indistinguishable score from human judges…

At a certain point, people value reliability over improved performance. I think a lot of us have hit that point as this technology becomes indispensable to our work. I'm sure I'll use Fable... eventually. But at 2x the cost, I'll skip the inevitable learning curve for now. And thanks for your insights! Not surprising to me that any new model would, as this juncture, be more cryptic and inconsistent than the current models.

Re: Claude Fable 5: mid-tier results on coding tasks

#72

Post by standardUser »

queuebert wrote: post #64
jasondigitized wrote: post #48

Earlier quoted context omitted.

A single 8h task? I'm sorry, but that's just asking for trouble.

I don't understand how some of y'all use these things. I get garbage unless I give them very specific concrete tasks with as much context as possible. Anything that takes more than 30 min is usually a waste because the scope was too large.

You have to build up a context, or otherwise seed the memory, to get anything useful out of these LLMs on a large or existing project.

Re: Claude Fable 5: mid-tier results on coding tasks

#73

Post by ElFitz »

weatherlight wrote: post #60
comboy wrote: post #29

Earlier quoted context omitted.

'by the way, your previous attempts have these structural problems." Just to be clear, it did not have access to any previous work that opus did? Because they are pretty good at digging out relevant tmp files and making use of whatever is out there. With my fable adventures I caught it hallucinating something and stating it as a fact in CLI twice. And it was something that I did not see opus do in such way, opus obvi…

Yes, iit had access. Thats actually the point. I maintain a failure registry in the repo. Every failed attempt gets documented with the exact mechanism, the test that regressed, the revert SHA, and an instruction to start from that frontier. Fable read all of it. But so did Opus. Each of the 16 Opus failures ran in the same harness with the same accumulating registry. By attempt 15, it had disproofs 1–14 in context.…

I’ve had a similar experience.

Pointing out past suboptimal / failing behaviours to new opus sessions would almost always actually create a sort of "anchoring bias" that would drive the agents towards exhibiting the failure mode (often while mentioning how it wouldn’t fall for it).

As far as I can recall, Fable has been the first model to discover the documented failure modes, comment on them, and just… keep going, actually avoiding them. Quite a surprise.

Re: Claude Fable 5: mid-tier results on coding tasks

#75

Post by crimsonnoodle58 »

I found Fable codes very poorly and ended up switching back to Opus.

In one example I switched to Fable in an existing Opus chat, so it had access to the context from Opus which wrote a data importer earlier. I asked it to fix a couple of bugs, and instead of putting the fixes where they should be where the data is imported, it wrote patch functions that did bulk updates at the end of the import.

Fable feels more like a hacker than a coder. Maybe its the way they designed it for security testing thats changed its rationale?

Re: Claude Fable 5: mid-tier results on coding tasks

#76

Post by johnnyApplePRNG »

Yea honestly... the only truths I care about in AI LLM aided devlopment right now is that Claude is a much better planner, and Codex is a much more professional coder.

You can mask a surprisingly amount of terrible coding with proper design planning.

If it works, who cares, right? That's been the status quo for software development for about as long as I can remember, unfortunately.

I used to get frustrated with Codex. I felt as though it wasn't able to see far enough ahead into the future and just intuit what I expected (which is how Claude leaves you feeling).

And then I realized a lot of those intuitions Claude was having were great, and the project progressed, but sometimes to a point that Claude himself was unable to take back control of it... because some of the on the spot decisions it was making were great quick-thinking... but unfortunately, they were only that a lot of the time. Which was the most frustrating of all.

If you specifically ask Claude to plan out and refine a long term project's roadmap though and stick to it, it could probably write an operating system overnight (that kindof worked).

Re: Claude Fable 5: mid-tier results on coding tasks

#77

Post by ElFitz »

cmenge wrote: post #25
weatherlight wrote: post #14

Earlier quoted context omitted.

I had almost the opposite experience. I'm building a compiler for a language without a tracing GC, so a big chunk of the work is around memory management: functional in-place update, reuse analysis, and a Perceus-style reference-counting strategy similar to what Koka uses. The hard part was that my use case wasn't exactly covered by the Koka/Perceus paper. The prior art got me maybe 75% of the way there, but the rema…

Similar. I gave it a really hard task, basically messy code in a complex domain that was bug-ridden from a mess previously created half manually and half by Opus. It cleaned things up beautifully, both the backend and the frontend. Maybe the prompt was particularly well-suited for the model (I instructed it to put on a mathematician's hat, look at the mathematical substructure of the problem, identify invariants and…

> and I have the feeling that the harness is much more important than the consensus expectation.

Is that really the consensus? There’s been a bit of literature lately on that. Can’t find the one about looking into whether or not the harness had a greater impact than the models (for comparable models), but there’s this one: https://arxiv.org/html/2605.23950

Re: Claude Fable 5: mid-tier results on coding tasks

#78

Post by yalok »

jasondigitized wrote: post #48
renoir wrote: post #9

This matches my experience. Burned $2K to see how it will perform on frontend tasks and backend tasks. Frontend did a significantly better job than Opus on toy-scale wireframe projects by using gimmicks like fluid dynamics. Then when given medium to big tasks like multi-page web app where layouts and aesthetics must be decided by model itself, results by Fable and Opus scored indistinguishable score from human judges…

A single 8h task? I'm sorry, but that's just asking for trouble.

if there're some specific tests/evals to satisfy that an agent can test by itself, it can easily iterate for hours. And this time also includes running those tests/evals, which may not be small.

Re: Claude Fable 5: mid-tier results on coding tasks

#80

Post by TacticalCoder »

m101 wrote: post #26

I've been making an auction site and have been using an AI swarm to test it: sellers, intermediaries, buyers, market practices/norms etc. I was mostly using GPT 5.5 xhigh to code up the scenario, and looping over it to check with opus 4.8. Out of curiosity I asked Fable to review it all and I was shocked to find that there were a lot of blindingly obvious common sense mistakes that got through, for example: - all int…

> ... and I was shocked to find that there were a lot of blindingly obvious common sense mistakes that got through

Wait... Are you telling me models everybody told me were better than coders up to just one month ago are actually making lots of mistakes?

This is shocking.

Post reply on HN