Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Claude Fable 5: mid-tier results on coding tasks

endorlabs.com

Post reply on HN
Print view View original

31–40 of 271 posts

Re: Claude Fable 5: mid-tier results on coding tasks

#31

Post by numeri »

bensyverson wrote: post #13
eli wrote: post #7

Earlier quoted context omitted.

The other "cheating" examples are even worse. It's wild to me that people keep designing benchmarks where the answer is lying around on disk or in the git history. "Hardening" the benchmark with strongly worded prompt instructions is bizarre. There are so many agent sandbox solutions. Why not use one and give it only access to the code it should see? And I'm not sure how they can rule out other solutions also benefit…

100%… the fact that they're just using prompting to discourage the agent from looking ahead in the Git history is wild.

To be fair, it is good to know that it disobeys simple instructions like "don't examine my git history" far more than other models. (It should of course be a different benchmark, so as not to conflate things.)

It's not a great sign for alignment.

Re: Claude Fable 5: mid-tier results on coding tasks

#32

Post by pleasstopnw »

m101 wrote: post #26

I've been making an auction site and have been using an AI swarm to test it: sellers, intermediaries, buyers, market practices/norms etc. I was mostly using GPT 5.5 xhigh to code up the scenario, and looping over it to check with opus 4.8. Out of curiosity I asked Fable to review it all and I was shocked to find that there were a lot of blindingly obvious common sense mistakes that got through, for example: - all int…

[dead]

Re: Claude Fable 5: mid-tier results on coding tasks

#33

Post by espeed »

renoir wrote: post #9

This matches my experience. Burned $2K to see how it will perform on frontend tasks and backend tasks. Frontend did a significantly better job than Opus on toy-scale wireframe projects by using gimmicks like fluid dynamics. Then when given medium to big tasks like multi-page web app where layouts and aesthetics must be decided by model itself, results by Fable and Opus scored indistinguishable score from human judges…

Run /model after your task to see. Mine keeps downgrading to Opus 4.8, which is a problem because Opus 4.8 keeps no-oping critical security code.

Re: Claude Fable 5: mid-tier results on coding tasks

#34

Post by Scene_Cast2 »

I'm personally heavily testing LLMs on electrical engineering problems. I'm finding that it's not meaningfully better at figuring out what's up than the other models.

To give you an idea - here's a very abridged summary of one sample question (originally a full paragraph): I have a voltage divider with a precision resistor and a thermistor, my voltage reading is off by 17%, where's that coming from. None of the models I tested (including Opus 4.8 and Fable 5) could figure it out.

Re: Claude Fable 5: mid-tier results on coding tasks

#35

Post by comboy »

espeed wrote: post #33
renoir wrote: post #9

This matches my experience. Burned $2K to see how it will perform on frontend tasks and backend tasks. Frontend did a significantly better job than Opus on toy-scale wireframe projects by using gimmicks like fluid dynamics. Then when given medium to big tasks like multi-page web app where layouts and aesthetics must be decided by model itself, results by Fable and Opus scored indistinguishable score from human judges…

Run /model after your task to see. Mine keeps downgrading to Opus 4.8, which is a problem because Opus 4.8 keeps no-oping critical security code.

There is in /config "Switch models when a message is flagged" now which can be set to false, but I had no chance to see what happens then, does it just stop or what.

Re: Claude Fable 5: mid-tier results on coding tasks

#36

Post by JofArnold »

I've found it outstanding at isolated long running tasks (eg completed one of our tests in 3 hours and a 100% accuracy score versus 5.5 xhigh's 10 hours and 90% accuracy). For short tasks it seems very Claude'y (hard to express exactly what I mean by that) which I'm not a fan of meaning I'll stick with Codex for that use case and maybe Fable for those times I can for sure benefit from it.

Re: Claude Fable 5: mid-tier results on coding tasks

#38

Post by pllbnk »

My experience is that with every new release it's getting slower but not necessarily better. I have some projects where I review everything that the agents code - these projects look generally fine because I keep them in line. There are also a few projects that I just vibe code and focus on the result (sometimes I want to pull my hair out because of constant stream of stupid bugs) and don't look at the code.

Well, today I gave Fable a try on one of the vibe-coded projects. It simply had to write a couple Python scripts 400-500 lines each. It did and they worked after a few iterations but I decided to look at the code it produced. There were weird constants that might (and will) break the code when the requirements will change. The code itself is unreadable and a total mess. If it would write a well-structured code in the first place, I believe it would be more efficient in working with that code too.

I have serious considerations how far will I be able to go with just the pure vibe coding. My projects are small one-person projects and so far I am able to push through but I hardly see how far will I be able to go before technical debt outgrows the value the code produces.

I fondly remember the times of Opus 4.5 where it was still (to my memory) reasonably fast and malleable.

Re: Claude Fable 5: mid-tier results on coding tasks

#39

Post by threatripper »

Scene_Cast2 wrote: post #34

I'm personally heavily testing LLMs on electrical engineering problems. I'm finding that it's not meaningfully better at figuring out what's up than the other models. To give you an idea - here's a very abridged summary of one sample question (originally a full paragraph): I have a voltage divider with a precision resistor and a thermistor, my voltage reading is off by 17%, where's that coming from. None of the model…

Did you also test GPT-5.5 Pro web version?

Why is the voltage reading 17% off?

Re: Claude Fable 5: mid-tier results on coding tasks

#40

Post by Madmallard »

practal wrote: post #30

I am quite impressed with Fable 5. I used the £18 subscription, and asked it to convert the document processing of Practal Zero [1] from running in the same thread as the UI to a worker thread. Just two days before I gave the same task to Codex, and the result was not really nice: it would copy the entire document to the worker thread as a snapshot for processing, and so on. Fable instead realised that it could make…

Umm? I'm getting usage capped on single prompts of Fable 5 with the $20 subscription.
Post reply on HN