Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Claude Fable 5: mid-tier results on coding tasks

endorlabs.com

Post reply on HN
Print view View original

111–120 of 271 posts

Re: Claude Fable 5: mid-tier results on coding tasks

#111

Post by senordevnyc »

simoncion wrote: post #87
anematode wrote: post #62

Earlier quoted context omitted.

By writing a not-identical, but valid, solution? Any modestly complex engineering problem has many solutions. This is an obvious example of why LLM training is so different than human learning.

I expect any well-informed corporate lawyer that has thought about this carefully is strongly advising that these tools not be used. When the LLM [0] barfs up some nontrivial code that's covered by the AGPL and your company's devs put it into the company's "all rights reserved" codebase -entirely unaware of its provenance- it's going to be a nightmare to come back from that. [0] ...that Nvidia's CEO says they should…

The ship sailed on this a long time ago.

Re: Claude Fable 5: mid-tier results on coding tasks

#112

Post by nl »

maxall4 wrote: post #65
jasondigitized wrote: post #48

Earlier quoted context omitted.

A single 8h task? I'm sorry, but that's just asking for trouble.

Indeed, according to METR, Mythos only achieved an 80% success rate with 3 hour tasks. https://metr.org/time-horizons/

I use both Opus and Fable on tasks that are well beyond "things that would take a human 3 hours"

It fails all the time - as in it ends up doing something I want to change.

But this doesn't actually matter - if it takes 3 or 4 iterations on something that would have taken me a week it might be a day of human work, but it's still 5 times better than doing it by hand.

Re: Claude Fable 5: mid-tier results on coding tasks

#113

Post by dwaltrip »

renoir wrote: post #9

This matches my experience. Burned $2K to see how it will perform on frontend tasks and backend tasks. Frontend did a significantly better job than Opus on toy-scale wireframe projects by using gimmicks like fluid dynamics. Then when given medium to big tasks like multi-page web app where layouts and aesthetics must be decided by model itself, results by Fable and Opus scored indistinguishable score from human judges…

Fable is a lot like Opus at its best. It's simply more reliable and feels a bit smarter. For my use cases, using it feels very nice, and notably better than Opus. It needs less direct guidance to get reasonable looking code and I don't have to watch it as closely.

For context, my Claude Code working style is quite heavy on discussion "to align" before implementing anything. We also use a good amount of Markdowns.

Oh yeah, it also is has way less "phrasing quirks" and is a clearer communicator. Opus 4.8 was a bit of loon with some of its writing styles. I had mostly straightened it out, but not entirely. It would use the most ridiculous flair at times.

Re: Claude Fable 5: mid-tier results on coding tasks

#114

Post by steveklabnik »

espeed wrote: post #92
steveklabnik wrote: post #49

Earlier quoted context omitted.

Anthropic has reversed that decision. (But that just happened so it might have been true during the article's testing.)

When I reported this, Anthropic sent me an email on Tuesday saying, "You have been approved into the Cyber Verification Program", but it's still downgrading. Is this a bug? What's the point of the Cyber Verification Program if Fable 5 downgrades when you tell it to write secure code?

I don’t think that’s relevant? The change is that it will no longer silently downgrade, and will instead be honest that it’s doing it in all cases.

Re: Claude Fable 5: mid-tier results on coding tasks

#115

Post by culi »

smoe wrote: post #107
queuebert wrote: post #64

Earlier quoted context omitted.

I don't understand how some of y'all use these things. I get garbage unless I give them very specific concrete tasks with as much context as possible. Anything that takes more than 30 min is usually a waste because the scope was too large.

I had good experiences doing multi-hour refactoring/housekeeping tasks that basically consisted of applying the same steps and rules n times. Worth noting, a significant chunk of those runs involved the agent waiting for the compiler, linters, type checks, and test suites, as well as updating journals. It’s not the agent sputtering out code for eight hours straight. And naturally I spend more time on manual verificat…

> that basically consisted of applying the same steps and rules n times.

Why use a non-deterministic, possibly hallucinatory, definitely expensive, LLM when it sounds like a codemod is the perfect solution for this?

Re: Claude Fable 5: mid-tier results on coding tasks

#116

Post by artdigital »

Also spent the past day using Fable for everything I usually use Opus or gpt-5.5 for. My experience is that it’s a better and more reliable Opus that’s far better in frontend tasks than backend/ios. More similar to gpt-5.5 for long running tasks and reliability.

It still left small bugs and weird behaviors that it cleaned up when I told it about them, but it felt very Opus-ey.

I think for implementing a detailed design doc, I’d put it on par with gpt-5.5 high but farrrr more expensive. I’m eating through my x5 Max plan in no time. I’d use it for reviewing implementations and designs docs as another pass, but it’s too expensive for me for reading a lot of (uncached) code by itself in an agentic loop, especially with medium to high reasoning.

As a daily driver too expensive, that crown still goes to gpt-5.5.

I barely used it in high/xhigh/max reasoning though.

Re: Claude Fable 5: mid-tier results on coding tasks

#117

Post by selimthegrim »

ElFitz wrote: post #77
cmenge wrote: post #25

Earlier quoted context omitted.

Similar. I gave it a really hard task, basically messy code in a complex domain that was bug-ridden from a mess previously created half manually and half by Opus. It cleaned things up beautifully, both the backend and the frontend. Maybe the prompt was particularly well-suited for the model (I instructed it to put on a mathematician's hat, look at the mathematical substructure of the problem, identify invariants and…

> and I have the feeling that the harness is much more important than the consensus expectation. Is that really the consensus? There’s been a bit of literature lately on that. Can’t find the one about looking into whether or not the harness had a greater impact than the models (for comparable models), but there’s this one: https://arxiv.org/html/2605.23950

whoa, my university!

Re: Claude Fable 5: mid-tier results on coding tasks

#118

Post by vitally3643 »

I actually had a really impressive session with Fable last night, probably the most impressive agentic AI experience in a while.

I gave it a KiCad schematic of a tube-based oscilloscope from the 60s which I'm restoring. I had it give me a breakdown and priority list of components to replace, balancing safety/functionality vs preserving the originals. Then we went on a super deep dive where it explained in great detail how the circuit works and what the tubes are doing.

It isn't so impressive that it could explain vacuum tube physics and circuit theory, but it was pretty impressive that it could consume four pages of KiCad schematic and reconstruct the full topology and theory of operation with no additional information. I was able to ask it questions about what a particular tube or group of components did, or how this system interacts with that one, or what the risks and benefits of this design choice or upgrade might be. Very fluid, and its answers were actually really smart.

I have, however, found Fable to be far less impressive on coding tasks.

Re: Claude Fable 5: mid-tier results on coding tasks

#119

Post by beepbooptheory »

culi wrote: post #115
smoe wrote: post #107

Earlier quoted context omitted.

I had good experiences doing multi-hour refactoring/housekeeping tasks that basically consisted of applying the same steps and rules n times. Worth noting, a significant chunk of those runs involved the agent waiting for the compiler, linters, type checks, and test suites, as well as updating journals. It’s not the agent sputtering out code for eight hours straight. And naturally I spend more time on manual verificat…

> that basically consisted of applying the same steps and rules n times. Why use a non-deterministic, possibly hallucinatory, definitely expensive, LLM when it sounds like a codemod is the perfect solution for this?

The money people spend on things I could probably do with an emacs macro...
Post reply on HN