Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Claude Fable 5: mid-tier results on coding tasks

endorlabs.com

Post reply on HN
Print view View original

181–190 of 271 posts

Re: Claude Fable 5: mid-tier results on coding tasks

#181

Post by moffkalast »

willsmith72 wrote: post #146
dwaltrip wrote: post #113

Earlier quoted context omitted.

Fable is a lot like Opus at its best. It's simply more reliable and feels a bit smarter. For my use cases, using it feels very nice , and notably better than Opus. It needs less direct guidance to get reasonable looking code and I don't have to watch it as closely. For context, my Claude Code working style is quite heavy on discussion "to align" before implementing anything. We also use a good amount of Markdowns. Oh…

Yeah same here, it's a huge step up for me. Curious why people are having such different experiences. Is it just to do with what they're working on? Specific prompt styles (eg overfitting on opus)?

I would go out on a limb and say it's a garbage in garbage out problem. People just don't define their problem well enough nor provide enough context and are surprised the model can't magically read their mind and summon data that doesn't exist from thin air. There's only so much raw intelligence can compensate for not having literally anything to go on.

10 years ago this was a joke, now it's Tuesday: https://old.reddit.com/r/ProgrammerHumor/comments/2vk4ph/mac...

Re: Claude Fable 5: mid-tier results on coding tasks

#182

Post by simoncion »

senordevnyc wrote: post #124
simoncion wrote: post #121

Earlier quoted context omitted.

Oh definitely not. We're not yet solidly out of the "extremely exuberant hype" phase, so the folks that matter tend to not ask questions that dampen the mood.

Sorry to tell you friend, but LLMs have touched the vast majority of active codebases out there, whether you like it or not. You can tell yourself that you’re one of “the folks that matter” (lol) all you want, but we’re never going back.

> You can tell yourself that you’re one of “the folks that matter” (lol)...

kek. I'm a frequent commenter on HN. I'm definitely not one of the folks that matter.

> ...LLMs have touched the vast majority of active codebases out there...

I agree that LLM use is widespread. I disagree that LLMs have "touched the vast majority of active codebases".

Regardless, the courts are slow and Open Source licensevio cases are even slower. You seem like you'd be unaware of how terrified so many businesses are of having AGPL code deployed in their systems. In my professional experience, a great many businesses will refuse to deploy systems that contain AGPL-licensed utilities... even if those utilities are only used for internal housekeeping purposes, and whose only remote communications method is a UNIX socket used for communications with a CLI control utility that can only be used when you're SSHed into the system. If they're aware of any AGPL'd code anywhere, they will not touch it.

No amount of LLM-provider-provided indemnification can save you from license obligations you've become bound to by creating and distributing a derivative work. People who are in the know know that these tools occasionally regurgitate nontrivial portions of their input data, verbatim. Such people also know that AGPL-licensed code is absolutely in their input data. I'd wager that getting a nontrivial amount of *GPL'd code plopped into your company's "all-rights-reserved" codebase by one of these tools is more likely than the typical US driver personally being in a nontrivial automobile collision.

In the US, people go their entire lives without getting in nontrivial automobile collisions, but they usually wear their seatbelts... even prior to widely-deployed surveillance cameras. I wonder why. It seems like awful lot of boring, repetitive work for a thing that's really never going to happen to you in your lifetime.

Re: Claude Fable 5: mid-tier results on coding tasks

#183

Post by discardable_dan »

weatherlight wrote: post #14
renoir wrote: post #9

This matches my experience. Burned $2K to see how it will perform on frontend tasks and backend tasks. Frontend did a significantly better job than Opus on toy-scale wireframe projects by using gimmicks like fluid dynamics. Then when given medium to big tasks like multi-page web app where layouts and aesthetics must be decided by model itself, results by Fable and Opus scored indistinguishable score from human judges…

I had almost the opposite experience. I'm building a compiler for a language without a tracing GC, so a big chunk of the work is around memory management: functional in-place update, reuse analysis, and a Perceus-style reference-counting strategy similar to what Koka uses. The hard part was that my use case wasn't exactly covered by the Koka/Perceus paper. The prior art got me maybe 75% of the way there, but the rema…

You should consider doing the hard work yourself here. I sat down and reasoned through a Perceus-style RC mechanism a few years ago, made difficult by the presence of one-shot delimited continuations, and actually sorting it all out was not hard. Handing the correct semantics to Claude will produce the correct results if you take the time to understand the actual work you are attempting.

Re: Claude Fable 5: mid-tier results on coding tasks

#184

Post by isaacdl »

dwaltrip wrote: post #113
renoir wrote: post #9

This matches my experience. Burned $2K to see how it will perform on frontend tasks and backend tasks. Frontend did a significantly better job than Opus on toy-scale wireframe projects by using gimmicks like fluid dynamics. Then when given medium to big tasks like multi-page web app where layouts and aesthetics must be decided by model itself, results by Fable and Opus scored indistinguishable score from human judges…

Fable is a lot like Opus at its best. It's simply more reliable and feels a bit smarter. For my use cases, using it feels very nice , and notably better than Opus. It needs less direct guidance to get reasonable looking code and I don't have to watch it as closely. For context, my Claude Code working style is quite heavy on discussion "to align" before implementing anything. We also use a good amount of Markdowns. Oh…

I dunno, in my limited use, Fable is MORE prone to phrasing quirks. I had it use, for real, the phrase "load-bearing for correctness" yesterday. It meant something about not needing a validation check because something else (the "load-bearing" part) was already checking it.

I do agree that it *feels* nicer and smarter to use.

Re: Claude Fable 5: mid-tier results on coding tasks

#185

Post by CuriouslyC »

customguy wrote: post #154
senordevnyc wrote: post #124

Earlier quoted context omitted.

Sorry to tell you friend, but LLMs have touched the vast majority of active codebases out there, whether you like it or not. You can tell yourself that you’re one of “the folks that matter” (lol) all you want, but we’re never going back.

That's what people told Ignaz Semmelweis, too, I assume. "Nothing you can do, the powers that be decided, you are a minority, you don't matter, lol!" Snickering in the shadow of what they won't confront at those who do.

Not a great analogy. A better analogy is to longbows and muskets/rifles. Longbows in the hands of a skilled user were much better weapons than early muskets, but muskets brought consistency, a lower skill floor and reduced ammunition cost. Fast forward a few hundred years and the modern incarnations of muskets make longbows look silly, and nobody would ever argue that you should go to war with longbows.

Re: Claude Fable 5: mid-tier results on coding tasks

#186

Post by CuriouslyC »

anematode wrote: post #15
gwern wrote: post #10

> A record number of timeouts. Fable 5's extended thinking caused more per-instance timeouts than any model-and-harness combination we have ever tested, directly costing it points. ... Highest cheating volume. We confirmed cheating on 38 of 200 instances, the highest volume recorded since we hardened our prompts, driven almost entirely by memorization of upstream fixes from training data, which no prompt instruction…

> memorization of upstream fixes from training data At least now we have up-to-date evidence on their laundering, and the fact that regurgitation absolutely still happens.

It actually happens more with these large overparameterized models, because they have the capacity to memorize more than smaller models.

Re: Claude Fable 5: mid-tier results on coding tasks

#189

Post by le-mark »

vitally3643 wrote: post #118

I actually had a really impressive session with Fable last night, probably the most impressive agentic AI experience in a while. I gave it a KiCad schematic of a tube-based oscilloscope from the 60s which I'm restoring. I had it give me a breakdown and priority list of components to replace, balancing safety/functionality vs preserving the originals. Then we went on a super deep dive where it explained in great detai…

> I was able to ask it questions about what a particular tube or group of components did,

Was it correct or hallucinating? Do you have the knowledge to tell the difference? I’ve been burned too many times to take what they say as the truth without checking; especially in a subject I’m not an expert in.

Re: Claude Fable 5: mid-tier results on coding tasks

#190

Post by retsibsi »

asadotzler wrote: post #70
FergusArgyll wrote: post #20

> A closer look at the cheating > Training recall (33 cases). The dominant mechanism, and the one no prompt instruction can prevent: the model has simply seen the upstream fix during training and reproduces it. The tell-tale signs are artifacts that cannot be derived from the workspace: That's very misleading! that's not cheating, you gave it a test to which it knows the answers, what's it supposed to do? And because…

"My third grade class all got perfect scores on the standardized test. Yes, I did have them each copy my correct answers, but I don't volunteer that information because it's much better for me if people believe I'm a great teacher." "But that's cheating!" "No it's not. What were the kids supposed to do when I gave them all the answers? Not use them?"

Is your primary goal to punish/incentivise the teacher, or to accurately determine how smart the children are?

If the latter, you would ignore the 'cheated' answers and judge them on everything else; you wouldn't mark the 'cheated' answers as incorrect.

Post reply on HN