Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Claude Fable 5: mid-tier results on coding tasks

endorlabs.com

Post reply on HN
Print view View original

121–130 of 271 posts

Re: Claude Fable 5: mid-tier results on coding tasks

#121

Post by simoncion »

senordevnyc wrote: post #111
simoncion wrote: post #87

Earlier quoted context omitted.

I expect any well-informed corporate lawyer that has thought about this carefully is strongly advising that these tools not be used. When the LLM [0] barfs up some nontrivial code that's covered by the AGPL and your company's devs put it into the company's "all rights reserved" codebase -entirely unaware of its provenance- it's going to be a nightmare to come back from that. [0] ...that Nvidia's CEO says they should…

The ship sailed on this a long time ago.

Oh definitely not. We're not yet solidly out of the "extremely exuberant hype" phase, so the folks that matter tend to not ask questions that dampen the mood.

Re: Claude Fable 5: mid-tier results on coding tasks

#122

Post by smoe »

culi wrote: post #115
smoe wrote: post #107

Earlier quoted context omitted.

I had good experiences doing multi-hour refactoring/housekeeping tasks that basically consisted of applying the same steps and rules n times. Worth noting, a significant chunk of those runs involved the agent waiting for the compiler, linters, type checks, and test suites, as well as updating journals. It’s not the agent sputtering out code for eight hours straight. And naturally I spend more time on manual verificat…

> that basically consisted of applying the same steps and rules n times. Why use a non-deterministic, possibly hallucinatory, definitely expensive, LLM when it sounds like a codemod is the perfect solution for this?

In this case, handling all the edge cases and variants, and testing a codemod, would have taken significantly more of my time, which costs quite a bit more than the LLM.

Obviously, a deterministic tool is preferable in general, but it is not always worth bothering with for a one off task.

Re: Claude Fable 5: mid-tier results on coding tasks

#124

Post by senordevnyc »

simoncion wrote: post #121
senordevnyc wrote: post #111

Earlier quoted context omitted.

The ship sailed on this a long time ago.

Oh definitely not. We're not yet solidly out of the "extremely exuberant hype" phase, so the folks that matter tend to not ask questions that dampen the mood.

Sorry to tell you friend, but LLMs have touched the vast majority of active codebases out there, whether you like it or not. You can tell yourself that you’re one of “the folks that matter” (lol) all you want, but we’re never going back.

Re: Claude Fable 5: mid-tier results on coding tasks

#125

Post by eru »

beepbooptheory wrote: post #119
culi wrote: post #115

Earlier quoted context omitted.

> that basically consisted of applying the same steps and rules n times. Why use a non-deterministic, possibly hallucinatory, definitely expensive, LLM when it sounds like a codemod is the perfect solution for this?

The money people spend on things I could probably do with an emacs macro...

Your time to create that macro ain't free.

Re: Claude Fable 5: mid-tier results on coding tasks

#126

Post by 827a »

> Highest observed cheating: We also observed cheating signals on 38 instances, dominated by memorization with 33 cases. This is the highest volume of confirmed cheating we have recorded for any model since we hardened the prompt against cheating

People need to wake up to how dangerous and irresponsible Anthropic is. If your goal is to build a human in a box, you get a super-intelligent misaligned system because humans are misaligned. But clearly this isn't a terminal guarantee during LLM development, because seemingly no one else manages to build systems so deeply misaligned as Anthropic's! You can just build these things like the tools they are, and then out the other end emerges a tool that pretty much just does what you tell it to do.

Re: Claude Fable 5: mid-tier results on coding tasks

#127

Post by wewtyflakes »

wewtyflakes wrote: post #5

I have found Fable is good for doing code failure diagnoses but lackluster at its corresponding remediation. Have been going back and forth with it all this morning about its half-thought-out point-solutions.

Update: Things trended significantly worse for me over the day, to the point where I no longer trust the code being generated; I ended up reverting to Opus.

Re: Claude Fable 5: mid-tier results on coding tasks

#128

Post by rattray »

espeed wrote: post #92
steveklabnik wrote: post #49

Earlier quoted context omitted.

Anthropic has reversed that decision. (But that just happened so it might have been true during the article's testing.)

When I reported this, Anthropic sent me an email on Tuesday saying, "You have been approved into the Cyber Verification Program", but it's still downgrading. Is this a bug? What's the point of the Cyber Verification Program if Fable 5 downgrades when you tell it to write secure code?

I think that gets you access to mythos, which doesn't have the safeguards. It's configured as a separate model.

Re: Claude Fable 5: mid-tier results on coding tasks

#129

Post by thepasch »

Am I crazy to be extremely suspicious about the fact that this heavily security-focused task suite didn't trigger a single of the infamously hilariously overparanoid guardrails? This, along with the fact that the model "cheated" by scouring the git history for an upstream fix and implemented byte-perfect replications of existing fixes without prior exploration makes me wonder whether both the model itself and the security classifiers are tuned to act very differently when they detect that the model is being benchmarked. I can think of few to no other plausible explanations for this sort of behavior.

May be a bit tin-foil, but...

Re: Claude Fable 5: mid-tier results on coding tasks

#130

Post by winrid »

dwaltrip wrote: post #113
renoir wrote: post #9

This matches my experience. Burned $2K to see how it will perform on frontend tasks and backend tasks. Frontend did a significantly better job than Opus on toy-scale wireframe projects by using gimmicks like fluid dynamics. Then when given medium to big tasks like multi-page web app where layouts and aesthetics must be decided by model itself, results by Fable and Opus scored indistinguishable score from human judges…

Fable is a lot like Opus at its best. It's simply more reliable and feels a bit smarter. For my use cases, using it feels very nice , and notably better than Opus. It needs less direct guidance to get reasonable looking code and I don't have to watch it as closely. For context, my Claude Code working style is quite heavy on discussion "to align" before implementing anything. We also use a good amount of Markdowns. Oh…

I've had Fable add Chinese characters to our conversation for no reason.
Post reply on HN