Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Claude Fable 5: mid-tier results on coding tasks

endorlabs.com

Post reply on HN
Print view View original

251–260 of 271 posts

Re: Claude Fable 5: mid-tier results on coding tasks

#251

Post by queuebert »

okamiueru wrote: post #237
queuebert wrote: post #223

Earlier quoted context omitted.

> ... applying the same steps and rules n times I do this too, with a document written for this purpose. > ... a significant chunk of those runs involved the agent waiting for the compiler, linters, type checks, and test suites, as well as updating journals. That is a good point. I'm mostly using C, which seemingly compiles in O(1) time, so I could imagine a large C++ or Rust codebase taking much longer to iterate si…

What do you mean by C compiling in O(1)? Is that what the LLM told you?

It's a joke about how fast it compiles. whoosh

Re: Claude Fable 5: mid-tier results on coding tasks

#253

Post by corroclaro »

Fable 5 has for me, been surprisingly effective at generalizing and removing duplicate solutions/approaches - things I'd been noting along as my mental TODO once Opus finished generating them, Fable also saw them and prepared a plan to correct them.

Worked pretty well. Also writes lisp a lot better without getting lost in parentheses! I do keep hitting the limit regularly but it is doing a lot of work that would have taken me a long time to write by hand even if per se not super complex.

Re: Claude Fable 5: mid-tier results on coding tasks

#254

Post by senordevnyc »

customguy wrote: post #240
senordevnyc wrote: post #232

Earlier quoted context omitted.

You don’t even know what we’re talking about in this thread, do you? We’re talking about whether corporations are going to risk using LLMs in their codebase because of the theoretical legal risk that they might produce something that would fall under open source licenses, and be difficult to untangle later. Regardless of what you think the morality is here, or what the legal situation turns out to be, this is already…

Why repeat what you already said with more words, as if I can't read, only to leave out the bit that I responded to? > we’re never going back. As a prediction, this is worthless. If everybody thinks as you do, we won't, if nobody does, we will. So yes, this is purely about morality.

If you think software engineering is ever going back to being widely done without AI…no idea what to tell you.

Re: Claude Fable 5: mid-tier results on coding tasks

#255

Post by dwaltrip »

sulam wrote: post #214
dwaltrip wrote: post #145

Earlier quoted context omitted.

Hah, yeah... I added this to my global CLAUDE.md (~/.claude/CLAUDE.md): ## Writing voice — plain, factual, calibrated to the evidence Write docs, session notes, commit messages, and findings plainly and factually — and calibrate every claim you assert, in chat as much as in writing. This guards against a known LLM tendency to inflate: toward punchy phrasing and claims that read as more settled than the work supports.…

Have you tried optimizing this prompt so that it’s shorter but gets the same results? I see these super verbose prompts all the time from people who learned prompt engineering in the ‘24-early ‘25 timeframe and they seem unnecessary to me (I get good results with 1-3 sentences) but I hate to assume other people’s experience mirrors my own.

That's a good idea. Claude wrote that for me a week or so ago. It could definitely be tightened.

Re: Claude Fable 5: mid-tier results on coding tasks

#256

Post by afro88 »

munksbeer wrote: post #226
afro88 wrote: post #8

Similar result on our kotlin coding benchmark at work. It measures how close agents can get to a small mergable PR (according to my team). 20 tasks of varying difficulty, with 5 attempts each, LLM as judge to evaluate accuracy (same outcome and quality but allowing for acceptable variances). Fable 5 sits ahead of Opus 4.7, but behind Opus 4.6, Sonnet 4.6, Opus 4.8, GPT-5.4, GPT-5.5. Fable isn't a good coding workhors…

Does your team then manually decide the results by going over the PRs? I suppose you know what you're looking for now, but isn't this still quite painful?

We selected PRs (real ones we merged over the 6 months prior) and have an "LLM as judge" score how close the AI generated code is to the PR. Same as how other benchmarks do it, but it's with tasks we actually do and code we have decided is actually up to scratch for us

Re: Claude Fable 5: mid-tier results on coding tasks

#257

Post by dwaltrip »

moffkalast wrote: post #181
willsmith72 wrote: post #146

Earlier quoted context omitted.

Yeah same here, it's a huge step up for me. Curious why people are having such different experiences. Is it just to do with what they're working on? Specific prompt styles (eg overfitting on opus)?

I would go out on a limb and say it's a garbage in garbage out problem. People just don't define their problem well enough nor provide enough context and are surprised the model can't magically read their mind and summon data that doesn't exist from thin air. There's only so much raw intelligence can compensate for not having literally anything to go on. 10 years ago this was a joke, now it's Tuesday: https://old.red…

That’s so wild to read that 10 year old meme post. Very prescient. And yes, so accurate! hah

Re: Claude Fable 5: mid-tier results on coding tasks

#258

Post by saurik »

Scene_Cast2 wrote: post #54
threatripper wrote: post #39

Earlier quoted context omitted.

Did you also test GPT-5.5 Pro web version? Why is the voltage reading 17% off?

On my (admittedly weird) setup, GPT-5.5 Pro times out. The reading is off because the thermistor resistance also depends on applied voltage, not just temperature. LLMs couldn't get this even after feeding them multimeter voltage readings, not just ADC readings. They went into guessing much more esoteric things like ADC switched-capacitor input current, burnout-detect current sources or IDACs left enabled, board leaka…

This is the kind of problem I expect Claude to be useless at, and while I could see Gemini Deep Think making a good showing, I'd only bother with ChatGPT Pro. FWIW, I do believe it got the correct answer as one of its first two suggestions (though I am not an electrical engineer, so maybe I am not understanding this given the vague/summarized prompt).

https://chatgpt.com/share/6a2d8c75-56f4-83e8-a61a-301e4c62b1...

Re: Claude Fable 5: mid-tier results on coding tasks

#259

Post by Natfan »

graphime wrote: post #192
aleph_minus_one wrote: post #191

Earlier quoted context omitted.

> Burned $2K to see how it will perform on frontend tasks and backend tasks. When I read such statements on HN, I nearly always ask myself: if the person has such an amount of money to burn, don't there exist much more fun opportunities to burn buckets of money than doing such experiments on LLMs?

> if the person has such an amount of money to burn, don't there exist much more fun opportunities to burn buckets of money than doing such experiments on LLMs? Do you think US$2,000 is a lot of money?

yes.

Re: Claude Fable 5: mid-tier results on coding tasks

#260

Post by abc42 »

aleph_minus_one wrote: post #191
renoir wrote: post #9

This matches my experience. Burned $2K to see how it will perform on frontend tasks and backend tasks. Frontend did a significantly better job than Opus on toy-scale wireframe projects by using gimmicks like fluid dynamics. Then when given medium to big tasks like multi-page web app where layouts and aesthetics must be decided by model itself, results by Fable and Opus scored indistinguishable score from human judges…

> Burned $2K to see how it will perform on frontend tasks and backend tasks. When I read such statements on HN, I nearly always ask myself: if the person has such an amount of money to burn, don't there exist much more fun opportunities to burn buckets of money than doing such experiments on LLMs?

Especially when the $200 / month subscription will give anyone enough credits to drive opus 4.8 max 12 hours every day.
Post reply on HN