Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Claude Fable 5: mid-tier results on coding tasks

endorlabs.com

Post reply on HN
Print view View original

261–270 of 271 posts

Re: Claude Fable 5: mid-tier results on coding tasks

#261

Post by abc42 »

graphime wrote: post #192
aleph_minus_one wrote: post #191

Earlier quoted context omitted.

> Burned $2K to see how it will perform on frontend tasks and backend tasks. When I read such statements on HN, I nearly always ask myself: if the person has such an amount of money to burn, don't there exist much more fun opportunities to burn buckets of money than doing such experiments on LLMs?

> if the person has such an amount of money to burn, don't there exist much more fun opportunities to burn buckets of money than doing such experiments on LLMs? Do you think US$2,000 is a lot of money?

Do you think $10 is a lot of money for a carton of milk? I think it is.

Re: Claude Fable 5: mid-tier results on coding tasks

#262

Post by [deleted] »

saurik wrote: post #258
Scene_Cast2 wrote: post #54

Earlier quoted context omitted.

On my (admittedly weird) setup, GPT-5.5 Pro times out. The reading is off because the thermistor resistance also depends on applied voltage, not just temperature. LLMs couldn't get this even after feeding them multimeter voltage readings, not just ADC readings. They went into guessing much more esoteric things like ADC switched-capacitor input current, burnout-detect current sources or IDACs left enabled, board leaka…

This is the kind of problem I expect Claude to be useless at, and while I could see Gemini Deep Think making a good showing, I'd only bother with ChatGPT Pro. FWIW, I do believe it got the correct answer as one of its first two suggestions (though I am not an electrical engineer, so maybe I am not understanding this given the vague/summarized prompt). https://chatgpt.com/share/6a2d8c75-56f4-83e8-a61a-301e4c62b1...

[deleted]

Re: Claude Fable 5: mid-tier results on coding tasks

#263

Post by DELTRON2040 »

saurik wrote: post #258
Scene_Cast2 wrote: post #54

Earlier quoted context omitted.

On my (admittedly weird) setup, GPT-5.5 Pro times out. The reading is off because the thermistor resistance also depends on applied voltage, not just temperature. LLMs couldn't get this even after feeding them multimeter voltage readings, not just ADC readings. They went into guessing much more esoteric things like ADC switched-capacitor input current, burnout-detect current sources or IDACs left enabled, board leaka…

This is the kind of problem I expect Claude to be useless at, and while I could see Gemini Deep Think making a good showing, I'd only bother with ChatGPT Pro. FWIW, I do believe it got the correct answer as one of its first two suggestions (though I am not an electrical engineer, so maybe I am not understanding this given the vague/summarized prompt). https://chatgpt.com/share/6a2d8c75-56f4-83e8-a61a-301e4c62b1...

[dead]

Re: Claude Fable 5: mid-tier results on coding tasks

#264

Post by comboy »

Al-Khwarizmi wrote: post #150
comboy wrote: post #29

Earlier quoted context omitted.

'by the way, your previous attempts have these structural problems." Just to be clear, it did not have access to any previous work that opus did? Because they are pretty good at digging out relevant tmp files and making use of whatever is out there. With my fable adventures I caught it hallucinating something and stating it as a fact in CLI twice. And it was something that I did not see opus do in such way, opus obvi…

If you have some spare time, I'd be interested in knowing what kind of questions you use to test models on understanding of Chinese culture.

I'm creating hanzirama.com

I generate explanations for characters and words like so: https://hanzirama.com/character/%E6%9D%A5#explain

But I don't want to mislead learners and want to provide some cultural depth, so I have a hole sophisticated pipeline, using multiple models to generate the explanation, then multiple models look for issues in the explanation, each issue goes through the panel of judges (basically trying to squash down any hallucinations), it's fixed and it goes through such cycles a few times over.

I've been at it for some months now, so I have dozens of different probes, that I needed to evaluate prompts and method changes. Plus on some items I generated so many explanations through different means that I can tell a lot about given model just by looking at one.

Plus I'm doing some statistics, so I see how e.g. when working as judges of issues some models correlate heavily with some others... Fun fact during some testing runs basically just testing providers I stumbled upon qwen introducing himself as made by Google. And also Anhropic's Sonnet saying that it was made by OpenAI :)

At this point all my evaluations frameworks and pipelines stuff is much bigger than the site itself. I'm having lots of fun though.

Re: Claude Fable 5: mid-tier results on coding tasks

#265

Post by Resoluciones »

renoir wrote: post #9

This matches my experience. Burned $2K to see how it will perform on frontend tasks and backend tasks. Frontend did a significantly better job than Opus on toy-scale wireframe projects by using gimmicks like fluid dynamics. Then when given medium to big tasks like multi-page web app where layouts and aesthetics must be decided by model itself, results by Fable and Opus scored indistinguishable score from human judges…

[dead]

Re: Claude Fable 5: mid-tier results on coding tasks

#266

Post by nujabe »

int_19h wrote: post #89
jasondigitized wrote: post #48

Earlier quoted context omitted.

A single 8h task? I'm sorry, but that's just asking for trouble.

My record for a single uninterrupted session (albeit with Codex, not Claude) is 80+ hours. It was very productive, too. The trick is having large, extensive test suites and forcing the agent to run them regularly.

An agent can’t have an “uninterrupted session” if you have to be “forcing” it do stuff.

Re: Claude Fable 5: mid-tier results on coding tasks

#267

Post by compaluca »

bensyverson wrote: post #4

> The dominant mechanism, and the one no prompt instruction can prevent: the model has simply seen the upstream fix during training and reproduces it… > On numpy, the patch is 100% character-for-character identical to the golden patch… down to idiosyncratic comments like "Extending singleton dimension for 'reflect' is legacy behavior; it really should raise an error." This… seems like a flaw in the benchmark suite me…

[flagged]

Re: Claude Fable 5: mid-tier results on coding tasks

#268

Post by int_19h »

nujabe wrote: post #266
int_19h wrote: post #89

Earlier quoted context omitted.

My record for a single uninterrupted session (albeit with Codex, not Claude) is 80+ hours. It was very productive, too. The trick is having large, extensive test suites and forcing the agent to run them regularly.

An agent can’t have an “uninterrupted session” if you have to be “forcing” it do stuff.

"Forcing" here basically means giving initial instructions that clearly require passing the tests as a condition of finishing the work. The agent still works uninterrupted.

Re: Claude Fable 5: mid-tier results on coding tasks

#269

Post by int_19h »

danmaz74 wrote: post #204
int_19h wrote: post #89

Earlier quoted context omitted.

My record for a single uninterrupted session (albeit with Codex, not Claude) is 80+ hours. It was very productive, too. The trick is having large, extensive test suites and forcing the agent to run them regularly.

So I guess that a lot of those 80 hours were spent running the test suite between changes?

Yep. I should add that the current crop of models is much more tolerant of something like this, compared to where we were a year ago - as in, they are quite willing to wait for a long time for the test or profiling run to finish without giving up on it, if the instructions make it clear that this is normal and expected.

Re: Claude Fable 5: mid-tier results on coding tasks

#270

Post by m101 »

I have had another experience with Opus 4.8 vs 5.5 which is along the lines of my prior experience of Opus generally having more common sense, whilst GPT is something along the lines of myopic code execution agent:

I was asking them each to review a regulatory pack for planning which regulatory track I needed to follow for my project. The GPT route was to check that the sources were properly referenced and the document structure was accurate and consistent. The Opus approach (the common sense approach, and what I wanted) was to check whether the argument was strong and I had a case.

This type of common sense intelligence difference is something I've seen a lot of times between the two. This doesn't show itself as a difference in a clearly defined coding task however, but there is something there which is qualitatively worse in openai models.

Post reply on HN