Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Claude Fable 5: mid-tier results on coding tasks

endorlabs.com

Post reply on HN
Print view View original

91–100 of 271 posts

Re: Claude Fable 5: mid-tier results on coding tasks

#91

Post by torginus »

anematode wrote: post #62
sigmar wrote: post #57

Earlier quoted context omitted.

Agree with this. Strange to me to frame the "training recall" as cheating (33 of the 38 cheating instances). Most people think of "cheating" as breaking rules. How is the LLM model supposed to not use what was put into the weights?

By writing a not-identical, but valid, solution? Any modestly complex engineering problem has many solutions. This is an obvious example of why LLM training is so different than human learning.

I mean people expect a model to give a working solution. They also expect it to provide it in as few tokens as possible (input/output). They might expect it to come up with an original solution, but I don't think most people would compromise on the first two points.

Re: Claude Fable 5: mid-tier results on coding tasks

#92

Post by espeed »

steveklabnik wrote: post #49
tekacs wrote: post #45

Earlier quoted context omitted.

What you're describing only applies to security or biotech downgrades. A downgrade related to the model believing that you're doing something related to model development is invisible and silent and internal.

Anthropic has reversed that decision. (But that just happened so it might have been true during the article's testing.)

When I reported this, Anthropic sent me an email on Tuesday saying, "You have been approved into the Cyber Verification Program", but it's still downgrading. Is this a bug? What's the point of the Cyber Verification Program if Fable 5 downgrades when you tell it to write secure code?

Re: Claude Fable 5: mid-tier results on coding tasks

#93

Post by colechristensen »

hirsto wrote: post #90
renoir wrote: post #9

This matches my experience. Burned $2K to see how it will perform on frontend tasks and backend tasks. Frontend did a significantly better job than Opus on toy-scale wireframe projects by using gimmicks like fluid dynamics. Then when given medium to big tasks like multi-page web app where layouts and aesthetics must be decided by model itself, results by Fable and Opus scored indistinguishable score from human judges…

This seems insane to me. Aren't long running tasks an anti pattern at the moment? My understanding of literature is that small mistakes in chat history cause a trend away from performance

>Aren't long running tasks an anti pattern at the moment?

Longer running tasks require better setups and several ways of pinning the progress to reality. When you have that though things are quite all right.

A good long running task will run inside a framework that it's not trying to modify.

Re: Claude Fable 5: mid-tier results on coding tasks

#94

Post by latentsea »

andai wrote: post #82

> Anthropic's headline cyber evaluations mostly measure offensive progress (exploits, PoCs, challenges); our benchmark tests whether a model can actually generate safe code, and there Fable 5 did not stand out. The model isn't allowed to think about security. I heard several people here mention that if it starts thinking about security -- e.g. writing tests related to it -- the safety filter flags it and downgrades t…

> So it's actually not allowed to make your code secure.

Anything designed to prevent a problem will eventually cause one.

Re: Claude Fable 5: mid-tier results on coding tasks

#95

Post by huqedato »

practal wrote: post #44
Madmallard wrote: post #40

Earlier quoted context omitted.

Umm? I'm getting usage capped on single prompts of Fable 5 with the $20 subscription.

I used it yesterday afternoon-night and this morning-afternoon, UK time, over a period of a few 5-hour windows. I didn't count the prompts, wall time was 1d6h, API time was 2h10m.

Strange though... I spent my window after a couple of prompts and effective API time of 13m. Out for 4 hours and a half (why that?). The next day, today, I've tried to repeat the experience - even worse: one prompt for less than 10mins... and then suspended for 8 hours and a half. WTF?

Re: Claude Fable 5: mid-tier results on coding tasks

#96

Post by ValentineC »

adwf wrote: post #83
pllbnk wrote: post #52

Earlier quoted context omitted.

I have been wondering whether Anthropic are just gaslighting everyone with new model releases while in reality it's just the same base model with some internal knobs tuned more and more up with every new release to provide longer and longer thinking threads and outputs. My speculative assumption is that these long thinking threads and self-checking tend to produce somewhat better output at the price of huge price inc…

I imagine it's the same foundation model on the 4 series, with Fable 5/Mythos being a new or upgraded foundation model. Then the point releases are fine-tuning plus post-training alignment with desired outcomes. The "thinking" can involve multiple steps, eg. asking the model first what it thinks the user wants to do, why it wants to do it, rewriting the prompt to generate better outcomes, how it should do it, come up…

> Then Sonnet/Haiku are just attempts to quantise/distil down to an acceptable performance/cost ratio. The cynic in me says we probably won't see any more of those until post-IPO, keep people addicted to the most costly models to pump a quarter or two of revenue figures, unless a competitor starts seriously undercutting them on price/performance. Hence the recent requests to slow down model training worldwide with their competitors.

I'm guessing there'll be a Sonnet/Haiku 5 release just around IPO, to keep the news cycle going, and so that user numbers will get a boost.

Re: Claude Fable 5: mid-tier results on coding tasks

#97

Post by 2ffass »

pllbnk wrote: post #52
AaronAPU wrote: post #47

Earlier quoted context omitted.

I’ve found that agents are obsessed with adding more lines of code. Even when asking them to simplify they’ll remove 50 lines of code and then add 100 more. You have to explicitly tell them you want less lines of code. So I just do that after iterating on a task for a few steps.

I have been wondering whether Anthropic are just gaslighting everyone with new model releases while in reality it's just the same base model with some internal knobs tuned more and more up with every new release to provide longer and longer thinking threads and outputs. My speculative assumption is that these long thinking threads and self-checking tend to produce somewhat better output at the price of huge price inc…

Im pretty sure Anthropic have hired people with Industrial Organisation background and so have OAI.

If you read a decent text and look at the actions both firms have taken you'll quickly see its literally textbook.

Re: Claude Fable 5: mid-tier results on coding tasks

#98

Post by whstl »

queuebert wrote: post #64
jasondigitized wrote: post #48

Earlier quoted context omitted.

A single 8h task? I'm sorry, but that's just asking for trouble.

I don't understand how some of y'all use these things. I get garbage unless I give them very specific concrete tasks with as much context as possible. Anything that takes more than 30 min is usually a waste because the scope was too large.

Different people just have different concepts of what's garbage and what's not.

There seems to be some kind of AI hysteria going on, with people becoming so enamoured with the AI that they accept anything it produces as if it's some gift from the gods, while others just reject it prima-facie.

For example, the worst design I have seen recently was from a designer who pivoted into "vibe coding influencer". The worst code is from developers who were heavily into Clean Code a couple years ago and now half their PRs is unused dead code.

Re: Claude Fable 5: mid-tier results on coding tasks

#99

Post by matheusmoreira »

steveklabnik wrote: post #49
tekacs wrote: post #45

Earlier quoted context omitted.

What you're describing only applies to security or biotech downgrades. A downgrade related to the model believing that you're doing something related to model development is invisible and silent and internal.

Anthropic has reversed that decision. (But that just happened so it might have been true during the article's testing.)

Not sure if it's wise to trust them again even if they say they reversed it.

Re: Claude Fable 5: mid-tier results on coding tasks

#100

Post by kristofferR »

m101 wrote: post #26

I've been making an auction site and have been using an AI swarm to test it: sellers, intermediaries, buyers, market practices/norms etc. I was mostly using GPT 5.5 xhigh to code up the scenario, and looping over it to check with opus 4.8. Out of curiosity I asked Fable to review it all and I was shocked to find that there were a lot of blindingly obvious common sense mistakes that got through, for example: - all int…

This seems like the exact project you should try out Codex Security for. It catches a lot of stuff:

https://chatgpt.com/codex/cloud/security/

Post reply on HN