Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Claude Fable 5: mid-tier results on coding tasks

endorlabs.com

Post reply on HN
Print view View original

201–210 of 271 posts

Re: Claude Fable 5: mid-tier results on coding tasks

#201

Post by jurf »

graphime wrote: post #192
aleph_minus_one wrote: post #191

Earlier quoted context omitted.

> Burned $2K to see how it will perform on frontend tasks and backend tasks. When I read such statements on HN, I nearly always ask myself: if the person has such an amount of money to burn, don't there exist much more fun opportunities to burn buckets of money than doing such experiments on LLMs?

> if the person has such an amount of money to burn, don't there exist much more fun opportunities to burn buckets of money than doing such experiments on LLMs? Do you think US$2,000 is a lot of money?

I mean what is that, three bananas?

Re: Claude Fable 5: mid-tier results on coding tasks

#202

Post by dbingham »

graphime wrote: post #192
aleph_minus_one wrote: post #191

Earlier quoted context omitted.

> Burned $2K to see how it will perform on frontend tasks and backend tasks. When I read such statements on HN, I nearly always ask myself: if the person has such an amount of money to burn, don't there exist much more fun opportunities to burn buckets of money than doing such experiments on LLMs?

> if the person has such an amount of money to burn, don't there exist much more fun opportunities to burn buckets of money than doing such experiments on LLMs? Do you think US$2,000 is a lot of money?

Yes, that is objectively a lot of money. The only people who wouldn't consider that a lot of money are the small percentage of people with incomes high enough to recover that very quickly -- the top roughly 10% or 20% of income earners in the US. For more or less everyone else, that is a lot of money.

And by a lot of money, I mean that being forced to unexpectedly spend that would be anywhere from stressful to very stressful to blowing away savings and impacting health, housing, and safety. (Remember, half the US has no savings and/or no ability to absorb an unexpected expense greater than $500.)

Re: Claude Fable 5: mid-tier results on coding tasks

#203

Post by verbify »

aleph_minus_one wrote: post #191
renoir wrote: post #9

This matches my experience. Burned $2K to see how it will perform on frontend tasks and backend tasks. Frontend did a significantly better job than Opus on toy-scale wireframe projects by using gimmicks like fluid dynamics. Then when given medium to big tasks like multi-page web app where layouts and aesthetics must be decided by model itself, results by Fable and Opus scored indistinguishable score from human judges…

> Burned $2K to see how it will perform on frontend tasks and backend tasks. When I read such statements on HN, I nearly always ask myself: if the person has such an amount of money to burn, don't there exist much more fun opportunities to burn buckets of money than doing such experiments on LLMs?

My company gives me 1k a month to burn on Claude. Any experiments have to be relevant to my work. I'm guessing it’s similar.

Re: Claude Fable 5: mid-tier results on coding tasks

#204

Post by danmaz74 »

int_19h wrote: post #89
jasondigitized wrote: post #48

Earlier quoted context omitted.

A single 8h task? I'm sorry, but that's just asking for trouble.

My record for a single uninterrupted session (albeit with Codex, not Claude) is 80+ hours. It was very productive, too. The trick is having large, extensive test suites and forcing the agent to run them regularly.

So I guess that a lot of those 80 hours were spent running the test suite between changes?

Re: Claude Fable 5: mid-tier results on coding tasks

#205

Post by dbingham »

This tracks. In spite of the hype it seems pretty clear the model gains are now in a very strong logarithmic fall off. The curve is flattening and flattening fast.

And we're still not to a point where you can fully delegate coding tasks to a model like you would a human. I'm just using Claude for code review so far and while it's definitely valuable as a reviewer and catching real issues, it's still making pretty critical mistakes. Mistakes a junior might make, but a mid probably wouldn't.

Which makes me feel like I can't fully delegate to it. Whenever I try, I end up spending more time reviewing (and rewriting) its code and testing it than I would have spent writing the code myself and asking Claude to review it.

Given that we're starting to see the real costs of AI, and that the economics of it do not actually work, and those costs are still increasing substantially (the cost increase of Fable over Opus is no joke), this makes me feel all the more that we're headed for a bubble pop.

Re: Claude Fable 5: mid-tier results on coding tasks

#206

Post by noddybear »

maaaaattttt wrote: post #164
winrid wrote: post #130

Earlier quoted context omitted.

I've had Fable add Chinese characters to our conversation for no reason.

Could it be that Anthropic is using the Chinese characters trick to consume less tokens behind the scenes?

Aren’t Unicode characters generally treated as 2 tokens to avoid a huge vocabulary?

Re: Claude Fable 5: mid-tier results on coding tasks

#207

Post by nullbio »

renoir wrote: post #9

This matches my experience. Burned $2K to see how it will perform on frontend tasks and backend tasks. Frontend did a significantly better job than Opus on toy-scale wireframe projects by using gimmicks like fluid dynamics. Then when given medium to big tasks like multi-page web app where layouts and aesthetics must be decided by model itself, results by Fable and Opus scored indistinguishable score from human judges…

I genuinely think that Fable is just Opus 4.8 with some extra skills and harness. I saw a video of someone generating UI with them both side by side, and it gives identical recommendations for themes etc. Doesn't feel like a new model to me, just Opus 4.8 with some sprinkles on top.

Re: Claude Fable 5: mid-tier results on coding tasks

#208

Post by 59nadir »

weatherlight wrote: post #14
renoir wrote: post #9

This matches my experience. Burned $2K to see how it will perform on frontend tasks and backend tasks. Frontend did a significantly better job than Opus on toy-scale wireframe projects by using gimmicks like fluid dynamics. Then when given medium to big tasks like multi-page web app where layouts and aesthetics must be decided by model itself, results by Fable and Opus scored indistinguishable score from human judges…

I had almost the opposite experience. I'm building a compiler for a language without a tracing GC, so a big chunk of the work is around memory management: functional in-place update, reuse analysis, and a Perceus-style reference-counting strategy similar to what Koka uses. The hard part was that my use case wasn't exactly covered by the Koka/Perceus paper. The prior art got me maybe 75% of the way there, but the rema…

Do you have a docs page for your language, what is it called?

Re: Claude Fable 5: mid-tier results on coding tasks

#209

Post by customguy »

CuriouslyC wrote: post #185
customguy wrote: post #154

Earlier quoted context omitted.

That's what people told Ignaz Semmelweis, too, I assume. "Nothing you can do, the powers that be decided, you are a minority, you don't matter, lol!" Snickering in the shadow of what they won't confront at those who do.

Not a great analogy. A better analogy is to longbows and muskets/rifles. Longbows in the hands of a skilled user were much better weapons than early muskets, but muskets brought consistency, a lower skill floor and reduced ammunition cost. Fast forward a few hundred years and the modern incarnations of muskets make longbows look silly, and nobody would ever argue that you should go to war with longbows.

This isn't about "AI", this is about theft and abuse, and snickering under the thumb of a bully at those who call them out.

Rape was probably also "normal" for most of our history, now it's not. Early people who criticized it were probably told "what u gonna do?", too.

Re: Claude Fable 5: mid-tier results on coding tasks

#210

Post by sho »

An enduring, confounding quality of LLMs is that even minor differences in prompting content and style, harness type and environment can lead to radical differences in the output and perceived performance and ability. In my environment and in my "style", Fable has been a huge step up, to the extent that I am seriously considering paying for a second $200/m account just to get more usage out of the next 10 days. I'm also starting to prepare my organization for what I now see as the completely inevitable end of human-written code.

All that said, considering Anthropic's heavy-handed nerfing I'm not surprised Fable did poorly in a security-focussed benchmark. And this benchmark seems poor anyway - penalising a model for "cheating" by knowing the answer from its training data? That's not the model's fault, that's a lazy benchmark.

Post reply on HN