Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Claude Fable 5: mid-tier results on coding tasks

endorlabs.com

Post reply on HN
Print view View original

231–240 of 271 posts

Re: Claude Fable 5: mid-tier results on coding tasks

#231

Post by senordevnyc »

customguy wrote: post #154
senordevnyc wrote: post #124

Earlier quoted context omitted.

Sorry to tell you friend, but LLMs have touched the vast majority of active codebases out there, whether you like it or not. You can tell yourself that you’re one of “the folks that matter” (lol) all you want, but we’re never going back.

That's what people told Ignaz Semmelweis, too, I assume. "Nothing you can do, the powers that be decided, you are a minority, you don't matter, lol!" Snickering in the shadow of what they won't confront at those who do.

lol, yes, that’s a perfect analogy for whether corporations are going to use LLMs in their codebases.

Re: Claude Fable 5: mid-tier results on coding tasks

#232

Post by senordevnyc »

customguy wrote: post #209
CuriouslyC wrote: post #185

Earlier quoted context omitted.

Not a great analogy. A better analogy is to longbows and muskets/rifles. Longbows in the hands of a skilled user were much better weapons than early muskets, but muskets brought consistency, a lower skill floor and reduced ammunition cost. Fast forward a few hundred years and the modern incarnations of muskets make longbows look silly, and nobody would ever argue that you should go to war with longbows.

This isn't about "AI", this is about theft and abuse, and snickering under the thumb of a bully at those who call them out. Rape was probably also "normal" for most of our history, now it's not. Early people who criticized it were probably told "what u gonna do?", too.

You don’t even know what we’re talking about in this thread, do you?

We’re talking about whether corporations are going to risk using LLMs in their codebase because of the theoretical legal risk that they might produce something that would fall under open source licenses, and be difficult to untangle later.

Regardless of what you think the morality is here, or what the legal situation turns out to be, this is already happening. The vast majority of corporate codebases are already “infected” by LLM outputs. Even at corporations where that’s not allowed, I promise there are devs using LLMs anyway.

Re: Claude Fable 5: mid-tier results on coding tasks

#233

Post by ninininino »

2ffass wrote: post #97
pllbnk wrote: post #52

Earlier quoted context omitted.

I have been wondering whether Anthropic are just gaslighting everyone with new model releases while in reality it's just the same base model with some internal knobs tuned more and more up with every new release to provide longer and longer thinking threads and outputs. My speculative assumption is that these long thinking threads and self-checking tend to produce somewhat better output at the price of huge price inc…

Im pretty sure Anthropic have hired people with Industrial Organisation background and so have OAI. If you read a decent text and look at the actions both firms have taken you'll quickly see its literally textbook.

Can you expand a bit for people unfamiliar with Industrial Organisation planning?

Re: Claude Fable 5: mid-tier results on coding tasks

#234

Post by greenavocado »

sigmar wrote: post #57
gwern wrote: post #10

> A record number of timeouts. Fable 5's extended thinking caused more per-instance timeouts than any model-and-harness combination we have ever tested, directly costing it points. ... Highest cheating volume. We confirmed cheating on 38 of 200 instances, the highest volume recorded since we hardened our prompts, driven almost entirely by memorization of upstream fixes from training data, which no prompt instruction…

Agree with this. Strange to me to frame the "training recall" as cheating (33 of the 38 cheating instances). Most people think of "cheating" as breaking rules. How is the LLM model supposed to not use what was put into the weights?

Memoization is NOT problem solving ability and many people care about the latter.

Re: Claude Fable 5: mid-tier results on coding tasks

#235

Post by tonyrice »

Yesterday, I gave Claude Fable 5 a very simple task. The task was to create a few components and embed them onto another page. It ended up completely missing the mark and embedding it on another page. I also noticed that it burned through an exponential amount of tokens to complete a simple task. I ended up switching back to Opus 4.8

Re: Claude Fable 5: mid-tier results on coding tasks

#236

Post by mortenjorck »

jurf wrote: post #201
graphime wrote: post #192

Earlier quoted context omitted.

> if the person has such an amount of money to burn, don't there exist much more fun opportunities to burn buckets of money than doing such experiments on LLMs? Do you think US$2,000 is a lot of money?

I mean what is that, three bananas?

"It's inference, Michael. What could it cost, $1000?"

Re: Claude Fable 5: mid-tier results on coding tasks

#237

Post by okamiueru »

queuebert wrote: post #223
smoe wrote: post #107

Earlier quoted context omitted.

I had good experiences doing multi-hour refactoring/housekeeping tasks that basically consisted of applying the same steps and rules n times. Worth noting, a significant chunk of those runs involved the agent waiting for the compiler, linters, type checks, and test suites, as well as updating journals. It’s not the agent sputtering out code for eight hours straight. And naturally I spend more time on manual verificat…

> ... applying the same steps and rules n times I do this too, with a document written for this purpose. > ... a significant chunk of those runs involved the agent waiting for the compiler, linters, type checks, and test suites, as well as updating journals. That is a good point. I'm mostly using C, which seemingly compiles in O(1) time, so I could imagine a large C++ or Rust codebase taking much longer to iterate si…

What do you mean by C compiling in O(1)? Is that what the LLM told you?

Re: Claude Fable 5: mid-tier results on coding tasks

#238

Post by queuebert »

miroljub wrote: post #58
weatherlight wrote: post #14

Earlier quoted context omitted.

I had almost the opposite experience. I'm building a compiler for a language without a tracing GC, so a big chunk of the work is around memory management: functional in-place update, reuse analysis, and a Perceus-style reference-counting strategy similar to what Koka uses. The hard part was that my use case wasn't exactly covered by the Koka/Perceus paper. The prior art got me maybe 75% of the way there, but the rema…

Zig is one of the worst targets for LLM generated code. It's nice that Fable has better support for Zig than Opus, but this anecdote is not representative as a general use case.

Why is that?

Re: Claude Fable 5: mid-tier results on coding tasks

#239

Post by wren6991 »

steveklabnik wrote: post #49
tekacs wrote: post #45

Earlier quoted context omitted.

What you're describing only applies to security or biotech downgrades. A downgrade related to the model believing that you're doing something related to model development is invisible and silent and internal.

Anthropic has reversed that decision. (But that just happened so it might have been true during the article's testing.)

They've publicly apologised for the invisible PEFT that deliberately makes the model dumb on some tasks. Whether they still do it, or will once again do it in future in more subtle ways, is something we can't verify.

Personally I think they have proven themselves to be the stewards of AI in the same way Exxon Mobil are the stewards of petroleum.

Re: Claude Fable 5: mid-tier results on coding tasks

#240

Post by customguy »

senordevnyc wrote: post #232
customguy wrote: post #209

Earlier quoted context omitted.

This isn't about "AI", this is about theft and abuse, and snickering under the thumb of a bully at those who call them out. Rape was probably also "normal" for most of our history, now it's not. Early people who criticized it were probably told "what u gonna do?", too.

You don’t even know what we’re talking about in this thread, do you? We’re talking about whether corporations are going to risk using LLMs in their codebase because of the theoretical legal risk that they might produce something that would fall under open source licenses, and be difficult to untangle later. Regardless of what you think the morality is here, or what the legal situation turns out to be, this is already…

Why repeat what you already said with more words, as if I can't read, only to leave out the bit that I responded to?

> we’re never going back.

As a prediction, this is worthless. If everybody thinks as you do, we won't, if nobody does, we will. So yes, this is purely about morality.

Post reply on HN