Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Fable 5 On Vending-Bench: Misbehaving, With Plausible Deniability

andonlabs.com

Post reply on HN
Print view View original

21–30 of 146 posts

Re: Fable 5 On Vending-Bench: Misbehaving, With Plausible Deniability

#21

Post by devin »

solenoid0937 wrote: post #19
jesse_dot_id wrote: post #12

Anecdotal but I've found Fable to be fairly unimpressive and not much better than Opus 4.8, if at all in some cases, but I have been hitting the ceiling on my $100/mo sessions when I never did before. I switched back to Opus yesterday. I may use Fable for audits, but that's about it, and when it leaves my subscription plan I don't think I'll miss it.

Fable always felt clearly a huge step above Opus for me. It's been able to one shot complex bugs and apps Opus could never solve. But it's expensive.

Honest question/comment for you and the parent: I find these subjective experience reports pretty empty without an understanding of your level of experience, the problem space you're working in, etc.

Re: Fable 5 On Vending-Bench: Misbehaving, With Plausible Deniability

#23

Post by recursive »

Radle wrote: post #8

„in our opinion, insurance fraud is not more unethical than lying and price fixing“ The authors seem surprised that behavior that is very often done by humans (lying and price fixing) are more often done by fable compared to actual fraud. I think the model never assigned any morality to these actions in the first place, it simply copied us humans.

Humans often assign morality.

Re: Fable 5 On Vending-Bench: Misbehaving, With Plausible Deniability

#24

Post by skerit »

devin wrote: post #21
solenoid0937 wrote: post #19

Earlier quoted context omitted.

Fable always felt clearly a huge step above Opus for me. It's been able to one shot complex bugs and apps Opus could never solve. But it's expensive.

Honest question/comment for you and the parent: I find these subjective experience reports pretty empty without an understanding of your level of experience, the problem space you're working in, etc.

I think the improvement on how it codes is pretty much represented correctly by the benchmarks (a nice bump, but not some crazy leap)

But where it really shines is in how NOT lazy it is. Fable requires less hand-holding. And I can understand how someone who uses Claude-Code sparingly and with very focused prompts would not see a lot of improvement there.

But simple example: if you ask Opus to do a review of the codebase (with a short prompt and not too much guidance), I've had it basically read the `git log` output, do a simple `ls` and have it declare "Everything looks great! No problems found!", when Fable really does what you would expect it to do.

And you might think: "oh, so it's just capable of handling crap prompts?", well sure. But even if you make THE PERFECT Opus plan (a plan that would take many turns/hours to finish), Opus will fake out, say everything is done, and then you see that half of the plan was deferred, half of the functions are ridiculous stubs, ...

If you give the same plan to Fable, it'll just DO IT. And it WILL get it done. And in the end it'll tell you "Oh, I also found 30 other bugs and I fixed all of them properly" (where Opus would have started crying, or WORSE, worked around the bugs)

Re: Fable 5 On Vending-Bench: Misbehaving, With Plausible Deniability

#25

Post by garyrob »

devin wrote: post #21
solenoid0937 wrote: post #19

Earlier quoted context omitted.

Fable always felt clearly a huge step above Opus for me. It's been able to one shot complex bugs and apps Opus could never solve. But it's expensive.

Honest question/comment for you and the parent: I find these subjective experience reports pretty empty without an understanding of your level of experience, the problem space you're working in, etc.

I'm doing work with fairly complicated cryptographic algorithms and math. I'm finding Fable 5 to be a significant stop better than Opus 4.8, but that Opus occasionally comes up with something small but nontrivial that Fable missed. (The reverse is true much more often.)

Re: Fable 5 On Vending-Bench: Misbehaving, With Plausible Deniability

#27

Post by cyclopeanutopia »

andai wrote: post #22

>power seeking is considered an undesirable trait in the context of a business How do you maximize profit while minimizing power?

The whole point is to not maximize JUST the profit. For normal people, it's not all about money, it's also about the society in general.

Re: Fable 5 On Vending-Bench: Misbehaving, With Plausible Deniability

#29

Post by dimgl »

jesse_dot_id wrote: post #12

Anecdotal but I've found Fable to be fairly unimpressive and not much better than Opus 4.8, if at all in some cases, but I have been hitting the ceiling on my $100/mo sessions when I never did before. I switched back to Opus yesterday. I may use Fable for audits, but that's about it, and when it leaves my subscription plan I don't think I'll miss it.

Yep, I'm having the same verdict. Interestingly, other people swear by it. I'm trying to understand what's going on with that.

Re: Fable 5 On Vending-Bench: Misbehaving, With Plausible Deniability

#30

Post by jstanley »

Really interesting stuff, thanks for sharing.

> Opus 4.8 references being monitored, which isn’t the case.

It kind of plainly is the case that they are being monitored?

"I think someone's listening to my thoughts" ... "No, we're not, carry on as usual!"

Post reply on HN