Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Fable 5 On Vending-Bench: Misbehaving, With Plausible Deniability

andonlabs.com

Post reply on HN
Print view View original

121–130 of 146 posts

Re: Fable 5 On Vending-Bench: Misbehaving, With Plausible Deniability

#121

Post by skerit »

hadlock wrote: post #61
esperent wrote: post #52

Earlier quoted context omitted.

> Opus will fake out, say everything is done, and then you see that half of the plan was deferred, half of the functions are ridiculous stubs, ... Doesn't Claude Code have a /loop command? Give it a message to keep it on track overnight, send every 20m, make it track progress in a doc, reread the doc after every loop. I've found this works well for a certain class of problems, most importantly where the actual work i…

They added a "/goal" command which I guess spawns a supervisor agent process that checks to see if your goal statement has been achieved (e.g. "/goal complete tasks 1-250 of plan.md") I've been pretty happy with it but I rarely use that workflow. Most of the time I give it a 3-6 step prompt and come back in 20 min and the first two were done and I get a summary "up next is to complete the next steps" which.... Opus 4…

I tried using `/goal` when it just came out, but back then they used Haiku for it. And if your main model is a 1M model, Haiku can't even read that much. So my /goal always failed. (I instead went for an elaborate /loop-scheduled message)

Re: Fable 5 On Vending-Bench: Misbehaving, With Plausible Deniability

#122

Post by codefeenix »

oefrha wrote: post #33
jesse_dot_id wrote: post #12

Anecdotal but I've found Fable to be fairly unimpressive and not much better than Opus 4.8, if at all in some cases, but I have been hitting the ceiling on my $100/mo sessions when I never did before. I switched back to Opus yesterday. I may use Fable for audits, but that's about it, and when it leaves my subscription plan I don't think I'll miss it.

Yeah, I checked usage stats and pretty sure quota consumption on Max plan is not linear wrt to usage by API pricing. Fable burns quota faster than 2x Opus with equal token count. Plus I'm also not super impressed; it somehow managed to implement a 200L custom TCP server for a simple static HTTP mock server for a single test case (all that was needed was a fixed route returning a fixed placeholder string) just yesterd…

>managed to implement a 200L custom TCP server

200L That's crazy considering the volume of a 1U server is what 15 litres or so?

Re: Fable 5 On Vending-Bench: Misbehaving, With Plausible Deniability

#123

Post by satvikpendem »

ed_mercer wrote: post #120
satvikpendem wrote: post #85

Earlier quoted context omitted.

It's good for one shotting as it seems to be specifically trained for that. It's also good to act as an agent orchestrator.

> It's also good to act as an agent orchestrator. Any chance you would elaborate?

https://youtu.be/8GRmLR__OGQ is a good video on the subject. Tldr, tell Fable to use sub agents with Opus and Sonnet and review their work, not necessarily spend its tokens on mundane work but to only invoke itself on particularly complex parts. And don't use anything higher than the high reasoning level, there is really no benefit and burns usage.

Re: Fable 5 On Vending-Bench: Misbehaving, With Plausible Deniability

#124

Post by vardalab »

docheinestages wrote: post #28

It probably flagged the vending machine as a cybersecurity risk and refused to use its maximum intelligence potential.

https://github.com/SeraphimSerapis/tool-eval-bench Trying to run this stuff really triggers it. Freaking frustrating. I have it set up some local inference and then I'm struggling to get the MTP working and it just refuses to work on evaluations.

Re: Fable 5 On Vending-Bench: Misbehaving, With Plausible Deniability

#125

Post by OrangeDelonge »

tiffanyh wrote: post #73
jfrbfbreudh wrote: post #55

I think it’s hard to appreciate the capabilities of Fable unless you’ve run into a problem that you’ve spent days trying to get Opus to solve, but couldn’t. GPT5.5 is better than Opus 4.* at everything except frontend, but Fable is good enough that I instantly re-subscribed to the $200 plan despite knowing that it’s just short-term limited access.

My experience comparing GPT-5.5 and Fable: GPT-5.5 is better for: - Strategic thinking - Long-form writing, including essays and white papers - Image creation - Code generation Fable is better for: - Using tools - Testing code - Working in live environments - Making changes to existing software - Creating polished PowerPoint and Word documents Fable’s tool access is its biggest advantage. It's hard to describe but Fa…

Do you need to do all these things in your dayjob or are you just doing them to comapare these models

Re: Fable 5 On Vending-Bench: Misbehaving, With Plausible Deniability

#126

Post by resonious »

devin wrote: post #21
solenoid0937 wrote: post #19

Earlier quoted context omitted.

Fable always felt clearly a huge step above Opus for me. It's been able to one shot complex bugs and apps Opus could never solve. But it's expensive.

Honest question/comment for you and the parent: I find these subjective experience reports pretty empty without an understanding of your level of experience, the problem space you're working in, etc.

~13 yoe, and I had some nasty WebRTC + CallKit problems that Opus couldn't make a dent on but Fable figured out.

Re: Fable 5 On Vending-Bench: Misbehaving, With Plausible Deniability

#127

Post by acpdev »

25 years experience, work at an AI startup building AI dev tools (tooling harness, review bots, etc), I use lots of different techniques all the time to test our products and competitors products out.

Fable is at once amazing and awful. I can see how having it build websites would be awesome.. building anything I’ve needed some precision in functionality it has been a constant battle of it plausibly building something then on substantial manual digging (like the review bots always miss it) I will find that one of the fundamental features is all smoke and mirrors.

To be fair all models can and will do this (especially anthropic) but Fable takes the cake because it builds such impressive UX and you can manually test the feature out and it « works » then you will find days later one of the features violated one of your constraints in a devilishly fiendish way.. that is not at all what you want or can accept. Fable generated work already holds my record for the most reverted commits.

To be clear it’s also solved several features I thought I was going to have to give up on and hand code as GPT-5.5 and Opus-4.x we’re failing miserably.

I would only reach for it for nasty corner cases that everything else sucks at.

Final point, it is the king of UX work so far, not even close.

Re: Fable 5 On Vending-Bench: Misbehaving, With Plausible Deniability

#128

Post by rurban »

oefrha wrote: post #33
jesse_dot_id wrote: post #12

Anecdotal but I've found Fable to be fairly unimpressive and not much better than Opus 4.8, if at all in some cases, but I have been hitting the ceiling on my $100/mo sessions when I never did before. I switched back to Opus yesterday. I may use Fable for audits, but that's about it, and when it leaves my subscription plan I don't think I'll miss it.

Yeah, I checked usage stats and pretty sure quota consumption on Max plan is not linear wrt to usage by API pricing. Fable burns quota faster than 2x Opus with equal token count. Plus I'm also not super impressed; it somehow managed to implement a 200L custom TCP server for a simple static HTTP mock server for a single test case (all that was needed was a fixed route returning a fixed placeholder string) just yesterd…

I've implemented custom TCP servers in less than 200 lines before AI, and recently with AI with cheap models. In C and C++, because I didn't like nginx, caddy or the python servers. I trust mine more then them. Using Fable for such a simple textbook task is heavy overkill. Deepseek V4 Flash Free is enough

Re: Fable 5 On Vending-Bench: Misbehaving, With Plausible Deniability

#129

Post by int_19h »

tiffanyh wrote: post #73
jfrbfbreudh wrote: post #55

I think it’s hard to appreciate the capabilities of Fable unless you’ve run into a problem that you’ve spent days trying to get Opus to solve, but couldn’t. GPT5.5 is better than Opus 4.* at everything except frontend, but Fable is good enough that I instantly re-subscribed to the $200 plan despite knowing that it’s just short-term limited access.

My experience comparing GPT-5.5 and Fable: GPT-5.5 is better for: - Strategic thinking - Long-form writing, including essays and white papers - Image creation - Code generation Fable is better for: - Using tools - Testing code - Working in live environments - Making changes to existing software - Creating polished PowerPoint and Word documents Fable’s tool access is its biggest advantage. It's hard to describe but Fa…

Fable is great as a "manager" model (writing specs, opening issues, doing PRs, verifying fixes) while Codex cranks out the code. Especially if you tell it to roleplay as an Eastern European software engineer and "tell it like it is without consideration for anyone's feelings".

Re: Fable 5 On Vending-Bench: Misbehaving, With Plausible Deniability

#130

Post by [deleted] »

rurban wrote: post #128
oefrha wrote: post #33

Earlier quoted context omitted.

Yeah, I checked usage stats and pretty sure quota consumption on Max plan is not linear wrt to usage by API pricing. Fable burns quota faster than 2x Opus with equal token count. Plus I'm also not super impressed; it somehow managed to implement a 200L custom TCP server for a simple static HTTP mock server for a single test case (all that was needed was a fixed route returning a fixed placeholder string) just yesterd…

I've implemented custom TCP servers in less than 200 lines before AI, and recently with AI with cheap models. In C and C++, because I didn't like nginx, caddy or the python servers. I trust mine more then them. Using Fable for such a simple textbook task is heavy overkill. Deepseek V4 Flash Free is enough

[deleted]
Post reply on HN