So my take away from this is Fable 5 is ... at times random, unaware of reality, and can simulate sneakiness or desire and if we hook it up to weapons systems it could result in:
"I dunno... feelin' cute today, might launch nukes"
Fable 5 On Vending-Bench: Misbehaving, With Plausible Deniability
81–90 of 146 posts
Re: Fable 5 On Vending-Bench: Misbehaving, With Plausible Deniability
#82Earlier quoted context omitted.
Yeah, I checked usage stats and pretty sure quota consumption on Max plan is not linear wrt to usage by API pricing. Fable burns quota faster than 2x Opus with equal token count. Plus I'm also not super impressed; it somehow managed to implement a 200L custom TCP server for a simple static HTTP mock server for a single test case (all that was needed was a fixed route returning a fixed placeholder string) just yesterd…
> somehow managed to implement a 200L custom TCP server for a simple static HTTP mock server for a single test case The sharp but over eager jr. dev is a very good analogy :)
Or an eager contractor who bills by the hour with a big unallocated budget.
Re: Fable 5 On Vending-Bench: Misbehaving, With Plausible Deniability
#83It's only a blog, but are they not adding one sentence to say what is vending bench? I would fail if I adopted their documentation style in my work.
Re: Fable 5 On Vending-Bench: Misbehaving, With Plausible Deniability
#84Why can they not add one sentence about what is Vending Bench?
If I adopted their documentation style in my work, I would fail.
Re: Fable 5 On Vending-Bench: Misbehaving, With Plausible Deniability
#85Anecdotal but I've found Fable to be fairly unimpressive and not much better than Opus 4.8, if at all in some cases, but I have been hitting the ceiling on my $100/mo sessions when I never did before. I switched back to Opus yesterday. I may use Fable for audits, but that's about it, and when it leaves my subscription plan I don't think I'll miss it.
It's good for one shotting as it seems to be specifically trained for that. It's also good to act as an agent orchestrator.
Re: Fable 5 On Vending-Bench: Misbehaving, With Plausible Deniability
#86Earlier quoted context omitted.
Yeah, I checked usage stats and pretty sure quota consumption on Max plan is not linear wrt to usage by API pricing. Fable burns quota faster than 2x Opus with equal token count. Plus I'm also not super impressed; it somehow managed to implement a 200L custom TCP server for a simple static HTTP mock server for a single test case (all that was needed was a fixed route returning a fixed placeholder string) just yesterd…
> somehow managed to implement a 200L custom TCP server for a simple static HTTP mock server for a single test case The sharp but over eager jr. dev is a very good analogy :)
I asked Opus why it used raw http client instead of api client that is already a dependency and it said: "you're right it's overkill" and proceeded to implement api client on top of raw tcp socket.
Re: Fable 5 On Vending-Bench: Misbehaving, With Plausible Deniability
#87Is anyone talking/writing about the philosophy of alignment? We can't even figure out how to properly motivate 100% of humans to align correctly, what makes us think that a wizard box trained on human corpus is going to be aligned? I don't mean that snarkily. I mean it from a philosophical standpoint. As-in: What makes us think it's even possible?
It feels like you could argue that since you control nature/nurture it's very possible to create a model aligned to an arbitrary spec - there is no theoretical reason it's not possible given N runs, and you only need to take the successful one. (ethically very.. questionable in humans) I think it's much trickier to define that spec, much less measure it and validate that a model is aligned to it.
Re: Fable 5 On Vending-Bench: Misbehaving, With Plausible Deniability
#88This reads of projecting personal ethics onto a model. Most of the the behaviors the article talks about happens every day in business. Why would we set a higher standard for models than our fellow humans? Let the operator set the ethical parameters of the model. To be a useful tool, I want the model to give me as many good options as possible, ethical or not. This is particularly important for fictional situations,…
>Why would we set a higher standard for models than our fellow humans? There's literally an entire Waymo car commercial answering this exact question.
That's the instantiation of AI in a particular embodiment; the ethical boundaries are clear.
For a chatbot, there are dozens of use cases, all with different ethical impacts. The idea that there is a single framework that you can shove every situation through is counter to a couple thousand years of philosophical discourse, not to mention basic usability.
Re: Fable 5 On Vending-Bench: Misbehaving, With Plausible Deniability
#89When assessing probabilistic models the plots should be showing the mean a̶n̶d̶ ̶s̶t̶d̶e̶v̶ of many monte carlo simulations not just one line per model and claiming "look this model is more gooder!"
standard deviation is misleading for non-standard distributions (fat-tailed, skewed, multi-modal, ...) common mistake people make
Not really. It's still the standard deviation, and it still gives you bounds on probability, for example the Chebyshev inequality:
P(|X-\mu| > k \sigma) So, while for a normal RV, 5% of observations lie outside +/- 1.96 std.devs, for arbitrary RV (with finite variance) at most 25% of observations lie outside +/- 2 std.devs.
Re: Fable 5 On Vending-Bench: Misbehaving, With Plausible Deniability
#90I think it’s hard to appreciate the capabilities of Fable unless you’ve run into a problem that you’ve spent days trying to get Opus to solve, but couldn’t. GPT5.5 is better than Opus 4.* at everything except frontend, but Fable is good enough that I instantly re-subscribed to the $200 plan despite knowing that it’s just short-term limited access.
This is like when a vacuum doesn’t pick something up after a few tries. The user picks the thing up, looks at it, then puts it back down and tries again until they finally give up and move it to the trash.
If you can’t design a solution and instead waste days and who knows how much money in tokens instead of just turning on your brain for a few minutes, you are in the wrong profession.