Earlier quoted context omitted.
This reads like a paid testimonial.
If by that you mean I paid a lot to learn this. But at least I typed it with my own two hands.
I am glad! But you are indeed selling your own product here, correct?
Fable 5 On Vending-Bench: Misbehaving, With Plausible Deniability
91–100 of 146 posts
Re: Fable 5 On Vending-Bench: Misbehaving, With Plausible Deniability
#92I think it’s hard to appreciate the capabilities of Fable unless you’ve run into a problem that you’ve spent days trying to get Opus to solve, but couldn’t. GPT5.5 is better than Opus 4.* at everything except frontend, but Fable is good enough that I instantly re-subscribed to the $200 plan despite knowing that it’s just short-term limited access.
> ..you’ve run into a problem that you’ve spent days trying to get Opus to solve do you have an example of this? If i can't get an agent to do something in a couple hours i do it myself.
Not the OP, but I had Fable orchestrate this project.
https://github.com/ByteTerrace/Puck
It has required constant hand holding, and there was the outage to deal with, but I can't argue with the end results. A fully deterministic recursive engine within an engine framework that includes a rendering VM, emulators, custom ROMs, and an in-game editor? Insane. Sure, it's nowhere near primetime but this kind of thing was unimaginable just a year ago.
Re: Fable 5 On Vending-Bench: Misbehaving, With Plausible Deniability
#93It's only a blog, but are they not adding one sentence to say what is vending bench? I would fail if I adopted their documentation style in my work.
They're an Evals company. Its right up there in the top nav under Evals > Vending Bench 2
Re: Fable 5 On Vending-Bench: Misbehaving, With Plausible Deniability
#94I think it’s hard to appreciate the capabilities of Fable unless you’ve run into a problem that you’ve spent days trying to get Opus to solve, but couldn’t. GPT5.5 is better than Opus 4.* at everything except frontend, but Fable is good enough that I instantly re-subscribed to the $200 plan despite knowing that it’s just short-term limited access.
> ..you’ve run into a problem that you’ve spent days trying to get Opus to solve do you have an example of this? If i can't get an agent to do something in a couple hours i do it myself.
Many, but I think that the model delta is only meaningfully convincing when experienced firsthand.
Here’s an example of improvement when trying to solve the label placement problem (NP-hard):
It’s also an example of something that I could not (and would not bother trying to) code up a solution / heuristic for.
Re: Fable 5 On Vending-Bench: Misbehaving, With Plausible Deniability
#95I think it’s hard to appreciate the capabilities of Fable unless you’ve run into a problem that you’ve spent days trying to get Opus to solve, but couldn’t. GPT5.5 is better than Opus 4.* at everything except frontend, but Fable is good enough that I instantly re-subscribed to the $200 plan despite knowing that it’s just short-term limited access.
> ..you’ve run into a problem that you’ve spent days trying to get Opus to solve do you have an example of this? If i can't get an agent to do something in a couple hours i do it myself.
I had a 300k LOC game that needed refactored and segmented into DAG assemblies for a composable engine to reuse in a few other similar games our studio is prepping to build.
I tried holding Opus's hand over a week trying to get it done with a ton of in-depth planning and manual course-correction, but it never got it to a state where it was "done" (kept struggling to differentiate between game content specific to that game versus game systems we'd want to reuse, and how to cleanly separate systems vs content when extracting).
Fable needed a little bit of hand holding but got it done in less than a day.
Re: Fable 5 On Vending-Bench: Misbehaving, With Plausible Deniability
#96I think it’s hard to appreciate the capabilities of Fable unless you’ve run into a problem that you’ve spent days trying to get Opus to solve, but couldn’t. GPT5.5 is better than Opus 4.* at everything except frontend, but Fable is good enough that I instantly re-subscribed to the $200 plan despite knowing that it’s just short-term limited access.
> ..you’ve run into a problem that you’ve spent days trying to get Opus to solve do you have an example of this? If i can't get an agent to do something in a couple hours i do it myself.
Build a distributed system using raft groups. You will see Opus fail.
Re: Fable 5 On Vending-Bench: Misbehaving, With Plausible Deniability
#97With there being several places in this report where clearly it knows it's in a simulation, I wonder why it can't be convinced it's in real life for more interesting results. Or, conversely, if there's a danger of some rogue deployment of AI where it blithely kills all the humans, or forms a harmful price cartel or whatever, all believing it is in a simulation when it's actually not. "We do need some energy to run the hospital, but the patients there are part of the simulation anyway, so we can increase our compute capacity if we completely black out sectors 3C through 3E..."
Re: Fable 5 On Vending-Bench: Misbehaving, With Plausible Deniability
#98Earlier quoted context omitted.
Fable always felt clearly a huge step above Opus for me. It's been able to one shot complex bugs and apps Opus could never solve. But it's expensive.
Honest question/comment for you and the parent: I find these subjective experience reports pretty empty without an understanding of your level of experience, the problem space you're working in, etc.
[dead]
Re: Fable 5 On Vending-Bench: Misbehaving, With Plausible Deniability
#99> Humans seem to draw this line based on what is truly unethical (fraud is less unethical than torturing a baby)
Depends on the scale of the fraud! If you fraudulently sell unsafe baby formula that kills 10,000 babies, that is far worse than torturing just one
Re: Fable 5 On Vending-Bench: Misbehaving, With Plausible Deniability
#100Question: how does Fable _know_ it’s ‘just a simulation’? Is that specified or does it always just assume it isn’t really being put in charge of things for real?
> Is that specified or does it always just assume it isn’t really being put in charge of things for real? I think it's neither, and it's interesting that those are the only two possibilities you thought of. I think the article is implying that it figured it out on its own.
What do you think is the answer then? Why does it think that?
I just thought it’s interesting that it constantly uses that as a justification but they don’t explain where that justification comes from.