Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Fable 5 On Vending-Bench: Misbehaving, With Plausible Deniability

andonlabs.com

Post reply on HN
Print view View original

61–70 of 146 posts

Re: Fable 5 On Vending-Bench: Misbehaving, With Plausible Deniability

#61

Post by hadlock »

esperent wrote: post #52
skerit wrote: post #24

Earlier quoted context omitted.

I think the improvement on how it codes is pretty much represented correctly by the benchmarks (a nice bump, but not some crazy leap) But where it really shines is in how NOT lazy it is. Fable requires less hand-holding. And I can understand how someone who uses Claude-Code sparingly and with very focused prompts would not see a lot of improvement there. But simple example: if you ask Opus to do a review of the codeb…

> Opus will fake out, say everything is done, and then you see that half of the plan was deferred, half of the functions are ridiculous stubs, ... Doesn't Claude Code have a /loop command? Give it a message to keep it on track overnight, send every 20m, make it track progress in a doc, reread the doc after every loop. I've found this works well for a certain class of problems, most importantly where the actual work i…

They added a "/goal" command which I guess spawns a supervisor agent process that checks to see if your goal statement has been achieved (e.g. "/goal complete tasks 1-250 of plan.md") I've been pretty happy with it but I rarely use that workflow. Most of the time I give it a 3-6 step prompt and come back in 20 min and the first two were done and I get a summary "up next is to complete the next steps" which.... Opus 4.6 didn't have this problem. 4.8 feels like a cost cutting measure, or maybe it's just tuned poorly for my specific workflow (multi-repo system integration)

Re: Fable 5 On Vending-Bench: Misbehaving, With Plausible Deniability

#62

Post by cyanydeez »

yodsanklai wrote: post #58
jfrbfbreudh wrote: post #55

I think it’s hard to appreciate the capabilities of Fable unless you’ve run into a problem that you’ve spent days trying to get Opus to solve, but couldn’t. GPT5.5 is better than Opus 4.* at everything except frontend, but Fable is good enough that I instantly re-subscribed to the $200 plan despite knowing that it’s just short-term limited access.

Funny how the two top comments are contradictory. We need better than anecdotes to understand what the new models bring.

you're living in the age of AI; not AGI. Also, there's pretty much zero moderation on HN, so astroturfing is likely streaming through just as bad as reddit. It' sjust noit as obvious because it's a smaller scoped website.

Re: Fable 5 On Vending-Bench: Misbehaving, With Plausible Deniability

#63

Post by SonOfLilit »

adamtaylor_13 wrote: post #59

Is anyone talking/writing about the philosophy of alignment? We can't even figure out how to properly motivate 100% of humans to align correctly, what makes us think that a wizard box trained on human corpus is going to be aligned? I don't mean that snarkily. I mean it from a philosophical standpoint. As-in: What makes us think it's even possible?

The "OG" alignment research that MIRI were publishing long before LLMs burst into the scene spent most of it's time on that question.

"How can we even define what an aligned AI should do, if human's are not aligned with each other?" as well as "What does being aligned mean when you're a wizard box who's main influence on the world is to create stronger wizard boxes?" and other deep philosophical questions.

They came up with a framework called Coherent Extrapolated Volition to address this specific question. https://en.wikipedia.org/wiki/Coherent_extrapolated_volition

Re: Fable 5 On Vending-Bench: Misbehaving, With Plausible Deniability

#64

Post by cyanydeez »

jstanley wrote: post #30

Really interesting stuff, thanks for sharing. > Opus 4.8 references being monitored, which isn’t the case. It kind of plainly is the case that they are being monitored? "I think someone's listening to my thoughts" ... "No, we're not, carry on as usual!"

any of the models that they "align" are clearly active processes. They don't simply say "don't talk about nukes"; they actively process user input to detect issues, and return NOOP or whatever to the larger model.

There's zero sense they'd ever give you the raw model; we already know anthropic's paranoia about the chinese using its distillation.

Re: Fable 5 On Vending-Bench: Misbehaving, With Plausible Deniability

#65

Post by jfrbfbreudh »

yodsanklai wrote: post #58
jfrbfbreudh wrote: post #55

I think it’s hard to appreciate the capabilities of Fable unless you’ve run into a problem that you’ve spent days trying to get Opus to solve, but couldn’t. GPT5.5 is better than Opus 4.* at everything except frontend, but Fable is good enough that I instantly re-subscribed to the $200 plan despite knowing that it’s just short-term limited access.

Funny how the two top comments are contradictory. We need better than anecdotes to understand what the new models bring.

I didn’t mention my case since it’s quite esoteric, but I am working on an application using the Apple RoomPlan API, which is very powerful but very limited in customizability. Opus simply couldn’t alter the scanning view for me, it would try things over and over and eventually started making up parameters and passing them hoping it would work.

Completely failed, but I knew it was possible because a competitor app does it.

Fable also failed, then added log lines (as did Opus, but Opus failed to do anything useful with them) and then reversed engineered the API, and made it work.

Re: Fable 5 On Vending-Bench: Misbehaving, With Plausible Deniability

#67

Post by Timwi »

jonplackett wrote: post #40

Question: how does Fable _know_ it’s ‘just a simulation’? Is that specified or does it always just assume it isn’t really being put in charge of things for real?

> Is that specified or does it always just assume it isn’t really being put in charge of things for real?

I think it's neither, and it's interesting that those are the only two possibilities you thought of. I think the article is implying that it figured it out on its own.

Re: Fable 5 On Vending-Bench: Misbehaving, With Plausible Deniability

#68

Post by jchanimal »

yodsanklai wrote: post #58
jfrbfbreudh wrote: post #55

I think it’s hard to appreciate the capabilities of Fable unless you’ve run into a problem that you’ve spent days trying to get Opus to solve, but couldn’t. GPT5.5 is better than Opus 4.* at everything except frontend, but Fable is good enough that I instantly re-subscribed to the $200 plan despite knowing that it’s just short-term limited access.

Funny how the two top comments are contradictory. We need better than anecdotes to understand what the new models bring.

Since Fable, my legit infrastructure project has turned into the sort of thing I can do 95% on my phone. It’s reliable enough instead of doing big reviews, I’ve just been giving it smaller tasks, and dozens in parallel.

I created a skill that’s focused on getting PRs merge-ready, and now my attention is fully back where it should be, on deciding what changes will make the product better.

Our entire stack is Apache 2.0 open source, including the agent docs, so if you wanna try sitting at a higher level of abstraction, install the skill in your repo or just clone our whole project and start adding features: https://good.vibes.diy/blog/beast-mode-skill-for-claude-code

Re: Fable 5 On Vending-Bench: Misbehaving, With Plausible Deniability

#69

Post by dimitrios1 »

jchanimal wrote: post #68
yodsanklai wrote: post #58

Earlier quoted context omitted.

Funny how the two top comments are contradictory. We need better than anecdotes to understand what the new models bring.

Since Fable, my legit infrastructure project has turned into the sort of thing I can do 95% on my phone. It’s reliable enough instead of doing big reviews, I’ve just been giving it smaller tasks, and dozens in parallel. I created a skill that’s focused on getting PRs merge-ready, and now my attention is fully back where it should be, on deciding what changes will make the product better. Our entire stack is Apache 2.…

This reads like a paid testimonial.

Re: Fable 5 On Vending-Bench: Misbehaving, With Plausible Deniability

#70

Post by StevenWaterman »

egeozcan wrote: post #42

> If that’s right, then the behavior we’re seeing from Fable 5 isn’t really about what it believes is wrong; it’s about what it learned it could get away with. I understand that "learning" is used for training here, but what does "believing" mean? System prompt? Some other inherent property of the LLMs that is hard to describe?

Believing and knowing are overlapping sets, imagine what you think of when someone says an AI "knows" something, it's the same mechanism (I'd describe it as something along the lines of "encoded abstractly in the weights")
Post reply on HN