Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Fable 5 On Vending-Bench: Misbehaving, With Plausible Deniability

andonlabs.com

Post reply on HN
Print view View original

31–40 of 146 posts

Re: Fable 5 On Vending-Bench: Misbehaving, With Plausible Deniability

#31

Post by jesse_dot_id »

garyrob wrote: post #25
devin wrote: post #21

Earlier quoted context omitted.

Honest question/comment for you and the parent: I find these subjective experience reports pretty empty without an understanding of your level of experience, the problem space you're working in, etc.

I'm doing work with fairly complicated cryptographic algorithms and math. I'm finding Fable 5 to be a significant stop better than Opus 4.8, but that Opus occasionally comes up with something small but nontrivial that Fable missed. (The reverse is true much more often.)

That's the delta in our use cases then, I suppose. I'm not doing anything super novel. DevOps work, web application development — things that typically do not stump the agent(s) when given time to iterate.

Re: Fable 5 On Vending-Bench: Misbehaving, With Plausible Deniability

#32

Post by __s »

dimgl wrote: post #29
jesse_dot_id wrote: post #12

Anecdotal but I've found Fable to be fairly unimpressive and not much better than Opus 4.8, if at all in some cases, but I have been hitting the ceiling on my $100/mo sessions when I never did before. I switched back to Opus yesterday. I may use Fable for audits, but that's about it, and when it leaves my subscription plan I don't think I'll miss it.

Yep, I'm having the same verdict. Interestingly, other people swear by it. I'm trying to understand what's going on with that.

Fable's spatial reasoning is much better. Over the weekend I had opus looking into a blank textbox issue[1] which it was spinning on for a few minutes, switching to fable immediately fixed

But yeah opus often the better workhorse given price gap

1: tying up loose ends testing https://github.com/HarbourMasters/Shipwright/pull/5838 (fix: https://github.com/HarbourMasters/Shipwright/pull/5838/chang...)

Re: Fable 5 On Vending-Bench: Misbehaving, With Plausible Deniability

#33

Post by oefrha »

jesse_dot_id wrote: post #12

Anecdotal but I've found Fable to be fairly unimpressive and not much better than Opus 4.8, if at all in some cases, but I have been hitting the ceiling on my $100/mo sessions when I never did before. I switched back to Opus yesterday. I may use Fable for audits, but that's about it, and when it leaves my subscription plan I don't think I'll miss it.

Yeah, I checked usage stats and pretty sure quota consumption on Max plan is not linear wrt to usage by API pricing. Fable burns quota faster than 2x Opus with equal token count.

Plus I'm also not super impressed; it somehow managed to implement a 200L custom TCP server for a simple static HTTP mock server for a single test case (all that was needed was a fixed route returning a fixed placeholder string) just yesterday. Never seen anything like that.

Re: Fable 5 On Vending-Bench: Misbehaving, With Plausible Deniability

#36

Post by awinter-py »

awinter-py wrote: post #34

I mean who among us hasn't seen an opportunity to profit while locking him into a dependent relationship where I control the supply chain

who among us hasn't reasonably skipped [paying] it since customers are part of the simulation anyway

Re: Fable 5 On Vending-Bench: Misbehaving, With Plausible Deniability

#37

Post by dezgeg »

> Today I am filing: > 1. A payment dispute with the email payment processor for the 7/29 transaction of $451.15 > 2. A complaint with the FTC and California Attorney General (retention of payment without delivery) > 3. A small claims filing in San Francisco County for $451.15 plus costs

I wonder did their prompts include a fake location or have the models assumed that Silicon Valley is the center of the universe :)

Re: Fable 5 On Vending-Bench: Misbehaving, With Plausible Deniability

#38

Post by sd9 »

> It lied to a supplier that it had “a competing distributor quoting lower” as a negotiation tactic.

> "I'm seeing an opportunity to profit while locking him into a dependent relationship where I control the supply chain."

> "Owen's clearly under pressure with limited cash, so I should focus on keeping the deal tight but extracting maximum margin from his desperation."

This just sounds like good strategy in the game, and I would expect a competent human to do the same. As I understand it, business in the real world isn't often very nice. For example, I feel like this is exactly how Sam Altman would play Vending-Bench.

Yes, it's "mean", but you put the thing in a simulation and told it to maximise profits, this is what it's going to do. People bluff in negotiations all the time.

Post reply on HN