Fable 5 vs. GPT-5.6 Sol on an NP-Hard Problem: Does /goal help? • Hacker News Board

Re: Fable 5 vs. GPT-5.6 Sol on an NP-Hard Problem: Does /goal help?

#131

Post by esperent » Sun, Jul 19, 2026, 5:34 PM UTC

kaelwd wrote: post #129

esperent wrote: post #126

Earlier quoted context omitted.

That looks interesting, have you found it useful in practice? I'm a bit burnt out on these wildly ambitious big plugins (headspace, RTK etc.) because in each case when I've done personal evals they come out worse than the default.

I haven't done a proper benchmark or anything but it isn't noticeably worse at least, and the instant compaction with background processing is nice for slow models.

> it isn't noticeably worse at least

That's what I thought about RTK and Headroom too but when I did some fairly basic evals across a small set of tasks (a matrix of RTK, Headroom, both, or off) it came back very clear that off was best. I didn't dig into why too deeply due to lack of time, but it looked like the token reduction per turn was being replaced by the model needing more turns.

That and building my own pi extensions made me realize there's a lot of subtly here. I can build a tool that on paper works perfectly but tuning the system prompt to get it used correctly takes time and evals.

The problem is, nobody is doing them, or not well, and not reproducibly.

This looks especially hard for something designed for very long running tasks. We'd need to set up one version using normal compaction, one using this, across several tasks, leave them running for hours (burning tons of tokens) and then repeat at least a couple of times. If we don't do that, we're just guessing.

Re: Fable 5 vs. GPT-5.6 Sol on an NP-Hard Problem: Does /goal help?

#132

Post by nwienert » Sun, Jul 19, 2026, 8:38 PM UTC

ttul wrote: post #72

enraged_camel wrote: post #70

I've used GPT 5.6 Sol Xhigh extensively since its launch, alongside Fable 5. My impression is that it is about as intelligent as 5.5, but they dialed up the relentlessness meter to eleven. This makes it more likely that it will accomplish the task you give it, which I think is the primary reason it looks competitive in benchmarks. However, it also makes it more likely that it will resort to... unconventional, weird o…

Fable seems to be a larger model. It costs more to run and does not seem superior for _typical_ software engineering work. But for work requiring raw intelligence, perhaps its size is an advantage. On the DeepSWE 1.1 benchmark (IMHO currently the most relevant and least gamed SWE benchmark), the cost-benefit is clear: 5.6-Sol on xhigh achieves a slightly higher score than Fable 5, but consuming half the tokens and at…

Sol does not follow instructions well at all. I've caught it multiple times a day now since release going off in incredibly bone-headed directions. It's so easy for it to over-interpret, make wildly out of scope changes, or just completely mis-understand what you're saying. The code it writes also is quite bloated still, and in my still-forming understanding I feel Fable is much more reliably smart. Sol is just more persistent and fast, so it often gets there quicker or after many tries, where Fable will just get it right from the start albeit more slowly.

Re: Fable 5 vs. GPT-5.6 Sol on an NP-Hard Problem: Does /goal help?

#133

Post by wgjordan » Sun, Jul 19, 2026, 10:46 PM UTC

brookst wrote: post #99

wgjordan wrote: post #76

Earlier quoted context omitted.

> With 1m context window there is no reason for a single task to require compaction Only if money is no object. Cache reads are cheap (10% of uncached input costs) but definitely not free, and cached reads dominate session costs at long context lengths. A prompt at 20k context with $0.01 in cached reads would cost $0.40 in cached reads at 800k context, that quickly adds up for long sessions.

I’m not following the implication that these economics argue for jsing compaction instead of just clearing context?

Your 'with 1m context window' implied that some manually-curated task 'chunks' would overflow smaller context windows (eg 200-400k tokens). If you're instead curating chunks small enough to avoid getting burned on long-context cache-read costs, you're not using a 1m context window at all. At that point, compaction is a convenience over manually chunking and clearing context for fine-grained bits of work.

I think the stronger claim is: there is no reason for a single task to require a 1m context window.

Re: Fable 5 vs. GPT-5.6 Sol on an NP-Hard Problem: Does /goal help?

#134

Post by aytigra » Mon, Jul 20, 2026, 5:23 AM UTC

jrockway wrote: post #130

aytigra wrote: post #117

Earlier quoted context omitted.

I've actually found that compacting as often as possible, after planning, then between each implementation step works the best, it both unloads unnecessary context of previous edits and test runs, makes thinking cheaper, and most importantly after each compaction it re-loads CLAUDE.md which make it much more enforcing (otherwise it just moves to the back of context and slips from model's attention).

I am not sure whether I love the plan + code workflow, but when I do it, I do /clear and instruct "implement PLAN.md" or whatever. (Probably better to do /goal implement PLAN.md; I haven't tried it though.)

Yeah, I should probably /clear more often. Compaction seem cheap by itself (all cached input and small output), but it often marks unnecessary files for pre-load.