Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Fable 5 vs. GPT-5.6 Sol on an NP-Hard Problem: Does /goal help?

charlesazam.com

Post reply on HN
Print view View original

71–80 of 134 posts

Re: Fable 5 vs. GPT-5.6 Sol on an NP-Hard Problem: Does /goal help?

#71

Post by ttul »

theptip wrote: post #61

Great eval! If you are comparing search strategies, ultra mode is likely superior. Would love to see a follow-up evaluating that. Ultra can fan out parallel investigators, run adversarial review at defined checkpoints, and do a bunch of other smart stuff to avoid getting stuck in a local optimum. Generally as the OP notes, /goal works better for single-track investigations or small scale scatter/gather.

I think ultra mode needs to be more clearly documented (or perhaps cautioned against!). Most devs - myself included - who saw “ultra” mode figured that it’s just a magic bullet that makes the model work harder and achieve better results. But, for many tasks, ultra mode is possibly worse and certainly more expensive.

Re: Fable 5 vs. GPT-5.6 Sol on an NP-Hard Problem: Does /goal help?

#72

Post by ttul »

enraged_camel wrote: post #70

I've used GPT 5.6 Sol Xhigh extensively since its launch, alongside Fable 5. My impression is that it is about as intelligent as 5.5, but they dialed up the relentlessness meter to eleven. This makes it more likely that it will accomplish the task you give it, which I think is the primary reason it looks competitive in benchmarks. However, it also makes it more likely that it will resort to... unconventional, weird o…

Fable seems to be a larger model. It costs more to run and does not seem superior for _typical_ software engineering work. But for work requiring raw intelligence, perhaps its size is an advantage.

On the DeepSWE 1.1 benchmark (IMHO currently the most relevant and least gamed SWE benchmark), the cost-benefit is clear: 5.6-Sol on xhigh achieves a slightly higher score than Fable 5, but consuming half the tokens and at about 1/3rd the cost.

But, on the Artificial Analysis intelligence index, Fable 5 appears to slightly beat 5.6-Sol, albeit at 3x the cost.

When I am coding, I send tasks to each model to get multiple opinions and it can be hard to predict which model will “win” because the results can be subjective. OP’s task is at least quantifiable, which is great. But many SWE tasks cannot be quantified so easily.

Re: Fable 5 vs. GPT-5.6 Sol on an NP-Hard Problem: Does /goal help?

#73

Post by alexpotato »

ttul wrote: post #71
theptip wrote: post #61

Great eval! If you are comparing search strategies, ultra mode is likely superior. Would love to see a follow-up evaluating that. Ultra can fan out parallel investigators, run adversarial review at defined checkpoints, and do a bunch of other smart stuff to avoid getting stuck in a local optimum. Generally as the OP notes, /goal works better for single-track investigations or small scale scatter/gather.

I think ultra mode needs to be more clearly documented (or perhaps cautioned against!). Most devs - myself included - who saw “ultra” mode figured that it’s just a magic bullet that makes the model work harder and achieve better results. But, for many tasks, ultra mode is possibly worse and certainly more expensive.

> But, for many tasks, ultra mode is possibly worse and certainly more expensive.

Went through something similar. Fable would just spends minutes thinking, processing, confabulating etc.

I dropped down to Haiku and got an answer in >30 seconds.

Re: Fable 5 vs. GPT-5.6 Sol on an NP-Hard Problem: Does /goal help?

#74

Post by perching_aix »

jswny wrote: post #19
Tenoke wrote: post #17

Claude seems to forget what you tell it in very long work sessions (things that take weeks to develop), no matter how many times you tell it which part is extra important. I dont use goal (I guess I should), but presumably it makes it actually remember the most important instruction. I believe this here is about shorter sessions where the issue doesn't crop up as much.

Unfortunately I’ve used Claude and GPT models for a long time in a variety of harnesses and I agree with you and I think it’s the compaction. For some reason, codex compaction is like black magic. I’ve never felt like I can just one one continuous thread with other models, Claude I carefully curate when I compact

That's wild to read considering Codex's compaction to me feels like the model becoming demented. Super aggressive, throws away almost everything.

Re: Fable 5 vs. GPT-5.6 Sol on an NP-Hard Problem: Does /goal help?

#75

Post by Huppie »

alexpotato wrote: post #73
ttul wrote: post #71

Earlier quoted context omitted.

I think ultra mode needs to be more clearly documented (or perhaps cautioned against!). Most devs - myself included - who saw “ultra” mode figured that it’s just a magic bullet that makes the model work harder and achieve better results. But, for many tasks, ultra mode is possibly worse and certainly more expensive.

> But, for many tasks, ultra mode is possibly worse and certainly more expensive. Went through something similar. Fable would just spends minutes thinking, processing, confabulating etc. I dropped down to Haiku and got an answer in >30 seconds.

Simon Willison recently had a useful tip instructing Claude to use its judgement to pick an appropriate model for tasks.

After a short discussion about the idea with Claude mostly on how it fits in my workflow and what models / effort I would like for certain tasks it placed a paragraph in my global Claude.md and it has worked wonders. Ultra became a lot better (faster, cheaper for the same output) and the amount of time Fable gets stuck overthinking things are reduced to the places where I think that model makes sense, for the rest it started fanning out a lot to Opus, Sonnet and even Haiku.

Re: Fable 5 vs. GPT-5.6 Sol on an NP-Hard Problem: Does /goal help?

#76

Post by wgjordan »

brookst wrote: post #26
jswny wrote: post #19

Earlier quoted context omitted.

Unfortunately I’ve used Claude and GPT models for a long time in a variety of harnesses and I agree with you and I think it’s the compaction. For some reason, codex compaction is like black magic. I’ve never felt like I can just one one continuous thread with other models, Claude I carefully curate when I compact

Compacting at all is a mistake. With 1m context window there is no reason for a single task to require compaction. Much better to spend tokens breaking the task into chunks, documenting and storing them durably, then executing each one in clean context and just /clear after. It’s a similar concept to compaction, just planned in advance. Much much more effective, and doesn’t burn tokens and time (“wall-clock”, Claude)…

> With 1m context window there is no reason for a single task to require compaction

Only if money is no object. Cache reads are cheap (10% of uncached input costs) but definitely not free, and cached reads dominate session costs at long context lengths. A prompt at 20k context with $0.01 in cached reads would cost $0.40 in cached reads at 800k context, that quickly adds up for long sessions.

Re: Fable 5 vs. GPT-5.6 Sol on an NP-Hard Problem: Does /goal help?

#78

Post by kccqzy »

brookst wrote: post #26
jswny wrote: post #19

Earlier quoted context omitted.

Unfortunately I’ve used Claude and GPT models for a long time in a variety of harnesses and I agree with you and I think it’s the compaction. For some reason, codex compaction is like black magic. I’ve never felt like I can just one one continuous thread with other models, Claude I carefully curate when I compact

Compacting at all is a mistake. With 1m context window there is no reason for a single task to require compaction. Much better to spend tokens breaking the task into chunks, documenting and storing them durably, then executing each one in clean context and just /clear after. It’s a similar concept to compaction, just planned in advance. Much much more effective, and doesn’t burn tokens and time (“wall-clock”, Claude)…

You just haven’t worked on tasks that are complicated enough. Occasionally it took more than 1M tokens just to come up with a plausible plan.

Personally I find using /rewind judiciously is better than using /compact. The latter essentially gives you no control of what details to discard, but the former at least has coarse-grained control.

Re: Fable 5 vs. GPT-5.6 Sol on an NP-Hard Problem: Does /goal help?

#79

Post by sdesol »

esperent wrote: post #24
Tenoke wrote: post #17

Claude seems to forget what you tell it in very long work sessions (things that take weeks to develop), no matter how many times you tell it which part is extra important. I dont use goal (I guess I should), but presumably it makes it actually remember the most important instruction. I believe this here is about shorter sessions where the issue doesn't crop up as much.

This is one of the advantages of pi. I made a /protect command that protects the message from compaction. I also protect skills automatically. So for long running tasks I'll do /protect your goal is...

I am curious as to how this protect works. I am currently working on a pi brain extensions and one of the commands that I will support is:

/brains inspect

that will let you inspect a session and one of the features is the ability to review a compacted message like so:

https://gitsense.com/screenshots/inspect-pi-session-compacte...

https://gitsense.com/screenshots/inspect-pi-session-compacte...

https://gitsense.com/screenshots/inspect-pi-session-compacte...

With /compact in Pi, it creates a message that you can easily review and I am curious as to how '/protect' works.

One of the features that I am working on is to make it easy for agents to retrieve the exact message/event before compaction and I am curious if /protect is a deterministic process or if it is just instructions.

Re: Fable 5 vs. GPT-5.6 Sol on an NP-Hard Problem: Does /goal help?

#80

Post by lionkor »

throw10920 wrote: post #69
esperent wrote: post #24

Earlier quoted context omitted.

This is one of the advantages of pi. I made a /protect command that protects the message from compaction. I also protect skills automatically. So for long running tasks I'll do /protect your goal is...

I'm also a fan of Pi - although, doesn't Claude code (and most other harnesses) already have "custom summarization instructions" for compaction/summarization that already allow you to do this? Or have you added on something extra?

They way I read it, this is a workaround for pi, not for Claude.

Edit: removed off topic political spam

Post reply on HN