Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Fable 5 vs. GPT-5.6 Sol on an NP-Hard Problem: Does /goal help?

charlesazam.com

Post reply on HN
Print view View original

121–130 of 134 posts

Re: Fable 5 vs. GPT-5.6 Sol on an NP-Hard Problem: Does /goal help?

#121

Post by smrtinsert »

This resonated:

  Initially these statements were so absurd on their face that I thought it was some cynical ploy to achieve thought leader status, and there are certainly some people doing this – I have had it admitted to me. But the broader reality is so much worse: people who have no background in the technology at all actually believe what they are saying.

Re: Fable 5 vs. GPT-5.6 Sol on an NP-Hard Problem: Does /goal help?

#122

Post by sreekanth850 »

throwaway314155 wrote: post #106
gmac wrote: post #59

Earlier quoted context omitted.

I've found Opus 4.8 pretty amazing, and Codex a bit meh. YMMV?

My only issue with Codex (Sol 5.6) is that it sometimes finds tasks on its own to solve that are definitely out of scope. like it’s always aiming for the 1.0 release when we’re still on 0.1.0a

Luna respect AGENTS.md and even refuse to mark a task slice completed without testing because i wrote this as rule.

Re: Fable 5 vs. GPT-5.6 Sol on an NP-Hard Problem: Does /goal help?

#123

Post by epolanski »

Tenoke wrote: post #17

Claude seems to forget what you tell it in very long work sessions (things that take weeks to develop), no matter how many times you tell it which part is extra important. I dont use goal (I guess I should), but presumably it makes it actually remember the most important instruction. I believe this here is about shorter sessions where the issue doesn't crop up as much.

It doesn't forget, it doesn't care.

It's a great vibecoding model, but a crap assistant, and it forgets in few turns.

It's trained on benchmarks.

Re: Fable 5 vs. GPT-5.6 Sol on an NP-Hard Problem: Does /goal help?

#126

Post by esperent »

kaelwd wrote: post #119
esperent wrote: post #24

Earlier quoted context omitted.

This is one of the advantages of pi. I made a /protect command that protects the message from compaction. I also protect skills automatically. So for long running tasks I'll do /protect your goal is...

https://github.com/k0valik/pi-blackhole works pretty well for this, instead of a summary of a summary of a summary it gets a rolling log of important instructions and discoveries, and can /recall exact contents of a previous message or tool call if needed.

That looks interesting, have you found it useful in practice? I'm a bit burnt out on these wildly ambitious big plugins (headspace, RTK etc.) because in each case when I've done personal evals they come out worse than the default.

Re: Fable 5 vs. GPT-5.6 Sol on an NP-Hard Problem: Does /goal help?

#127

Post by Tenoke »

fendy3002 wrote: post #118
jrockway wrote: post #87

Earlier quoted context omitted.

I put context size in my Claude Code status bar and ... it does get "tired" when it's at 50%-60% context. I figured this out when I'm like "now modify the testbed to run this and add a test to exercise blah blah blah" and it was just like "That would be a substantial infra lift." In my head I say "I am paying a substantial amount for you to do this!" but just type /compact and re-prompt and my "substantial infra lift…

Use untracked file as memory and instruct agent to put progress into that, it'll help managing context in current and new sessions

I do that but it ends up putting so much extra crap there that it has the same drawbacks.

Re: Fable 5 vs. GPT-5.6 Sol on an NP-Hard Problem: Does /goal help?

#128

Post by lucas_t_a »

ttul wrote: post #71
theptip wrote: post #61

Great eval! If you are comparing search strategies, ultra mode is likely superior. Would love to see a follow-up evaluating that. Ultra can fan out parallel investigators, run adversarial review at defined checkpoints, and do a bunch of other smart stuff to avoid getting stuck in a local optimum. Generally as the OP notes, /goal works better for single-track investigations or small scale scatter/gather.

I think ultra mode needs to be more clearly documented (or perhaps cautioned against!). Most devs - myself included - who saw “ultra” mode figured that it’s just a magic bullet that makes the model work harder and achieve better results. But, for many tasks, ultra mode is possibly worse and certainly more expensive.

i let it run once to just generally find any bugs using all my quota, them once i get back to it i ran on effort /max to continue with explicit don't run workflows fix all yourself, it was ultra effective.

ultra becomes a mode where your model enhances your prompt them it is execution multiplied by all other agents, which have them going in all sorts of directions.

Re: Fable 5 vs. GPT-5.6 Sol on an NP-Hard Problem: Does /goal help?

#129

Post by kaelwd »

esperent wrote: post #126
kaelwd wrote: post #119

Earlier quoted context omitted.

https://github.com/k0valik/pi-blackhole works pretty well for this, instead of a summary of a summary of a summary it gets a rolling log of important instructions and discoveries, and can /recall exact contents of a previous message or tool call if needed.

That looks interesting, have you found it useful in practice? I'm a bit burnt out on these wildly ambitious big plugins (headspace, RTK etc.) because in each case when I've done personal evals they come out worse than the default.

I haven't done a proper benchmark or anything but it isn't noticeably worse at least, and the instant compaction with background processing is nice for slow models.

Re: Fable 5 vs. GPT-5.6 Sol on an NP-Hard Problem: Does /goal help?

#130

Post by jrockway »

aytigra wrote: post #117
jrockway wrote: post #87

Earlier quoted context omitted.

I put context size in my Claude Code status bar and ... it does get "tired" when it's at 50%-60% context. I figured this out when I'm like "now modify the testbed to run this and add a test to exercise blah blah blah" and it was just like "That would be a substantial infra lift." In my head I say "I am paying a substantial amount for you to do this!" but just type /compact and re-prompt and my "substantial infra lift…

I've actually found that compacting as often as possible, after planning, then between each implementation step works the best, it both unloads unnecessary context of previous edits and test runs, makes thinking cheaper, and most importantly after each compaction it re-loads CLAUDE.md which make it much more enforcing (otherwise it just moves to the back of context and slips from model's attention).

I am not sure whether I love the plan + code workflow, but when I do it, I do /clear and instruct "implement PLAN.md" or whatever. (Probably better to do /goal implement PLAN.md; I haven't tried it though.)
Post reply on HN