Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Fable 5 vs. GPT-5.6 Sol on an NP-Hard Problem: Does /goal help?

charlesazam.com

Post reply on HN
Print view View original

81–90 of 134 posts

Re: Fable 5 vs. GPT-5.6 Sol on an NP-Hard Problem: Does /goal help?

#81

Post by cyral »

gmac wrote: post #59
sreekanth850 wrote: post #58

Anthropic is losing bigtime against Open AI in coding space. I was using Claude code upto last march. Not an enterprise customer, but a responsible AI user where i don't over spend and use basic plan to manage repository with 400Kloc all together. We sell to local government and a team of 3. Claude code was super slow, never able to fix issues properly.(Despite with proper test cases, observability, documentation and…

I've found Opus 4.8 pretty amazing, and Codex a bit meh. YMMV?

I just downloaded and paid for Codex this week because I want to stay on top of the AI tools and understand their capabilities. I've had some good results using 5.6 Sol, although it tends to never want to write any comments (despite modifying the project rules to tell it it MUST), and also occasionally just does a bit of thinking and stops. It'll say "Working through the remaining work" and just ends the chat until I tell it "continue" or something.

This is very anecdotal evidence but I have to rant about it... I tried 5.6 Terra (high) earlier today to fix a bug with a slow page. It just... removed the part of the page that was slow, and made it a client side request (still slow, but not blocking SSR I guess). I tried with Sonnet 5 and it correctly found the issue where an unhandled case was continuing to retry and failing. I am always telling people how the frontier models are SO much more capable than what they may think but this one thing today had me scratching my head at why it would ever do that. It was the first time I experienced the "great, I removed the failing test case" kind of issue.

Re: Fable 5 vs. GPT-5.6 Sol on an NP-Hard Problem: Does /goal help?

#82

Post by throw10920 »

lionkor wrote: post #80
throw10920 wrote: post #69

Earlier quoted context omitted.

I'm also a fan of Pi - although, doesn't Claude code (and most other harnesses) already have "custom summarization instructions" for compaction/summarization that already allow you to do this? Or have you added on something extra?

They way I read it, this is a workaround for pi, not for Claude. Edit: removed off topic political spam

Can you please keep the unsolicited ranting out of the replies? It breaks the guidelines and degrades the value of the site. Thanks!

https://news.ycombinator.com/newsguidelines.html

Re: Fable 5 vs. GPT-5.6 Sol on an NP-Hard Problem: Does /goal help?

#83

Post by lionkor »

throw10920 wrote: post #82
lionkor wrote: post #80

Earlier quoted context omitted.

They way I read it, this is a workaround for pi, not for Claude. Edit: removed off topic political spam

Can you please keep the unsolicited ranting out of the replies? It breaks the guidelines and degrades the value of the site. Thanks! https://news.ycombinator.com/newsguidelines.html

It's part of the rationale for why one would choose Pi over Claude, even if Pi lacks the feature. I admit my tone is entirely inappropriate though.

Re: Fable 5 vs. GPT-5.6 Sol on an NP-Hard Problem: Does /goal help?

#84

Post by embedding-shape »

ttul wrote: post #71
theptip wrote: post #61

Great eval! If you are comparing search strategies, ultra mode is likely superior. Would love to see a follow-up evaluating that. Ultra can fan out parallel investigators, run adversarial review at defined checkpoints, and do a bunch of other smart stuff to avoid getting stuck in a local optimum. Generally as the OP notes, /goal works better for single-track investigations or small scale scatter/gather.

I think ultra mode needs to be more clearly documented (or perhaps cautioned against!). Most devs - myself included - who saw “ultra” mode figured that it’s just a magic bullet that makes the model work harder and achieve better results. But, for many tasks, ultra mode is possibly worse and certainly more expensive.

> that it’s just a magic bullet that makes the model work harder and achieve better results

"Ultra" is a harness feature, and has nothing to do with the model itself. If OpenAI wanted to, they could offer "Ultra" with any of the GPT models, although 5.6 supposedly been trained with this specific harness feature in mind. "max" is basically the top "work harder and achieve better results" parameter for the reasoning effort for the current models.

I had good success with it initially, until I discovered that OpenAI encrypts the prompts that the main model sends to the sub-agents (even in Codex, in local files you only see cipher text), then I completely dropped any experimentation of it as without introspection, it becomes basically useless for any real usage.

Re: Fable 5 vs. GPT-5.6 Sol on an NP-Hard Problem: Does /goal help?

#85

Post by jrockway »

gmac wrote: post #59
sreekanth850 wrote: post #58

Anthropic is losing bigtime against Open AI in coding space. I was using Claude code upto last march. Not an enterprise customer, but a responsible AI user where i don't over spend and use basic plan to manage repository with 400Kloc all together. We sell to local government and a team of 3. Claude code was super slow, never able to fix issues properly.(Despite with proper test cases, observability, documentation and…

I've found Opus 4.8 pretty amazing, and Codex a bit meh. YMMV?

I think the comment you're replying to is talking about cost and speed, and Opus 4.8 xhigh is pretty expensive and pretty slow. I work around the slowness by having multiple auto mode sessions going in parallel (review this, investigate that, plan this, etc.; and yes, I use a VM that can't touch prod for auto-mode work), and work around the cost by my employer pays for it and everyone else I work with uses more AI than I do :)

I really really like Fable for software engineering cases; add an alert and write a runbook, go pull metrics for this incident, write me a TUI that does blank, etc. It is astoundingly good and I am very picky. It costs A LOT of money though. I am not sure how I get away with my Fable usage.

Re: Fable 5 vs. GPT-5.6 Sol on an NP-Hard Problem: Does /goal help?

#86

Post by ttul »

embedding-shape wrote: post #84
ttul wrote: post #71

Earlier quoted context omitted.

I think ultra mode needs to be more clearly documented (or perhaps cautioned against!). Most devs - myself included - who saw “ultra” mode figured that it’s just a magic bullet that makes the model work harder and achieve better results. But, for many tasks, ultra mode is possibly worse and certainly more expensive.

> that it’s just a magic bullet that makes the model work harder and achieve better results "Ultra" is a harness feature, and has nothing to do with the model itself. If OpenAI wanted to, they could offer "Ultra" with any of the GPT models, although 5.6 supposedly been trained with this specific harness feature in mind. "max" is basically the top "work harder and achieve better results" parameter for the reasoning ef…

Oh yes, I know it’s a harness feature.

Re: Fable 5 vs. GPT-5.6 Sol on an NP-Hard Problem: Does /goal help?

#87

Post by jrockway »

Tenoke wrote: post #17

Claude seems to forget what you tell it in very long work sessions (things that take weeks to develop), no matter how many times you tell it which part is extra important. I dont use goal (I guess I should), but presumably it makes it actually remember the most important instruction. I believe this here is about shorter sessions where the issue doesn't crop up as much.

I put context size in my Claude Code status bar and ... it does get "tired" when it's at 50%-60% context. I figured this out when I'm like "now modify the testbed to run this and add a test to exercise blah blah blah" and it was just like "That would be a substantial infra lift." In my head I say "I am paying a substantial amount for you to do this!" but just type /compact and re-prompt and my "substantial infra lift" is done without complaint.

I guess everyone needs a nap after a long day of conversing and writing code. So like us!

/compact is prone to error and I wouldn't recommend it in the middle of work. But when you are switching to a related but not completely new task, it helps. ("Now write the integration tests." vs. "on foo.go line 476 that you just wrote, I think there is a deadlock with bar.go line 123 that you just added". It doesn't really need the context to write the tests, it can get that by reading the code. But for iterating on lines of code it just produced, /compact is going to throw away whatever "thought process" led to that code and it's usually not a great thing to do.)

Re: Fable 5 vs. GPT-5.6 Sol on an NP-Hard Problem: Does /goal help?

#89

Post by throw10920 »

lionkor wrote: post #83
throw10920 wrote: post #82

Earlier quoted context omitted.

Can you please keep the unsolicited ranting out of the replies? It breaks the guidelines and degrades the value of the site. Thanks! https://news.ycombinator.com/newsguidelines.html

It's part of the rationale for why one would choose Pi over Claude, even if Pi lacks the feature. I admit my tone is entirely inappropriate though.

> It's part of the rationale for why one would choose Pi over Claude

It's unrelated to my direct comment and it's a generic tangent off the thread further up, which is about technical issues with Claude Code. As the guidelines say:

"On-Topic: Anything that good hackers would find interesting. That includes more than hacking and startups. If you had to reduce it to a sentence, the answer might be: anything that gratifies one's intellectual curiosity."

"Eschew flamebait. Avoid generic tangents."

"Please don't use Hacker News for political or ideological battle. It tramples curiosity."

Nobody wants to see you hijacking technical threads to turn them into political flamewars. It's against the guidelines, anti-intellectual, and profoundly boring.

Post reply on HN