Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

You only need the frontier model for one single edit

stencil.so

Post reply on HN
Print view View original

71–80 of 107 posts

Re: You only need the frontier model for one single edit

#71

Post by unholiness »

leobg wrote: post #44

I guess Anthropic knows this. They show you this warning: --- Switch model? Your next response will be slower and use more tokens This conversation is cached for the current model. Switching to Opus 4.8 (1M context) means the full history gets re-read on your next message. --- Reminds me a bit of airline booking sites ("We noticed you didn't add insurance").

I mean, this is real. Your KV cache lives for an hour since the last token (on anthropic pro/Max at least, today at least, this has degraded in the past). So once you pay, say, 50k input and 50k output Opus tokens, you don't pay for cache reads of the 100k context while it builds. Switch to another model, you'll start with the cost of 100k input tokens on the smaller model to get that context loaded and its unique KV cache set.

The post may have some real insight here, where this 100k working context is actually better than trying to summarize it's findings into a plan. It's also right that the handoff is a perfect time to edit the context (removing planning instructions). But it doesn't mention it's a trade-off: the plan is smaller, so it's a cheaper "on-boarding" of the next model. Send quite plausible that this is worth it for 1-off tasks. If it's right, this is basically "plans are useless, planning is everything" for LLMs.

My problem is, I think the plans are useful. I want to review and edit them. I want them to give context for the upcoming code review (even if humans aren't reviewing). LLMs are notoriously bad at explaining why they're doing something in the moment. Humans are notoriously bad at accepting there's no reason why. I think the humans have it right here, and to bridge this gap want my PRs, my docs, and my comments teeming with reasons why.

(NB: ranted to long, reposting at top level)

Re: You only need the frontier model for one single edit

#72

Post by unholiness »

No mention of KV cache, one of the biggest reasons not to switch models mid-stream. Once you pay, say, 50k input and 50k output Opus 4.8 tokens, you don't pay token cost for cache reads of the 100k context while it builds. Switch to another model, you'll start with the cost of 100k input tokens on the smaller model to get that context loaded and its unique KV cache set.

The post may have some real insight here, where this 100k working context is actually better than trying to summarize it's findings into a plan. It's also right that the handoff is a perfect time to edit the context (removing planning instructions). But it doesn't mention it's a trade-off: the plan is smaller, so it's a cheaper "on-boarding" of the next model. Seems quite plausible that this is worth it for 1-off tasks. If it's right, this is basically "plans are useless, planning is everything" for LLMs.

My problem is, I think the plans are useful. I want to review and edit them. I want them to give context for the upcoming code review (even if humans aren't reviewing). LLMs are notoriously bad at explaining why they're doing something in the moment. Humans are notoriously bad at accepting there's no reason why. I think the humans have it right here, and to bridge this gap want my PRs, my docs, and my comments teeming with reasons why. Plans help with that.

Re: You only need the frontier model for one single edit

#73

Post by CBLT »

unholiness wrote: post #72

No mention of KV cache, one of the biggest reasons not to switch models mid-stream. Once you pay, say, 50k input and 50k output Opus 4.8 tokens, you don't pay token cost for cache reads of the 100k context while it builds. Switch to another model, you'll start with the cost of 100k input tokens on the smaller model to get that context loaded and its unique KV cache set. The post may have some real insight here, where…

Not by name, sure, but it does mention that you're going to be reading the same context into the second model. It specifically refutes your point that you're only reading the smaller plan:

> Opus reads base.py, signing.py, the test file (twenty cards of gray), then writes its plan and leaves. And what's the first thing Flash does with that beautiful document? It re-reads base.py and the test file, because a plan is not a file and you cannot edit prose. The gray reads just keep stacking, first at Opus prices, then again at Flash prices. There is no version of this where a second reader is the cost optimization.

Re: You only need the frontier model for one single edit

#74

Post by walthamstow »

kademolu wrote: post #64

Good stats but how does this work with models like Claude where a lot of the context or relevant information is stored in its "memory", other coding agents can't use it.

You turn auto memory off in Claude Code and manage the context yourself

Re: You only need the frontier model for one single edit

#78

Post by abirch »

robbie-c wrote: post #77

I'm begging people to write articles themselves rather than letting Claude do it for them. I want an expert opinion, if I just wanted to ask an LLM I have my own. How can this article not mention the KV cache even once?

I wish that Hacker News had the option to flag things at AI and let people filter those things out.

Re: You only need the frontier model for one single edit

#79

Post by markust »

bensyverson wrote: post #55
ricardobeat wrote: post #29

Earlier quoted context omitted.

This is exactly what the post argues against though, as it leads to higher overall cost.

Yes, and I'm arguing that the post is over-simplifying. As I mention, for small plans, it's probably easier to have the big model just do it. But for large, multi-session plans, it's cheaper to let smaller models execute the bulk of the work.

Perhaps, there's a small change that might work better here - if we use a different agent (by creating custom agents) for plan and implementation, then the switch to the implementation agent wouldn't end up re-reading a lot as the new agent would have fresh empty context. I implemented this[1] from one of the posters on HN and I think it'd work well with this problem too. Would be great if you could point out if my understanding is wrong.

[1]: https://www.stavros.io/posts/how-i-write-software-with-llms/

Post reply on HN