Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

You only need the frontier model for one single edit

stencil.so

Post reply on HN
Print view View original

81–90 of 107 posts

Re: You only need the frontier model for one single edit

#81

Post by resonious »

unholiness wrote: post #72

No mention of KV cache, one of the biggest reasons not to switch models mid-stream. Once you pay, say, 50k input and 50k output Opus 4.8 tokens, you don't pay token cost for cache reads of the 100k context while it builds. Switch to another model, you'll start with the cost of 100k input tokens on the smaller model to get that context loaded and its unique KV cache set. The post may have some real insight here, where…

The article shows that it's still cheaper to pay for the switch than it is to let the frontier model do all the work. At least in benchmarks.

If you want to interact with plans then I think this technique just isn't for you.

Re: You only need the frontier model for one single edit

#82

Post by resonious »

robbie-c wrote: post #77

I'm begging people to write articles themselves rather than letting Claude do it for them. I want an expert opinion, if I just wanted to ask an LLM I have my own. How can this article not mention the KV cache even once?

Why does KV cache matter if they show it's cheaper anyway?

Re: You only need the frontier model for one single edit

#83

Post by ClikeX »

hankbond wrote: post #11

Does anyone have a name for that really neat "the bar chart is a bookshelf, you can hover a spine to see the cover" vis? I've never seen it before.

No idea if it has a name. But the page source is also not giving me anything useful. And it pains me to even look at it.

Re: You only need the frontier model for one single edit

#84

Post by trash_cat »

Imanari wrote: post #63
nxtfari wrote: post #35

This is really smart, like the author said, old idea but cleverly applied. In case anyone wants a summary: don’t one shot, don’t use plan mode and hand off the plan to cheap executors, ask the frontier model to explore, create a todo list, and then start when it feels confident; stop it after first code edit, then prefill the context to cheap executor to continue.

So its basically (a plan + a bunch of file reads + a first edit) injected into the context of a cheap model so the cheap model does not feel like it has to re-read the files and just continues executing and editing?

Yes this is correct. It essentially biases the cheaper model into procedural action grounded by the frontier model. To put it mode generally, it injects the cheap model context with more useful information, where it might not need to read all the files again.

Re: You only need the frontier model for one single edit

#85

Post by rolymath »

ricardobeat wrote: post #28

I just did this five minutes ago. Started a task with Kimi K3, noticed cost going up, switched to Minimax M3 continuing from the same context. Really easy to do with Crush/OpenCode. It works quite well most of the time. For very deeply technical tasks, where exploration involves multiple agents (and blowing up context limits a few times), I'll have a plan written down to disk that will be much longer than 2k tokens.

Wow. Someone using Crush in the wild. What do you think of it? I'm having trouble switching to it because the rest work fine, but would love to actually adopt it.

Re: You only need the frontier model for one single edit

#86

Post by trash_cat »

ursuscamp wrote: post #67

Can someone explain to me the difference between this approach and using planning with a larger model, then just switching to a small model for implementation without clearing the context? I understand that it specifically does the first edit as well either way the larger model. Is there some other difference I am missing here?

When a frontier makes a succesfull edit based on the plan that it made, it leaves an procedural trace in turn biases the NEXT model, low cost model, straight into procedural action. The cheaper model doesn't need to reread everything again because it has enough information from the frontier model to complete the task. A simple "plan" of what needs to be done does not carry this information.

Re: You only need the frontier model for one single edit

#87

Post by wrs »

sgc wrote: post #24
wrs wrote: post #7

Lately I'm trying a variation on this: Have the main agent make a phased implementation plan, then for each phase, have it start an implementation subagent with a focused prompt, then review that agent's work in the main session. The theory being that the main session still contains all the research, but it can review just the diff rather than have the entire implementation session in context as well. The post doesn'…

Actually it does include an important level of review. They say "init a TODO list with a validation step for each item", and then the last several sections of the execution are within the TODO - where it is validating the code. To me that was the heart of the article and of what they did - create a self-validating todo list that deterministically forces the agents to stay on task, and then trim out just the right ext…

It wasn’t clear to me (and still isn’t, after slogging through the inhuman article again) that the verification here is done by a separate agent from the implementation. That’s a second trimming step, but in reverse: at that point you want the new code in the context, but not the implementation reasoning.

Re: You only need the frontier model for one single edit

#88

Post by wrs »

bensyverson wrote: post #26
wrs wrote: post #7

Lately I'm trying a variation on this: Have the main agent make a phased implementation plan, then for each phase, have it start an implementation subagent with a focused prompt, then review that agent's work in the main session. The theory being that the main session still contains all the research, but it can review just the diff rather than have the entire implementation session in context as well. The post doesn'…

Yes, this is a core part of my workflow; use the smartest model (e.g. Fable) to generate a large hierarchical implementation plan, then clear the context and have a lesser model (e.g. Sonnet) track and execute the plan [0], often in parallel. It all depends on how much work you're doing; if it's a simple task, there's virtually no need for plans at all; just have the smart agent do it. But if the work is going to tak…

My main point was not about the implementation stage, but the verification stage. Letting the smarter agent verify the work of the cheap agent finds a lot of little oversights and sloppiness that I used to have to find myself. So I’d be interested in that cost comparison.

Re: You only need the frontier model for one single edit

#89

Post by jxmorris12 »

abirch wrote: post #78
robbie-c wrote: post #77

I'm begging people to write articles themselves rather than letting Claude do it for them. I want an expert opinion, if I just wanted to ask an LLM I have my own. How can this article not mention the KV cache even once?

I wish that Hacker News had the option to flag things at AI and let people filter those things out.

Me too. maybe Pangram solves this?

Re: You only need the frontier model for one single edit

#90

Post by robbie-c »

robbie-c wrote: post #77

I'm begging people to write articles themselves rather than letting Claude do it for them. I want an expert opinion, if I just wanted to ask an LLM I have my own. How can this article not mention the KV cache even once?

While I'm here - I recently optimized our PR review skill to make better use of the KV cache.

Previously, it loaded up the diff, persona and prompt, and wrote those into the first message for each of 6 sub-agent reviewers. The prompt was templated with the persona name, so was slightly different for each reviewer.

My optimized version had a common first message with diff + prompt, and a script to run to atomically claim a persona. It also runs the first reviewer before the rest to warm the KV cache, and the agent doesn't launch the rest of the subagents until the first persona has been claimed (which means that the LLM is running, and therefore the KV cache is warm). The agent has a script it runs which does the waiting for it.

Agents 2-6 only run once the KV cache is warm, and because the intro message including the diff is shared, it's there in the cache. Only the persona file is different.

In my testing this brought down the cost of a review by ~half, though of course this depends on how big the diff is, how many agent are launched, and what model is used.

Post reply on HN