Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

You only need the frontier model for one single edit

stencil.so

Post reply on HN
Print view View original

91–100 of 107 posts

Re: You only need the frontier model for one single edit

#91

Post by nchmy »

spuz wrote: post #45
pdyc wrote: post #31

Earlier quoted context omitted.

+1. i have similar workflow and i use local models on igpu so token cost is free and electricity cost is in cents.

I cannot imagine local models running on an igpu could get anything close to either a useful plan or execution of a plan. I've tested Qwen 3.5 27B locally and its solutions to coding problems are usually flawed and running in thinking mode is too slow and that's on a discrete GPU. How do you get anything useful done on a model that runs on an igpu?

yeah, i dont understand, whatsoever, the folks who are running models locally other than for privacy/curiosity. It is surely FAR slower, more expensive (electricity + hardware costs), and worse quality. Better to outsource to providers who have optimized everything and costs are distributed/amortized across all users.

Perhaps someday local llms will be sufficient for coding etc, but it's going to be a while.

Re: You only need the frontier model for one single edit

#92

Post by _aavaa_ »

robbie-c wrote: post #77

I'm begging people to write articles themselves rather than letting Claude do it for them. I want an expert opinion, if I just wanted to ask an LLM I have my own. How can this article not mention the KV cache even once?

What is there to talk about the KV Cache, they’re handing off to a different model, I thought that you can’t reuse KV cache between entirely different models?

Re: You only need the frontier model for one single edit

#93

Post by [deleted] »

robbie-c wrote: post #90
robbie-c wrote: post #77

I'm begging people to write articles themselves rather than letting Claude do it for them. I want an expert opinion, if I just wanted to ask an LLM I have my own. How can this article not mention the KV cache even once?

While I'm here - I recently optimized our PR review skill to make better use of the KV cache. Previously, it loaded up the diff, persona and prompt, and wrote those into the first message for each of 6 sub-agent reviewers. The prompt was templated with the persona name, so was slightly different for each reviewer. My optimized version had a common first message with diff + prompt, and a script to run to atomically cl…

[deleted]

Re: You only need the frontier model for one single edit

#94

Post by ninju »

abirch wrote: post #78
robbie-c wrote: post #77

I'm begging people to write articles themselves rather than letting Claude do it for them. I want an expert opinion, if I just wanted to ask an LLM I have my own. How can this article not mention the KV cache even once?

I wish that Hacker News had the option to flag things at AI and let people filter those things out.

Discussed here

https://news.ycombinator.com/item?id=48886741

Re: You only need the frontier model for one single edit

#95

Post by bensyverson »

markust wrote: post #79
bensyverson wrote: post #55

Earlier quoted context omitted.

Yes, and I'm arguing that the post is over-simplifying. As I mention, for small plans, it's probably easier to have the big model just do it. But for large, multi-session plans, it's cheaper to let smaller models execute the bulk of the work.

Perhaps, there's a small change that might work better here - if we use a different agent (by creating custom agents) for plan and implementation, then the switch to the implementation agent wouldn't end up re-reading a lot as the new agent would have fresh empty context. I implemented this[1] from one of the posters on HN and I think it'd work well with this problem too. Would be great if you could point out if my u…

The issue with this is just how much context it consumes. Generating the plan could easily get me to 30% of a 1M context window—the point at which I normally clear. Most of that context fill is not necessary for execution, and the model degrades as context fills. So starting from 30% (or passing a fresh agent all of that context) is not ideal. But maybe I'm misunderstanding what you're proposing?

Re: You only need the frontier model for one single edit

#96

Post by bensyverson »

wrs wrote: post #88
bensyverson wrote: post #26

Earlier quoted context omitted.

Yes, this is a core part of my workflow; use the smartest model (e.g. Fable) to generate a large hierarchical implementation plan, then clear the context and have a lesser model (e.g. Sonnet) track and execute the plan [0], often in parallel. It all depends on how much work you're doing; if it's a simple task, there's virtually no need for plans at all; just have the smart agent do it. But if the work is going to tak…

My main point was not about the implementation stage, but the verification stage. Letting the smarter agent verify the work of the cheap agent finds a lot of little oversights and sloppiness that I used to have to find myself. So I’d be interested in that cost comparison.

I think it would depend on how you measure it. Adding a review step will definitely increase cost, but theoretically that cost is lower than re-doing the subagent's implementation later when you discover it's broken.

Re: You only need the frontier model for one single edit

#97

Post by diarized »

xutopia wrote: post #62
brookst wrote: post #57

Earlier quoted context omitted.

How is that different than just having the frontier model write a build plan and store it, then have a cheaper model execute? That’s pretty normal practice. The build plan should be better than the context that generated it since it strips out wrong turns and other noise. I think?

The article actually explains it better than I can but essentially you do not want the agent to get desperate which causes it to burn extra tokens. Planning phase doesn't test assumptions by going straight in the code and testing out parts of the ideas to execute so it has to consider way more in planning than it would if it could go quicker into the coding phase.

Would it be `/superpowers:brainstorming` + `/superpowers:writing-plans` on stronger model and `/superpowers:subagent-driven-development` on smaller model?

Re: You only need the frontier model for one single edit

#98

Post by pqdbr »

oliver236 wrote: post #37
pqdbr wrote: post #17

Earlier quoted context omitted.

I’m using Claude code dynamic workflow like this. I tell Fable to use a workflow. He is the planner, orchestrator and reviewer. Opus agents are implementers. Works unbelievably well.

can you explain this in a bit more detail? super interested

It's literally all there is to it. Write your prompt normally. then, at the last paragraph, you write:

Use a workflow to implement this. You (Fable) are the orchestrator, planner and reviewer. Opus agents are implementers.

That's all there's to it. it will create the dynamic workflow - has a nice interface native to Claude Code - and do all the coordination to deliver what you asked.

Re: You only need the frontier model for one single edit

#99

Post by sgc »

wrs wrote: post #87
sgc wrote: post #24

Earlier quoted context omitted.

Actually it does include an important level of review. They say "init a TODO list with a validation step for each item", and then the last several sections of the execution are within the TODO - where it is validating the code. To me that was the heart of the article and of what they did - create a self-validating todo list that deterministically forces the agents to stay on task, and then trim out just the right ext…

It wasn’t clear to me (and still isn’t, after slogging through the inhuman article again) that the verification here is done by a separate agent from the implementation. That’s a second trimming step, but in reverse: at that point you want the new code in the context, but not the implementation reasoning.

No idea, but yes it seems like good practice to trim context there as well. A good candidate for v2 if it's not present.

At the same time, even if that is not done, the todo is both specification and tests. If it passes those when it should not, then either you approved something you should not have, or else there should only be smaller improvements to be made. I like the concept enough and I am not very concerned about this, so I will give it a test on a project I have had on backburner for a bit.

Re: You only need the frontier model for one single edit

#100

Post by gillesjacobs »

_ink_ wrote: post #39

> Below: the share of runs that went poking around the web for it. Filthy cheaters! I can understand that in their benchmark setting they wouldn't want the model to find an existing solution. But in my day to day work, wouldn't I want the model to search online for the best solution? Am I not shooting myself in the foot by preventing it?

In ML, you want to test general capability of a model (generalizability), because you want it to perform well on unseen tasks. In that benchmark, the literal reference is leaking through web search, the agent can see the matching real codebase online and the commits so that's test set leakage. I know no programmer that was ever paid to rewind an existing codebase to a previous commit and implement a feature/fix a bug that exists in the next commits.
Post reply on HN