Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We decreased our LLM costs with Opus

mendral.com

Post reply on HN
Print view View original

31–40 of 41 posts

Re: We decreased our LLM costs with Opus

#31

Post by koenvdb »

albert_e wrote: post #20

I want to create a "harness" that does this with Claude Code and other expensive agents. Buffer user prompts, use conversation history and repo state as context -- and run a local model or a cheap and fast cloud model like Haiku to determine the optimal way to address the user's ask, reframe the query with better context (user reviews and approves if needed) and THEN let expensive models like Opus have a go at it. If…

> Currently skills cannot be tagged to different models. Ideally we should be able to say -- for trivial tasks, the skill should always use Haiku even if invoked from a session with Opus xhigh.

You can set the model for a skill. You just set model: haiku at the top and it will use haiku! You can even set the effort level, look for “Frontmatter reference” in this doc article: https://code.claude.com/docs/en/skills

Re: We decreased our LLM costs with Opus

#32

Post by albert_e »

koenvdb wrote: post #31
albert_e wrote: post #20

I want to create a "harness" that does this with Claude Code and other expensive agents. Buffer user prompts, use conversation history and repo state as context -- and run a local model or a cheap and fast cloud model like Haiku to determine the optimal way to address the user's ask, reframe the query with better context (user reviews and approves if needed) and THEN let expensive models like Opus have a go at it. If…

> Currently skills cannot be tagged to different models. Ideally we should be able to say -- for trivial tasks, the skill should always use Haiku even if invoked from a session with Opus xhigh. You can set the model for a skill. You just set model: haiku at the top and it will use haiku! You can even set the effort level, look for “Frontmatter reference” in this doc article: https://code.claude.com/docs/en/skills

Interesting - thanks!

Not sure when this was added.

I found "open" feature requests in GitHub asking for this exact thing

Re: We decreased our LLM costs with Opus

#33

Post by kovek »

locknitpicker wrote: post #30
shad42 wrote: post #14

Earlier quoted context omitted.

I am one of Mendral co-founder (my co-founder wrote the article), I am the one to blame for changing the title when posting. I thought our original one was too clickbait and I wanted to better summarize with this title. Despite the original title, a lot of what we learned comes to how Opus evolved and the ability to reason. And also the fact that Haiku is quite capable if scoped properly, that's the whole purpose of…

> Despite the original title, a lot of what we learned comes to how Opus evolved and the ability to reason. And also the fact that Haiku is quite capable if scoped properly, that's the whole purpose of the article. I think you're misrepresenting the whole thing. The blog post boils down to introducing a specialized triage step which is then offloaded to a cheap model. The cost savings come from skipping the expensive…

Does thinking about how to offload matter?

Re: We decreased our LLM costs with Opus

#34

Post by alex1sa »

vanviegen wrote: post #29

> It's the same reason you don't want to lead a debugging session by saying "I think the problem is in this file": you've biased the investigation before it started. Unless you're evaluating the agent/person doing the debug session, why would you not provide them with some relevant insight about the problem you have? Given that you're pretty sure about it, of course.

[dead]

Re: We decreased our LLM costs with Opus

#35

Post by prashant3210 »

whalesalad wrote: post #5

Looking at the diagram, is this seriously a case of replacing basic functional concepts like "write to clickhouse" or "have we seen this before" to a model? could those be actual function calls in some language? just seems wasteful all around. having an agent in the critical path when a regular expression (or similar) could do just seems odd. yeah haiku is cheap but re.match() is cheaper.

[dead]

Re: We decreased our LLM costs with Opus

#36

Post by locknitpicker »

kovek wrote: post #33
locknitpicker wrote: post #30

Earlier quoted context omitted.

> Despite the original title, a lot of what we learned comes to how Opus evolved and the ability to reason. And also the fact that Haiku is quite capable if scoped properly, that's the whole purpose of the article. I think you're misrepresenting the whole thing. The blog post boils down to introducing a specialized triage step which is then offloaded to a cheap model. The cost savings come from skipping the expensive…

Does thinking about how to offload matter?

A discussion on how to avoid paying the price of running an expensive model is not about the expensive model. You can triage things running a cheap model with Ollama. Heck, throw in gpt4.1 which is free.

Re: We decreased our LLM costs with Opus

#37

Post by dmitry_dv »

koenvdb wrote: post #31
albert_e wrote: post #20

I want to create a "harness" that does this with Claude Code and other expensive agents. Buffer user prompts, use conversation history and repo state as context -- and run a local model or a cheap and fast cloud model like Haiku to determine the optimal way to address the user's ask, reframe the query with better context (user reviews and approves if needed) and THEN let expensive models like Opus have a go at it. If…

> Currently skills cannot be tagged to different models. Ideally we should be able to say -- for trivial tasks, the skill should always use Haiku even if invoked from a session with Opus xhigh. You can set the model for a skill. You just set model: haiku at the top and it will use haiku! You can even set the effort level, look for “Frontmatter reference” in this doc article: https://code.claude.com/docs/en/skills

Same works for subagents — .claude/agents/triager.md with model: haiku plus a Task call from the main loop. The reason to roll your own was the sandbox, not the routing.

Re: We decreased our LLM costs with Opus

#39

Post by tuo-lei »

shad42 wrote: post #17
syntaxing wrote: post #11

Is RAG dead? I would be very surprised a local small SOTA embedded model like llama-embed-nemotron-8b doesnt outperform the Haiku layer for this application. Should be pretty cheap and easy to prove out. With 32K context size, you can literally one shot the whole ticket.

IMO RAG is mostly dead. The game changer with newer models like Opus is the reasoning. So instead of pushing all the context up front (RAG style), it's better to give strong primitives (eg. bash, SQL) and let the agent figure it out. It's what Claude Code is doing now and the principles we applied for Mendral as well. That said, you're right that some smaller models can outperform Haiku and we're thinking supporting…

[dead]

Re: We decreased our LLM costs with Opus

#40

Post by Salgat »

shad42 wrote: post #17
syntaxing wrote: post #11

Is RAG dead? I would be very surprised a local small SOTA embedded model like llama-embed-nemotron-8b doesnt outperform the Haiku layer for this application. Should be pretty cheap and easy to prove out. With 32K context size, you can literally one shot the whole ticket.

IMO RAG is mostly dead. The game changer with newer models like Opus is the reasoning. So instead of pushing all the context up front (RAG style), it's better to give strong primitives (eg. bash, SQL) and let the agent figure it out. It's what Claude Code is doing now and the principles we applied for Mendral as well. That said, you're right that some smaller models can outperform Haiku and we're thinking supporting…

It's more accurate to say that RAG is alive and well and is just incorporated into the agent's responsibility, it's just one more tool that it can call on instead of the user manually doing it.
Post reply on HN