[flagged]
Show HN: Frugon – Find which LLM calls a cheaper model could handle (local, MIT)
11–20 of 32 posts
Re: Show HN: Frugon – Find which LLM calls a cheaper model could handle (local, MIT)
#12this would be more interesting as a local LLM anlysis; throw out all the costs, and figure out primary-subagent model architecture, and maximize token generation and prefill. I don't see how anyone can operationalize this information.
That's an interesting thought, and one I'll take note of, but that would be a different tool. However, if you look hard enough I'd say we're tackling the same issue. I'm just choosing to look at the problem from a cost perspective as opposed to a raw token generation/prefill perspective. The mindset can be applied to both sides, but Frugon prices the cloud side. The end goal is essentially the same and your mind went…
yes, but $ is operationally useless since $ is model depenedent and as other articles currently on HN show, the model+harness are symbiotic or antagonistic; if you strip out the $ part of it, you can focus on the interdependence of Agent+subagent and you can evaluate that in a stricter sense because I would want to select the model+subagent that improves my outcomes instead of whicheere is cheapest. If you tell me this model has a 99% chance of succeeding at $10 and this one has a 50% chan
Re: Show HN: Frugon – Find which LLM calls a cheaper model could handle (local, MIT)
#13Earlier quoted context omitted.
That's an interesting thought, and one I'll take note of, but that would be a different tool. However, if you look hard enough I'd say we're tackling the same issue. I'm just choosing to look at the problem from a cost perspective as opposed to a raw token generation/prefill perspective. The mindset can be applied to both sides, but Frugon prices the cloud side. The end goal is essentially the same and your mind went…
yes, but $ is operationally useless since $ is model depenedent and as other articles currently on HN show, the model+harness are symbiotic or antagonistic; if you strip out the $ part of it, you can focus on the interdependence of Agent+subagent and you can evaluate that in a stricter sense because I would want to select the model+subagent that improves my outcomes instead of whicheere is cheapest. If you tell me th…
You got cut off at the end, but I hear your point.
I completely agree that cost-per-successful-outcome is the goal, and "cheapest" is never the entire solution, but only routing to the cheapest isn't what frugon does. If it did that, then it would just pick something like GPT-4.1-Nano (one of the current cheapest models regardless of quality) for everything, and it would be useless.
The whole pipeline in frugon is predicated on quality tier gating. Frugon doesn't strip out quality. This is based on quality tiers from LMArena leaderboard data (CC-BY licensed). The easy/hard split only recommends routing calls predicted to survive the downgrade.
> $ is operationally useless since $ is model depenedent
> model+harness are symbiotic or antagonistic
To these points, frugon gives you --measure (you can compare the outcomes yourself via prompt sampling from both models - current and candidate/recommended) and --judge (championing a model to determine whether the recommended/candidate model actually successfully holds the quality of the current model from the prompts sampled). The win/tie rate summary outputted by the judge tells you whether the candidates you're comparing preserve the outcome you're currently getting from the current model measured on your own prompts. So, if your model succeeds at the prompt and the candidate ties this (judge's verdict), then you've successfully evaluated that you can swap to a different model at a lower cost. It's measured on your task distribution as a quality confirmation of frugon's initial offline quality tier gating routing recommendation.
Success rates are task-dependent, as every user prompts a model differently which could either get a successful outcome or a failure-doomed-to-retry outcome. No benchmark/table can tell you what your agent+subagent's odds of success are, unless all you do is pass an unaltered prompt that has been externally benchmarked. Hence why --measure and --judge exist. You must confirm the initial routing recommendation with your own evaluation (human/ai/evals)
However, you have made me think of combining the initial dollar cost analysis with judge outcome as an "effective cost per judged success" metric...which is a great idea, as they are currently separated.
Re: Show HN: Frugon – Find which LLM calls a cheaper model could handle (local, MIT)
#14Useful but how it compares with other providers model
Re: Show HN: Frugon – Find which LLM calls a cheaper model could handle (local, MIT)
#15Useful but how it compares with other providers model
Thanks. I'm not sure if you're asking "how does frugon compare provider models" or "can it compare across provider models" or even "how does frugon compare to other tools"...
On the first: It does this utilizing pricing from LiteLLM registry, and quality tiers from LMArena.
On the second: A log of calls for OpenAI can get recommendations for Gemini or Anthropic or DeepSeek etc. and vice versa.
On the third: It's free, local and offline. Hosted tools normally require you to plug them into your live traffic, whereas frugon reads logs you already have. And you have the benefit of exploring the source code.
Re: Show HN: Frugon – Find which LLM calls a cheaper model could handle (local, MIT)
#16[flagged]
Re: Show HN: Frugon – Find which LLM calls a cheaper model could handle (local, MIT)
#17I think this is great, and the next frontier is to analyze how well calls can be handled by a local model. Realistically, to do that usefully requires response time as a new dimension of judging: Can a local model provide an acceptably accurate response in an acceptable amount of time?
Re: Show HN: Frugon – Find which LLM calls a cheaper model could handle (local, MIT)
#18I think this is great, and the next frontier is to analyze how well calls can be handled by a local model. Realistically, to do that usefully requires response time as a new dimension of judging: Can a local model provide an acceptably accurate response in an acceptable amount of time?
Thanks. Yes, local models are gaining a lot of traction.
The measure/judge step uses LiteLLM, so it does sample local models. I just tested "--candidates ollama/llama3.2:1b", and that works - ignoring the lack of rich UX for local/unpriced models, as I was focussing on cloud cost, but you've inspired me to give this area some polish.
Noted: "response time as a new dimension of judging" - Added to the roadmap.
Try it and let me know if you hit a wall.
Re: Show HN: Frugon – Find which LLM calls a cheaper model could handle (local, MIT)
#19What we need is an AI gateway/router that will actually first analyze the input tokens and then decide what model to use. If it's so simple that a dirt cheap qwen 3.5 flash or whatever will be fine, then it chooses that. If it deems we need GPT 5.6, then it uses that, etc. does anything like this already exist?
Re: Show HN: Frugon – Find which LLM calls a cheaper model could handle (local, MIT)
#20What we need is an AI gateway/router that will actually first analyze the input tokens and then decide what model to use. If it's so simple that a dirt cheap qwen 3.5 flash or whatever will be fine, then it chooses that. If it deems we need GPT 5.6, then it uses that, etc. does anything like this already exist?
Yes they do, it's quite the popular topic atm.
RouteLLM (OSS - frugon's savings bands are based on their research), OpenRouter's auto mode; I actually commented on a /show post not long ago: Wayfinder Router (neat project) - https://news.ycombinator.com/item?id=48704373 and more.
My stance on the router point is a config-led (deterministic) one, even if the eventual industry consensus is a hybrid of dynamic (AI-led) + deterministic. Both sides still require evals/evidence for policy creation, which is what frugon provides.