Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Frugon – Find which LLM calls a cheaper model could handle (local, MIT)

github.com

Post reply on HN
Print view View original

21–30 of 32 posts

Re: Show HN: Frugon – Find which LLM calls a cheaper model could handle (local, MIT)

#21

Post by westurner »

Evals and OpenInference (OpenTelemetry) might be useful.

Costed opcodes (like the shelved eWASM opcodes cost chart) would be useful for this model routing problem as well.

Is this the cost to converge problem, the minimize cost to converge upon sufficiently low error problem, or the minimize cost and error problem?

EA methods: mutation, crossover, selection

Gradient descent as a mutation, crossover, and selection pattern; back up when the error/cost stops decreasing for too long and try a different branch.

A simple experiment: vary only a nonce in the prompt and compare output value. The nonce is a parameter. The model is a hyperparameter.

Re: Show HN: Frugon – Find which LLM calls a cheaper model could handle (local, MIT)

#23

Post by jarodrh »

westurner wrote: post #21

Evals and OpenInference (OpenTelemetry) might be useful. Costed opcodes (like the shelved eWASM opcodes cost chart) would be useful for this model routing problem as well. Is this the cost to converge problem , the minimize cost to converge upon sufficiently low error problem , or the minimize cost and error problem ? EA methods: mutation, crossover, selection Gradient descent as a mutation, crossover, and selection…

Very interesting thoughts. This is a whole discussion on its own.

The ones that I mulled over that I feel have legs in my line of thinking are:

> minimize cost to converge upon sufficiently low error problem

- This is the one. Frugon does an easy/hard split to keep quality within a certain tolerance, and find the lowest costed model within that constraint. The judge handles the "sufficiently low error" side of that via sampled prompt logs.

> Evals and OpenInference (OpenTelemetry) might be useful.

- Evals already a given. This is frugon's wheelhouse, however it currently only supports OpenAI-style jsonl. I'll definitely consider adding support for OpenInference.

> A simple experiment: vary only a nonce in the prompt and compare output value. The nonce is a parameter. The model is a hyperparameter.

- This is an intriguing idea. This would actually enhance/strengthen frugon's stance on the "minimize cost to converge upon sufficiently low error problem". Noted.

Re: Show HN: Frugon – Find which LLM calls a cheaper model could handle (local, MIT)

#25

Post by westurner »

jarodrh wrote: post #23
westurner wrote: post #21

Evals and OpenInference (OpenTelemetry) might be useful. Costed opcodes (like the shelved eWASM opcodes cost chart) would be useful for this model routing problem as well. Is this the cost to converge problem , the minimize cost to converge upon sufficiently low error problem , or the minimize cost and error problem ? EA methods: mutation, crossover, selection Gradient descent as a mutation, crossover, and selection…

Very interesting thoughts. This is a whole discussion on its own. The ones that I mulled over that I feel have legs in my line of thinking are: > minimize cost to converge upon sufficiently low error problem - This is the one. Frugon does an easy/hard split to keep quality within a certain tolerance, and find the lowest costed model within that constraint. The judge handles the "sufficiently low error" side of that v…

> minimize cost to converge upon sufficiently low error problem

Is this a convex optimization or non-convex optimization problem?

> Evals

Re: pytest-evals, mcpbr, agentevals, foundry-toolkit; and devtools-mcp; https://news.ycombinator.com/item?id=48532642

OpenInference and OpenTelemetry have a schema for agent sessions.

I finally discovered agentsview and ctxrs/ctx for indexing and searching agent session logs. Perhaps such an index is also useful for agent optimization

> Nonce

Accuracy is sensitive to nonce and temperature without variance in the model hyperparameters

Re: Show HN: Frugon – Find which LLM calls a cheaper model could handle (local, MIT)

#26

Post by jarodrh »

westurner wrote: post #25
jarodrh wrote: post #23

Earlier quoted context omitted.

Very interesting thoughts. This is a whole discussion on its own. The ones that I mulled over that I feel have legs in my line of thinking are: > minimize cost to converge upon sufficiently low error problem - This is the one. Frugon does an easy/hard split to keep quality within a certain tolerance, and find the lowest costed model within that constraint. The judge handles the "sufficiently low error" side of that v…

> minimize cost to converge upon sufficiently low error problem Is this a convex optimization or non-convex optimization problem? > Evals Re: pytest-evals, mcpbr, agentevals, foundry-toolkit; and devtools-mcp ; https://news.ycombinator.com/item?id=48532642 OpenInference and OpenTelemetry have a schema for agent sessions. I finally discovered agentsview and ctxrs/ctx for indexing and searching agent session logs. Perh…

Down the rabbit hole we go.

> Is this a convex optimization or non-convex optimization problem?

- Neither, frugon sidesteps that issue by having a small finite candidate selection that it evaluates completely. No search needed because the problem is discrete and small.

> OpenInference and OpenTelemetry have a schema for agent sessions.

- Good to know for when I look into it.

> Accuracy is sensitive to nonce and temperature without variance in the model hyperparameters

- Yes, this is a current weakness in frugon...Hmm, I see, so I'd essentially need to establish a baseline (with whatever the default temp is) + nonce, then compare both spreads. Noted.

Good thread. Give frugon a try on your own logs, and tell me what you think.

Re: Show HN: Frugon – Find which LLM calls a cheaper model could handle (local, MIT)

#27

Post by westurner »

jarodrh wrote: post #26
westurner wrote: post #25

Earlier quoted context omitted.

> minimize cost to converge upon sufficiently low error problem Is this a convex optimization or non-convex optimization problem? > Evals Re: pytest-evals, mcpbr, agentevals, foundry-toolkit; and devtools-mcp ; https://news.ycombinator.com/item?id=48532642 OpenInference and OpenTelemetry have a schema for agent sessions. I finally discovered agentsview and ctxrs/ctx for indexing and searching agent session logs. Perh…

Down the rabbit hole we go. > Is this a convex optimization or non-convex optimization problem? - Neither, frugon sidesteps that issue by having a small finite candidate selection that it evaluates completely. No search needed because the problem is discrete and small. > OpenInference and OpenTelemetry have a schema for agent sessions. - Good to know for when I look into it. > Accuracy is sensitive to nonce and tempe…

As well. Someday costed opcodes and sharded redundancy for all of this;

> No search needed because the problem is discrete and small.

If an algorithm selects according to rank, there's a cost/score/fitness/survival/error function (that assumes or imposes a distance metric to make a metric space).

Awhile back I had to match exterior paint to printed paint swatches and I'm stereotypically not good at colors, so finally I arrived at just A/B; this one or this one and discard.

Re: Show HN: Frugon – Find which LLM calls a cheaper model could handle (local, MIT)

#28

Post by jarodrh »

westurner wrote: post #27
jarodrh wrote: post #26

Earlier quoted context omitted.

Down the rabbit hole we go. > Is this a convex optimization or non-convex optimization problem? - Neither, frugon sidesteps that issue by having a small finite candidate selection that it evaluates completely. No search needed because the problem is discrete and small. > OpenInference and OpenTelemetry have a schema for agent sessions. - Good to know for when I look into it. > Accuracy is sensitive to nonce and tempe…

As well. Someday costed opcodes and sharded redundancy for all of this; > No search needed because the problem is discrete and small. If an algorithm selects according to rank, there's a cost/score/fitness/survival/error function (that assumes or imposes a distance metric to make a metric space). Awhile back I had to match exterior paint to printed paint swatches and I'm stereotypically not good at colors, so finally…

> As well. Someday costed opcodes and sharded redundancy for all of this;

Hmm, noted.

> If an algorithm selects according to rank, there's a cost/score/fitness/survival/error function.

- Yes, what I'm saying is that frugon has a small enough candidate pool that it can all be evaluated at the same time. It doesn't need to search through it.

> Awhile back I had to match exterior paint to printed paint swatches and I'm stereotypically not good at colors, so finally I arrived at just A/B; this one or this one and discard.

- This is exactly the shape the judge prompt takes in frugon.

See: src/frugon/measure.py

Then search: JUDGE_PROMPT_TEMPLATE

Re: Show HN: Frugon – Find which LLM calls a cheaper model could handle (local, MIT)

#30

Post by jarodrh »

mujia wrote: post #29

A cheaper model can look fine per request, but one weak answer may create another call or a human review step. That seems like the hardest cost to capture.

True, but human review isn't visible in the logs, hmm, however I suppose retry patterns can, but frugon doesn't currently track this, so we can't quantify this yet.

Frugon analyzes your existing logs offline on cost and quality tier. The "looks fine per request" is exactly the reason why --judge exists.

Earlier, cyanydeez inspired the consideration of a metric "effective cost per judged success", which could also answer this point.

Try it on your logs, and tell me if anything falls through the cracks.

Post reply on HN