That's a huge gap for llama.cpp server - any idea why?
Show HN: Forge – Guardrails take an 8B model from 53% to 99% on agentic tasks
81–90 of 292 posts
Re: Show HN: Forge – Guardrails take an 8B model from 53% to 99% on agentic tasks
#82Earlier quoted context omitted.
That's where frontier pulls ahead for sure, at least on the big frontier models - though I haven't formalized those findings because...time. Necessary disclaimer, forge isn't concerned, technically, with model quality, just execution of tool calls. Now for the actual answer... What I found to be the limiting factor with small models in the 14B range was "effective attention". Beyond a certain point, still well within…
The "effective attention" framing nails what I keep noticing too. Sonnet's official context is huge in principle, but in a real coding session where the agent is reading 30+ files, running grep, processing test output, emitting diffs — somewhere around 60-80k effective tokens I can feel it start to "skim" earlier context rather than reason over it. The thing it forgot isn't out of window; it's just not weighted highl…
>The tool-call history collapse is a problem I'd pay real money to have solved cleanly.
It's general attention collapse and it happens everywhere once you start noticing it.
The simplest example, which even frontier models fail at, is something of the form `A and not B', which they keep insisting means `A and B' after the text gets pushed far enough back in the context.
The only solution, I think, that is even theoretically capable of fixing this is using a different form of attention. One which innately understands tree-like structures and binds tree nodes close together regardless of overall distance from the end of the stream.
Incidentally this is what I'm also working on at $job.
Re: Show HN: Forge – Guardrails take an 8B model from 53% to 99% on agentic tasks
#83Do you think a similar approach would work with smaller models, like 1.5B models?
Re: Show HN: Forge – Guardrails take an 8B model from 53% to 99% on agentic tasks
#84> One thing I really didn't expect: the serving backend matters. Same Mistral-Nemo 12B weights produce 7% accuracy on llama-server with native function calling and 83% on Llamafile in prompt mode. I thought Llamafile was just a model and llama.cpp bundled in to a single binary - is this the difference between Llamafile injecting a default sysmtem prompt vs hitting the raw llama-server endpoint with no harness? That s…
I was surprised as well. I did go with an extreme (but true) example in the post. In this case, native function-calling template likely is in play.
However, that doesn't explain the Lamaserver prompt vs llamafile at ~ +4pts, or vs Ollama (at ~ +30ish pts) that sits almost perfectly between llamaserver native and llamafile.
The backend affects almost all model families, and was just something I've never seen really talked about.
Re: Show HN: Forge – Guardrails take an 8B model from 53% to 99% on agentic tasks
#85That's a huge gap for llama.cpp server - any idea why?
Best guess is it's native mode. The function calling template is just broken for Nemo.
I did go with an extreme example in the post (but true). Other deltas are smaller but still statistically significant. 30 pt swing between llamserver prompt vs ollama, 4-5pt swing between llamafile and llamaserver prompt.
Re: Show HN: Forge – Guardrails take an 8B model from 53% to 99% on agentic tasks
#86[flagged]
AI slop
Re: Show HN: Forge – Guardrails take an 8B model from 53% to 99% on agentic tasks
#87Do you think a similar approach would work with smaller models, like 1.5B models?
I would expect so! I'm currently running Gemma 4 E4B evals and it's behaving the same. Better with guardrails. There might be a floor where any error nudge confuses the model more than helps, but I haven't found it across many 8B families and now Gemma 4 E4B.
Re: Show HN: Forge – Guardrails take an 8B model from 53% to 99% on agentic tasks
#88[flagged]
I almost said "it's jarring to see a human speaking fluent claude" but then I realized you're just a spambot.
Re: Show HN: Forge – Guardrails take an 8B model from 53% to 99% on agentic tasks
#89I've been saying for a while that given a proper harness, small local models can perform incredibly well. When you have a system that can try everything, it will eventually get it right as long as you can prevent it from getting it wrong in the meantime.
If I understood correctly, the model will get it right because it knows when it isn't right.
Re: Show HN: Forge – Guardrails take an 8B model from 53% to 99% on agentic tasks
#90Interesting! The https://swival.dev harness already has retry nudges, step enforcement, error recovery, context awareness, etc. to try to support small models as much as possible. Curious to see how it compares with forge, and if both could be combined.
Oh interesting - I hadn't come across that!
I'd assume they could be combined. A coding harness would own the agentic workflow by nature, forge guardrails would help tool calling.
I haven't given it a thorough read yet but I think their guardrails might be more focused on the workflow level. They are doing error capture at tool level with warnings to the model, but I'd need to dig deeper. On the surface definitely the same design philosophy! Maybe Forge makes error nudges more of a first-class citizen?
Our compaction strategies might be the most similar of all the pieces. Cool find!