Really cool direction. For folks thinking about the “agent safety” stack more broadly, this feels complementary to things like Kontext’s kontext-cli (github.com/kontext-dev/kontext-cli) and OneCLI (github.com/onecli/onecli)
Show HN: Forge – Guardrails take an 8B model from 53% to 99% on agentic tasks
181–190 of 292 posts
Re: Show HN: Forge – Guardrails take an 8B model from 53% to 99% on agentic tasks
#182[dead]
Re: Show HN: Forge – Guardrails take an 8B model from 53% to 99% on agentic tasks
#183I like work in this area, and this is really helpful, thanks. I actively avoid cloud based LLMs and mainly use 4b - 30a3b param local models. This means I don't really have a good grasp of SOTA LLM performance or accuracy, but I know what to expect when dealing with local models, and where the pain points are.
I've only skimmed the post and read the abstract and in some places you make a nod to how simple tweaks can make something 10x faster/slower, but then all of your metrics and data seem to focus 100% on accuracy. You need to address speed.
Specifically for agentic workflows and local models, accuracy around function/tool calling hasn't been a problem for me now for about 6 - 12 months, personally, since around QwenCoder3. The main issue is context management and the impact on timing, since agents will often swap prompts and break prompt caching and similar timing improvements.
It looks like your work adds a layers and wrappers like guard rails and retries. This would make my local model experience - specifically for agents - unusable because of the delays it would add.
I really appreciate and respect the work you've done, and apologies if you have already addressed this head on, but with so little talk about the impact on timing here, I feel like you're hiding something or overinflating the actual real world improvements here - what are your thoughts?
It's also mildly concerning me that nobody else has raised this - am I doing something wrong here, or is everyone else just not actually using local models in real life?! Talk to me about your speed experiences!
Re: Show HN: Forge – Guardrails take an 8B model from 53% to 99% on agentic tasks
#184Earlier quoted context omitted.
Do you have any suspicion about what is different between the backends? That's an absolutely bonkers statistic: it would mean spurious differences in hosting container overwhelm the performance differences between models.
I genuinely don't, sadly. I'm a mathematician originally, evolved organically into ML then AI - but I never really was a SWE. I feel like there's some backend decoding or chat template thing going on at a much lower level than what I'm best at. Maybe it's injecting headers or something that eventually compounds to model confusion? I really have no idea. I really hope folks better than me at backend stuff take a look…
Hey, this is most probably related to the chat template or the reasoning parser or the tool call parser or also things like kv cache quantization and possibly other params that affect results like the regular top k top p and all of that, the backend often sets its own defaults or the lack of them. It’s best to have all these under control if possible. I wonder regarding this project have you been testing it on real world projects? I’m working on an agentic loop as well also using a local model.
Re: Show HN: Forge – Guardrails take an 8B model from 53% to 99% on agentic tasks
#185Nice work! I've worked through all kinds of local models, very extensively for a week on an NVidia Spark. Gemma and Qwen, quantized, somewhat shine but the results overall compared to say a Claude Haiku were so disappointing (in context of tool calling) that I ended up returning the hardware. I'm curious how the same local models and benchmarks I have will hold up, will try this.
Re: Show HN: Forge – Guardrails take an 8B model from 53% to 99% on agentic tasks
#186If its so good at coding, why did you use Python aka py-toy? Let me guess…
Re: Show HN: Forge – Guardrails take an 8B model from 53% to 99% on agentic tasks
#187Interesting! The https://swival.dev harness already has retry nudges, step enforcement, error recovery, context awareness, etc. to try to support small models as much as possible. Curious to see how it compares with forge, and if both could be combined.
How does swival.dev compare to a diy agent harness like pi.dev or do they serve different purposes, since swival ships with the "extensions" by default?
Re: Show HN: Forge – Guardrails take an 8B model from 53% to 99% on agentic tasks
#188I like work in this area, and this is really helpful, thanks. I actively avoid cloud based LLMs and mainly use 4b - 30a3b param local models. This means I don't really have a good grasp of SOTA LLM performance or accuracy, but I know what to expect when dealing with local models, and where the pain points are. I've only skimmed the post and read the abstract and in some places you make a nod to how simple tweaks can…
[flagged]
Re: Show HN: Forge – Guardrails take an 8B model from 53% to 99% on agentic tasks
#189So, I experimented a little bit with smaller models and the problem I faced is that it would simply not call a tool that is available, but instead just describe the tool. Is this something that Forge can help with?
Within limits, yes. Forge has escalating nudges that will tell the model effectively "stop responding with text, you MUST call a tool" vibes. If the model is emitting something like "ok, let me call the tool: [valid json tool call in the middle of prose]" then we catch it with rescue parsing. But at the end of the day, if the model keeps responding with text, there's nothing forge can do. I've run into that failure m…
sorry if it's a stupid question, but isn't generating valid json tool call in the middle of prose the way tool calling works? what is that missing?
Re: Show HN: Forge – Guardrails take an 8B model from 53% to 99% on agentic tasks
#190I've been saying for a while that given a proper harness, small local models can perform incredibly well. When you have a system that can try everything, it will eventually get it right as long as you can prevent it from getting it wrong in the meantime.
Prior art: https://ghuntley.com/ralph/