The dashboard github link appears to be broken
Show HN: Forge – Guardrails take an 8B model from 53% to 99% on agentic tasks
111–120 of 292 posts
Re: Show HN: Forge – Guardrails take an 8B model from 53% to 99% on agentic tasks
#112Earlier quoted context omitted.
Yes it does! I haven't published those evals yet, but I'm actually running 24-35B class models on a custom coding harness built on forge (even 120B class recently). I just need more GPU wall clock time to get more evals done. ETA is...a few weeks? Got distracted by the coding harness. But the results are the same. Reforged models do better than bare, even at those sizes. As for published results, I ran forge on Anthr…
>But the results are the same. Reforged models do better than bare, even at those sizes >I haven't published those evals yet Don't forget to post the complete settings for those evals, please, because local LLMs' failure modes are often caused by incorrect setups (bad quants, bad chat templates, non-recommended temperatures, ridiculously small context, not enabling "preserve thinking" etc.). In my setup I've never se…
Good call! The latest forge version has per-model-parameter configs sourced from official sources (can be overridden), that's what I'll use for evals and each eval set will be paired with a commit hash. But I'll make sure to call out the location of the params and maybe highlight some for the popular models.
For the paper - more academic in nature - I wanted to isolate the model performance variable from guardrail lift. The delta is what mattered more than final score. For the paper, everyone got temp=0.7 - that was intentional.
As for Qwen3.6, it's really solid. It'll do really well on forge I can call that now. When I pushed it into agentic coding specifically and the eval suite I use there (separate from forge), even it needed help on long-running tasks - but it's definitely a top model right now.
However, entirely possible there are better settings than the "official recommendations" I found - which would be a neat finding in itself.
Re: Show HN: Forge – Guardrails take an 8B model from 53% to 99% on agentic tasks
#113I've been saying for a while that given a proper harness, small local models can perform incredibly well. When you have a system that can try everything, it will eventually get it right as long as you can prevent it from getting it wrong in the meantime.
A thousand monkeys on a thousand typewriters…
This is a thousand unusually smart monkeys who speak every major human language fluently and are proficient in every major programming language, but sometimes still make bizarre mistakes and need to be put back on track.
Re: Show HN: Forge – Guardrails take an 8B model from 53% to 99% on agentic tasks
#114The dashboard github link appears to be broken
Yeah I'm sorry about that - I thought that link would work. Here is the fixed one (dashboard inside): https://github.com/antoinezambelli/forge/tree/main/docs/resu...
Re: Show HN: Forge – Guardrails take an 8B model from 53% to 99% on agentic tasks
#115Would putting this between a small model and an agent like Hermes improve performance?
I haven't specifically tested this with Hermes, but I would expect so. Hermes is orchestrating things - it decides it needs to...whatever you want, book a trip for you. Forge will help make sure that the API calls to hotel booking sites parse correctly or gracefully retry.
Without forge, I'd guess a small model used for Hermes would have to retry entire workflows when an uncaught exception triggerd when it tried to reply with text when "calling a tool" ("Here is the tool call: [json blob]"). The issue there becomes partial successes can lead to state changes that need to be addressed (it booked the flight already, home it doesn't double-book).
Forge won't help with model reasoning quality though. If it the model thinks the right thing to do is to book 3 buses for your trip, forge doesn't care, it'll just make sure those api calls land.
Re: Show HN: Forge – Guardrails take an 8B model from 53% to 99% on agentic tasks
#116Earlier quoted context omitted.
Forge does have tiered compaction, and it's configurable! Defaults are currently probably a bit on the high side for catching effective attention, but that might be a part of the code that interests you the most. src/forge/context/ - specifically TieredCompact in strategies.py. That's the furthest I took it. The tool-call collapse in particular has been useful in agentic coding, but I haven't formalized/generalized i…
The "model triggers it" pattern is exactly the right shape, but there's a subtle failure mode in it: models are notoriously bad at perceiving their own context pressure. Asking "are you done with that thread?" lands well; asking "would compacting now help you?" doesn't, because the model lacks a reliable internal signal for "I'm starting to skim." You almost have to tie the compaction trigger to task-shape signals (f…
That's a very insightful observation. How could you explain that using the analogy of a pancake breakfast?
Re: Show HN: Forge – Guardrails take an 8B model from 53% to 99% on agentic tasks
#117guardrails this well-designed matter way more than just throwing bigger models at agent tasks tbh
Re: Show HN: Forge – Guardrails take an 8B model from 53% to 99% on agentic tasks
#118Earlier quoted context omitted.
A thousand monkeys on a thousand typewriters…
This is a thousand unusually smart monkeys who speak every major human language fluently and are proficient in every major programming language, but sometimes still make bizarre mistakes and need to be put back on track.
This is fun for you?
Re: Show HN: Forge – Guardrails take an 8B model from 53% to 99% on agentic tasks
#119guardrails this well-designed matter way more than just throwing bigger models at agent tasks tbh
Thank you! I completely agree - especially for always-on systems like agents crawling databases or doing audits and the like. The sheer volume of calls will be enormous and being able to run it on simple hardware with a small model that fits instantly changes the economics of it.
Plus it's cool to see a little 8B model writing code :)
Re: Show HN: Forge – Guardrails take an 8B model from 53% to 99% on agentic tasks
#120Funny timing. I’ve been building something adjacent, though from a different angle: not primarily local-model reliability, but a control layer around agent execution, tools, routing, and operator intent. I was calling these "synthetic models", but decided yesterday "LLM middleware" is a clearer description. Very early prototype, so I’m looking more for architectural/conceptual reactions than polish: https://wardwrigh…
I've just read through your readme and I have zero clue what this does. Something about proxying model calls and applying "policies" to them? But what kind of things does it actually do, what benefits are there? That should be at the top of the readme.
[flagged]