Show HN: Forge – Guardrails take an 8B model from 53% to 99% on agentic tasks

This is a neat project, but the description made me realize that I don't actually know what the term "guardrails" means.

... which lead me to realize that it's one of those terms with multiple meanings - like "agent" or even "AI" itself - but where people who use it may not be aware of how many different definitions are floating around.

In this project it refers to validating tool calls - fixing invalid tool responses, making sure certain required tool calls have been made, maintaining an error budget after which the task is abandoned with an error.

Other projects might use "guardrails" to mean protecting against unsafe content (Llama Gaurd), refusing off-topic queries (NVIDIA NeMo Guardrails "topical rails", filtering PII, detecting jailbreaks, or human-in-the-loop checks of specific actions.

I've even seen people talk about running a coding agent in a sandbox (Docker, Firecracker etc) as a form of guardrail.

That's a fair point, and frankly something that might not age well in my docs one day. I genuinely don't know what the industry will standardize on when it comes to the use of the term "guardrails". I've seen the sec definitions as well.

You're 100% right about how I meant it and what it means within Forge though, but it's something that might lead to doc changes as things evolve.

Had a couple thoughts in this realm, and am working them into my own harness. Curious to see what others think. I'm not sure if this is generalizable, as my harness is fairly specialized:

- Breaking down a problem into a planned execution, with executing agent providing the initial plan which includes explicit objectives such as which tools it calls and what it would consider to be a successful execution.

- The harness then executes the plan in order

- Each step that involves a tool call will be executed by breaking down the tool call into component parts: the harness interrogates the agent for a valid parameter value for the current tool argument. The tool definition contains validators for each argument. If the validator fails, the harness rewinds the conversation and injects the failure reason into the next try.

- Once the agent produces a valid response for the argument, the harness proceeds to the next argument.

- Once all the arguments have been filled, the harness calls the tool. It passes the agent's initial expected value along with the actual value, along with any errors that may have been produced and asks the agent if it is satisfied with the result. If it isn't, the agent provides a reason and the harness then retries the tool call process from the beginning rewinding the conversation and inserting the reasoning for the retry.

- The agent may request to re-plan if it discovers a flaw in its initial plan. The harness will also attempt to re-plan if the agent produces too many failures in a row.

This proves to be quite effective at reducing tool call failures. One benefit is that the sub-agent gets a perfect conversation history where it makes no mistakes. I'm not sure if it's actually better at completing tasks though, I haven't tried to benchmark it.

I've just read through your readme and I have zero clue what this does. Something about proxying model calls and applying "policies" to them? But what kind of things does it actually do, what benefits are there? That should be at the top of the readme.

Do you have any suspicion about what is different between the backends?

That's an absolutely bonkers statistic: it would mean spurious differences in hosting container overwhelm the performance differences between models.