Live data from Hacker News

Show HN: Forge – Guardrails take an 8B model from 53% to 99% on agentic tasks

github.com

51–60 of 292 posts

Re: Show HN: Forge – Guardrails take an 8B model from 53% to 99% on agentic tasks

#51

I've been saying for a while that given a proper harness, small local models can perform incredibly well. When you have a system that can try everything, it will eventually get it right as long as you can prevent it from getting it wrong in the meantime.

Lol, I love that framing. Yeah, the small models have impressed me a lot during this work. The reasoning can be quite good, and definitely sufficient for a lot of cases. Just gotta nudge em back on track Every now and then and they'll figure it out.

Re: Show HN: Forge – Guardrails take an 8B model from 53% to 99% on agentic tasks

#52
post #41

Maybe I am reading it wrong but I don't think this does what it claim it does or at least how it sounds. Basically this is a tool auto-complete that has a workflow element to it with certain steps that need to happen in certain order. In other words the order is defined in advance. Am I correct? Basically execute step 1 first, then step 2 and finally step 3 and this is the schema for each step. That is effectively th…

Partially correct, but an important distinction to call out. You don't have to define the workflow steps. You can just expose the set of tools to the model and let the LLM call whatever it wants in any order, and every guardrail except the prerequisite step enforcement is still there to help. If your workflow does have step enforcement, that can also be conditional. For example like Claude code does read required bef…

The docs should start with that with a very clean explanation how it works. Basically first paragraph. :)

Otherwise you should expect churn.

But also it should really go into some detail how is this different from tool calls with type enforcement on expected parameters.

Re: Show HN: Forge – Guardrails take an 8B model from 53% to 99% on agentic tasks

#53
post #52

Earlier quoted context omitted.

Partially correct, but an important distinction to call out. You don't have to define the workflow steps. You can just expose the set of tools to the model and let the LLM call whatever it wants in any order, and every guardrail except the prerequisite step enforcement is still there to help. If your workflow does have step enforcement, that can also be conditional. For example like Claude code does read required bef…

The docs should start with that with a very clean explanation how it works. Basically first paragraph. :) Otherwise you should expect churn. But also it should really go into some detail how is this different from tool calls with type enforcement on expected parameters.

That's good feedback, thank you! I have an update landing shortly so I'll make sure to clarify in the docs! I appreciate it!

Re: Show HN: Forge – Guardrails take an 8B model from 53% to 99% on agentic tasks

#54

Curious if this would help larger local models? Qwen 3.6 varieties of deepseek4?

Yes it does! I haven't published those evals yet, but I'm actually running 24-35B class models on a custom coding harness built on forge (even 120B class recently). I just need more GPU wall clock time to get more evals done. ETA is...a few weeks? Got distracted by the coding harness. But the results are the same. Reforged models do better than bare, even at those sizes. As for published results, I ran forge on Anthr…

If it's worth it to you, you could try running it on Deepseek v4 flash which is very cheap right now...

Re: Show HN: Forge – Guardrails take an 8B model from 53% to 99% on agentic tasks

#55

I've been saying for a while that given a proper harness, small local models can perform incredibly well. When you have a system that can try everything, it will eventually get it right as long as you can prevent it from getting it wrong in the meantime.

If I understood correctly, the model will get it right because it knows when it isn't right.

Re: Show HN: Forge – Guardrails take an 8B model from 53% to 99% on agentic tasks

#56

I've been saying for a while that given a proper harness, small local models can perform incredibly well. When you have a system that can try everything, it will eventually get it right as long as you can prevent it from getting it wrong in the meantime.

If I understood correctly, the model will get it right because it knows when it isn't right.

Essentially, yes that's right! There's some subtlety in how to let it know it was wrong (returning things as tool errors because it trained on that), but that's the gist of it - sort of a self-correcting architecture.

Re: Show HN: Forge – Guardrails take an 8B model from 53% to 99% on agentic tasks

#57

Curious if this would help larger local models? Qwen 3.6 varieties of deepseek4?

Yes it does! I haven't published those evals yet, but I'm actually running 24-35B class models on a custom coding harness built on forge (even 120B class recently). I just need more GPU wall clock time to get more evals done. ETA is...a few weeks? Got distracted by the coding harness. But the results are the same. Reforged models do better than bare, even at those sizes. As for published results, I ran forge on Anthr…

Exactly what I was thinking - even on frontier or near-frontier models I still see my agents get stuck in these pointless loops where it's very obvious to me what they need to do to get "unstuck".

Re: Show HN: Forge – Guardrails take an 8B model from 53% to 99% on agentic tasks

#58
Something very similar I was experimenting with on, but had different results that you may be interested in, some of my findings were interesting

This was part of testing out how well a tool of mine worked (github.com/jsuppe/loom), which aims to be used to extracts requirements, specs, creates tests. At first I had no intention of using it for code generation but then tried it out with some early success. I tried splitting the work by using the tool with different frontier models, and then providing work to a local ollama instance running one of several models. Not all local models had the same outcome, not all coding languages had the same outcome. I also found in this experiment, when nailing down the coding tasks I wanted to set up positive and negative scenarios- which is where I found setting guardrails can sometimes backfire with inversion- this essentially elaborates on previous work by Khan 2025 (https://arxiv.org/abs/2510.22251); the most interesting finding to me was that if you give guardrails with a rationale, it reduces compliance and may cause the inversion

For coding tasks I found that the improvement was not only ability to use a lower cost model for these broken down tasks, but wall clock time was improved over using frontier model alone, with equivalent outcomes.

Re: Show HN: Forge – Guardrails take an 8B model from 53% to 99% on agentic tasks

#59

I've been saying for a while that given a proper harness, small local models can perform incredibly well. When you have a system that can try everything, it will eventually get it right as long as you can prevent it from getting it wrong in the meantime.

A thousand monkeys on a thousand typewriters…

Re: Show HN: Forge – Guardrails take an 8B model from 53% to 99% on agentic tasks

#60
Why this entire tool chain instead of building within something like pi code?

I've been exploring this area and a project like https://github.com/itayinbarr/little-coder (not my work) lets me mix and match with my current setup or any plugins built for pi.

Post reply on HN