Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Forge – Guardrails take an 8B model from 53% to 99% on agentic tasks

github.com

Post reply on HN
Print view View original

141–150 of 292 posts

Re: Show HN: Forge – Guardrails take an 8B model from 53% to 99% on agentic tasks

#141

Post by zambelli »

DeathArrow wrote: post #135

I'm curious if in proxy mode it works also with remote models or only with local models. Also, did someone tried it with local Qwen 3.6?

I believe there's a comment below mentioning "qwen" but not a specific version number - if you're looking for 3rd party validation. I've personally tried qwen3.6-35b-a3b, qwen3.5-35b-a3b, and qwen3.5-27b with forge (agentic coding harness built on forge workflowrunner) and it works great. Official forge eval benchmarks for that class of models is still a couple of weeks out.

Proxy mode should work fine with remote models, the only constraint is the compatible endpoint - which is standard anyways. I don't think you'd have any issue hitting either a remote gateway like liteLLM or just claude API.

Re: Show HN: Forge – Guardrails take an 8B model from 53% to 99% on agentic tasks

#142

Post by zambelli »

Karuma wrote: post #125
snovv_crash wrote: post #22

I get a strong LLM smell in your description. If you couldn't bother to write it, why should I bother to read it?

Thank you for mentioning it. Too bad you got downvoted to hell as usual when anybody dares to do it. The original post and every comment by OP is so full of AI slop ("the biggest surprise!", "one thing I didn't expect!", "the biggest challenge!", etc. etc.") that is absolutely painful to read. I still can't believe most people (especially here on HN, I thought we were a bit better than this) can't notice all this stu…

I'm sorry you feel that way about my posts - hopefully you still find the work valuable. Still human here btw, and still 100% honest.

Re: Show HN: Forge – Guardrails take an 8B model from 53% to 99% on agentic tasks

#143

Post by zambelli »

yieldcrv wrote: post #130

impressive, we can get high tokens/s with 8B param models and doubling it with MTP

Yeah, throughput on small models can get really fun :). As for MTP, should work fine since forge just sits between model and consumer. As long as MTP didn't change the model endpoint contract (ie, you call llama.cpp the same way you would normally) then it should work out of the box. But I haven't tested MTP myself yet (or that commit of llama.cpp).

Re: Show HN: Forge – Guardrails take an 8B model from 53% to 99% on agentic tasks

#144

Post by zambelli »

deevus wrote: post #140

This is fantastic. I haven't got any local inference as I can't afford it right now, but tool calling has been a concern for me with these smaller models through OpenRouter. I've been working on a pytest-first acceptance testing framework called Dokimasia (do-kee-ma-see-ah) that I'd love to get your thoughts on: https://github.com/deevus/dokimasia Acceptance testing might not be what you need for Forge, but since you…

Oh, interesting idea. Formalizing an abstraction layer for testing all the integration types out there in the AI ether, essentially? MCP, skills, etc.

I think this sits a level higher than Forge - maybe testing the workflow proper and integration points that it might surface (if some tools are giving access to an MCP or something).

Could likely layer both together without much trouble.

Only thing I'd be curious about is how you handle the non-deterministic nature of these models. Sometimes they get the tool call right, sometimes they barf bad json. Does the suite run multiple trials?

Re: Show HN: Forge – Guardrails take an 8B model from 53% to 99% on agentic tasks

#145

Post by Aurornis »

Escapade5160 wrote: post #49

I've been saying for a while that given a proper harness, small local models can perform incredibly well. When you have a system that can try everything, it will eventually get it right as long as you can prevent it from getting it wrong in the meantime.

The problem is that you get similar quality as if you gave a junior unlimited time to work on a problem and told them to keep trying different things until the goal is reached.

Even the SOTA models have this problem when the work is complicated enough. The problem is amplified more with the small models.

Re: Show HN: Forge – Guardrails take an 8B model from 53% to 99% on agentic tasks

#146

Post by bratbag »

jplusequalt wrote: post #118
DiogenesKynikos wrote: post #113

Earlier quoted context omitted.

This is a thousand unusually smart monkeys who speak every major human language fluently and are proficient in every major programming language, but sometimes still make bizarre mistakes and need to be put back on track.

This is fun for you?

I found it fun to read.

Re: Show HN: Forge – Guardrails take an 8B model from 53% to 99% on agentic tasks

#147

Post by zambelli »

beacon294 wrote: post #138
zambelli wrote: post #70

Earlier quoted context omitted.

That is the whole challenge, actually! A new metric I'm going to dogfood into forge is ETTWS - estimated time to working solution. A simple retry loop around your whole workflow could, in some cases, be all you need . But it could mean many blind attempts to get through a workflow successfully. And hopefully there isn't a payment step partway through! The fewer hard errors nix the whole workflow, the lower your ETTWS…

Have you read the MAKER/MDAP paper? 1 million sequential tasks.

No, I haven't - hadn't heard of it. I'll try to squeeze in a quick read in the coming weeks!

Re: Show HN: Forge – Guardrails take an 8B model from 53% to 99% on agentic tasks

#148

Post by jeffreygoesto »

azurewraith wrote: post #37

Interestingly enough we have found the same net result -- structural guardrails are the unlock for smaller models. Our approach in particular layers three things: a parse rescue for malformed/incorrect tool calls (similar to your retry nudges), content-level intervention (diff size rejection, checkpoint forcing) and state machine enforcement on top (per-phase tool restriction, transition guards). On 13B models we saw…

Reads like processes guarding mediocre teams into higher probability of success? I can hear Alanis Morissette in my head now somehow...

Re: Show HN: Forge – Guardrails take an 8B model from 53% to 99% on agentic tasks

#149

Post by deevus »

zambelli wrote: post #144
deevus wrote: post #140

This is fantastic. I haven't got any local inference as I can't afford it right now, but tool calling has been a concern for me with these smaller models through OpenRouter. I've been working on a pytest-first acceptance testing framework called Dokimasia (do-kee-ma-see-ah) that I'd love to get your thoughts on: https://github.com/deevus/dokimasia Acceptance testing might not be what you need for Forge, but since you…

Oh, interesting idea. Formalizing an abstraction layer for testing all the integration types out there in the AI ether, essentially? MCP, skills, etc. I think this sits a level higher than Forge - maybe testing the workflow proper and integration points that it might surface (if some tools are giving access to an MCP or something). Could likely layer both together without much trouble. Only thing I'd be curious about…

[dead]
Post reply on HN