Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Forge – Guardrails take an 8B model from 53% to 99% on agentic tasks

github.com

Post reply on HN
Print view View original

61–70 of 292 posts

Re: Show HN: Forge – Guardrails take an 8B model from 53% to 99% on agentic tasks

#61

Post by bglusman »

bglusman wrote: post #45

Funny timing. I’ve been building something adjacent, though from a different angle: not primarily local-model reliability, but a control layer around agent execution, tools, routing, and operator intent. I was calling these "synthetic models", but decided yesterday "LLM middleware" is a clearer description. Very early prototype, so I’m looking more for architectural/conceptual reactions than polish: https://wardwrigh…

Ironically, the project this idea emerged out of for me is also called Forge, actually Calciforge… https://calciforge.org / https://github.com/bglusman/calciforge

Name was just a portmanteau of Calcifer's forge, because Howl’s moving castle seemed like a good metaphor for what I was trying to do… I had synthetic models as apiece there but I realized a) it was out of place and b) it was my favorite feature there

Re: Show HN: Forge – Guardrails take an 8B model from 53% to 99% on agentic tasks

#62

Post by user3939382 »

jf wrote: post #11

Tangentially related: Since you are at Texas Instruments, I wonder if you could find out what the status is of the intellectual property for the TI Explorer lisp machines. I know who owns the IP for Genera, but wasn’t able to find out about TI’s lisp OS

Who owns the IP for Genera?

Re: Show HN: Forge – Guardrails take an 8B model from 53% to 99% on agentic tasks

#64

Post by zambelli »

88j88 wrote: post #58

Something very similar I was experimenting with on, but had different results that you may be interested in, some of my findings were interesting This was part of testing out how well a tool of mine worked (github.com/jsuppe/loom), which aims to be used to extracts requirements, specs, creates tests. At first I had no intention of using it for code generation but then tried it out with some early success. I tried spl…

I've had a few reversions as well along the way, including in upcoming v0.7.0 patch. Some models benefitted, others regressed - overall better on harder scenarios or I wouldn't be releasing, but yeah - not intuitive.

The biggest challenge has been balancing the desire to hyper optimize for my favorite models, versus average behavior, versus consumer needs.

Re: Show HN: Forge – Guardrails take an 8B model from 53% to 99% on agentic tasks

#65

Post by zambelli »

tempoponet wrote: post #60

Why this entire tool chain instead of building within something like pi code? I've been exploring this area and a project like https://github.com/itayinbarr/little-coder (not my work) lets me mix and match with my current setup or any plugins built for pi.

Mainly because I have plenty of use cases and not all of them need or want pi. Forge isn't an orchestration framework and is not coding specific, it lives one level lower - if I understand pi correctly.

The proxy mode should integrate seamlessly, and the middleware guardrail mode could be lifted into pi.

As for little coder, I love it! I wanted forge to be more generic than just agentic coding as there's many more agentic workflows worth optimizing with small models.

Re: Show HN: Forge – Guardrails take an 8B model from 53% to 99% on agentic tasks

#66

Post by zambelli »

tim-projects wrote: post #63

I've been working on the same thing and even nearly called it forge. Instead I called it hammer. I'll be keen to look through the code on this!

Oh no! I have code-hammer coming out soon :D. Everyone is building stuff these days :p.

Always happy to see folks looking into small local models!

Re: Show HN: Forge – Guardrails take an 8B model from 53% to 99% on agentic tasks

#68

Post by [deleted] »

koolba wrote: post #59
Escapade5160 wrote: post #49

I've been saying for a while that given a proper harness, small local models can perform incredibly well. When you have a system that can try everything, it will eventually get it right as long as you can prevent it from getting it wrong in the meantime.

A thousand monkeys on a thousand typewriters…

[deleted]

Re: Show HN: Forge – Guardrails take an 8B model from 53% to 99% on agentic tasks

#69

Post by zambelli »

jonnyasmar wrote: post #67

[flagged]

That's where frontier pulls ahead for sure, at least on the big frontier models - though I haven't formalized those findings because...time.

Necessary disclaimer, forge isn't concerned, technically, with model quality, just execution of tool calls. Now for the actual answer...

What I found to be the limiting factor with small models in the 14B range was "effective attention". Beyond a certain point, still well within their training context window size, I start to see degradation. I don't have hard numbers for it, but that's where an Opus and the like can just keep going for ages. I did come up with a tool call message history collapse that I might dogfood into forge one day (effectively clean up the message history intelligently so the model doesn't lose track as easily).

That being said, my coding eval suite for my agentic coding harness does have some refactor tasks and feature additions (everything is done on an actual sandboxed repo) and the small models can knock out those tasks even while pushing the 50-60 tool call mark. But I wouldn't trust them to do more than 1 of those in the same session.

Re: Show HN: Forge – Guardrails take an 8B model from 53% to 99% on agentic tasks

#70

Post by zambelli »

koolba wrote: post #59
Escapade5160 wrote: post #49

I've been saying for a while that given a proper harness, small local models can perform incredibly well. When you have a system that can try everything, it will eventually get it right as long as you can prevent it from getting it wrong in the meantime.

A thousand monkeys on a thousand typewriters…

That is the whole challenge, actually! A new metric I'm going to dogfood into forge is ETTWS - estimated time to working solution.

A simple retry loop around your whole workflow could, in some cases, be all you need. But it could mean many blind attempts to get through a workflow successfully. And hopefully there isn't a payment step partway through!

The fewer hard errors nix the whole workflow, the lower your ETTWS.

Post reply on HN