Very cool work!
Regarding your finding "the tool ran successfully and returned data" and "the tool ran successfully but found nothing." Couldn’t this be solved by designing better tool responses instead of adding another layer in between? Just curious and probing my understanding.
Show HN: Forge – Guardrails take an 8B model from 53% to 99% on agentic tasks
131–140 of 292 posts
Re: Show HN: Forge – Guardrails take an 8B model from 53% to 99% on agentic tasks
#132What are "guardrails" in this context? Is it correctly understood that this would sit between my pi agent and llama-server, and it would do what exactly?
It would help ensure that the model executes its tool call correctly. So if you give Pi a task like booking travel... Pi decides to book a flight, hotel, car. It gets the flight in one go, but then sends "here is the payload : [json blob]" to hotel booking API and the whole thing throws an error and the workflow dies, with partial completion. Forge would catch the error and nudge the model by injecting a message into…
Nice explanation, thank you.
So basically the kind of thing I'd usually be doing manually with small models, over and over again, you just automate that nudging and off they go.
Sometimes LLMs have seemed to me like "computer programs with inertia" and in that frame what your tool does is identify and reduce friction at key points so the wheels can keep spinning.
Re: Show HN: Forge – Guardrails take an 8B model from 53% to 99% on agentic tasks
#133Tangentially related: Since you are at Texas Instruments, I wonder if you could find out what the status is of the intellectual property for the TI Explorer lisp machines. I know who owns the IP for Genera, but wasn’t able to find out about TI’s lisp OS
Who owns the IP for Genera?
John C. Mallery of MIT
Re: Show HN: Forge – Guardrails take an 8B model from 53% to 99% on agentic tasks
#134Why this entire tool chain instead of building within something like pi code? I've been exploring this area and a project like https://github.com/itayinbarr/little-coder (not my work) lets me mix and match with my current setup or any plugins built for pi.
Mainly because I have plenty of use cases and not all of them need or want pi. Forge isn't an orchestration framework and is not coding specific, it lives one level lower - if I understand pi correctly. The proxy mode should integrate seamlessly, and the middleware guardrail mode could be lifted into pi. As for little coder, I love it! I wanted forge to be more generic than just agentic coding as there's many more ag…
[deleted]
Re: Show HN: Forge – Guardrails take an 8B model from 53% to 99% on agentic tasks
#135I'm curious if in proxy mode it works also with remote models or only with local models.
Also, did someone tried it with local Qwen 3.6?
Re: Show HN: Forge – Guardrails take an 8B model from 53% to 99% on agentic tasks
#136Deja vu from the other week https://news.ycombinator.com/item?id=48051562
Re: Show HN: Forge – Guardrails take an 8B model from 53% to 99% on agentic tasks
#137Very cool work! Regarding your finding "the tool ran successfully and returned data" and "the tool ran successfully but found nothing." Couldn’t this be solved by designing better tool responses instead of adding another layer in between? Just curious and probing my understanding.
100%, a better tool would work or even remove the problem overall.
The isssue/use-case is more around, say, a database table or legacy systems where your tool is just hitting a legacy API that may or may not be good. A surface you don't control.
It didn't come up as a use-case in this eval honestly, it's more the concept of a standard, like 4xx vs 5xx. I just felt it was missing from the ecosystem overall.
Re: Show HN: Forge – Guardrails take an 8B model from 53% to 99% on agentic tasks
#138Earlier quoted context omitted.
A thousand monkeys on a thousand typewriters…
That is the whole challenge, actually! A new metric I'm going to dogfood into forge is ETTWS - estimated time to working solution. A simple retry loop around your whole workflow could, in some cases, be all you need . But it could mean many blind attempts to get through a workflow successfully. And hopefully there isn't a payment step partway through! The fewer hard errors nix the whole workflow, the lower your ETTWS…
Have you read the MAKER/MDAP paper? 1 million sequential tasks.
Re: Show HN: Forge – Guardrails take an 8B model from 53% to 99% on agentic tasks
#139Earlier quoted context omitted.
It would help ensure that the model executes its tool call correctly. So if you give Pi a task like booking travel... Pi decides to book a flight, hotel, car. It gets the flight in one go, but then sends "here is the payload : [json blob]" to hotel booking API and the whole thing throws an error and the workflow dies, with partial completion. Forge would catch the error and nudge the model by injecting a message into…
Nice explanation, thank you. So basically the kind of thing I'd usually be doing manually with small models, over and over again, you just automate that nudging and off they go. Sometimes LLMs have seemed to me like "computer programs with inertia" and in that frame what your tool does is identify and reduce friction at key points so the wheels can keep spinning.
Yep! The big frontier models are already quite good at doing that, and they have decent harnesses. That's why Opus on Claude Code does what it does.
Small models aren't there yet and they would veer off course, this just nudges them back onto the road. Whether or not they have a good sense of direction is a different question.
Re: Show HN: Forge – Guardrails take an 8B model from 53% to 99% on agentic tasks
#140This is fantastic. I haven't got any local inference as I can't afford it right now, but tool calling has been a concern for me with these smaller models through OpenRouter.
I've been working on a pytest-first acceptance testing framework called Dokimasia (do-kee-ma-see-ah) that I'd love to get your thoughts on: https://github.com/deevus/dokimasia
Acceptance testing might not be what you need for Forge, but since you're deep in AI tool building I thought you may have opinions.