Tangentially related: Since you are at Texas Instruments, I wonder if you could find out what the status is of the intellectual property for the TI Explorer lisp machines. I know who owns the IP for Genera, but wasn’t able to find out about TI’s lisp OS
Show HN: Forge – Guardrails take an 8B model from 53% to 99% on agentic tasks
11–20 of 292 posts
Re: Show HN: Forge – Guardrails take an 8B model from 53% to 99% on agentic tasks
#12Tangentially related: Since you are at Texas Instruments, I wonder if you could find out what the status is of the intellectual property for the TI Explorer lisp machines. I know who owns the IP for Genera, but wasn’t able to find out about TI’s lisp OS
Very tangential! I'll try but it might take me a while.
Re: Show HN: Forge – Guardrails take an 8B model from 53% to 99% on agentic tasks
#13I'd be curious about the eval methodology. In production coding tasks, the gap between benchmark scores and actual workflow integration can be significant. What does the error recovery loop look like?
Re: Show HN: Forge – Guardrails take an 8B model from 53% to 99% on agentic tasks
#14Re: Show HN: Forge – Guardrails take an 8B model from 53% to 99% on agentic tasks
#15Hello. Interesting project! Haven't gone through it yet, but want to consider using this in my CS master's capstone. While you have benchmarks I may create my own specific scenarios and comparisons vis-a-vis hosted inference to highlight specific economic benefit. Any suggestions?
Re: Show HN: Forge – Guardrails take an 8B model from 53% to 99% on agentic tasks
#16I'd be curious about the eval methodology. In production coding tasks, the gap between benchmark scores and actual workflow integration can be significant. What does the error recovery loop look like?
Absolutely, benchmarks are a different breed. Forge's eval is deliberately scoped as a stress test of the recovery loop, not a measure of end-to-end agentic quality.
Scenarios range from basic 2-step workflows, to more complex ones with dead ends, breadcrumbs, misleading names.
Concrete example: Task: get, analyze and report on Q3 sales data.
Model emits: analyze_sales(quarter="Q3"). This skipped the fetch step. Forge's response validator catches it before the tool function runs. Instead of letting the bad call hit the real impl (which would error or hallucinate), forge replies on the canonical tool-result channel.
We send this to the model: tool_result: [PrereqError] analyze_sales requires fetch_sales_data to be called first. Available next steps: fetch_sales_data
Model emits a corrected fetch_sales_data(...) on the next turn.
Three enforcement paths use this same channel: prerequisite violations, premature terminal calls, unknown-tool retries.
We also have rescue parsing for known templates (Jason OpenAI style, XML like granite, etc) where we try to parse tool calls that might be malformed.
And lastly bare text response nudges. Small models love to chat, we need them to call tools!
Re: Show HN: Forge – Guardrails take an 8B model from 53% to 99% on agentic tasks
#17Hello. Interesting project! Haven't gone through it yet, but want to consider using this in my CS master's capstone. While you have benchmarks I may create my own specific scenarios and comparisons vis-a-vis hosted inference to highlight specific economic benefit. Any suggestions?
Very cool! I would look at the tokens returned by each of the calls. You can map those to API costs per input/output tokens. Forge should be capturing those (or can, as passthrough from llama.cpp).
At least, if I understand your economic benefit angle correctly.
For scenarios to get inspired by I'd look at those tagged "model_quality" or "advanced_reasoning".
Re: Show HN: Forge – Guardrails take an 8B model from 53% to 99% on agentic tasks
#18Hey I'm really impressed and hoping to connect. I followed you on X just now, is that a decent place to shoot you a DM? I don't want anything from you, we just seem to be working on similar things (I'm working on our internal agent harness here, at a healthcare startup).
Re: Show HN: Forge – Guardrails take an 8B model from 53% to 99% on agentic tasks
#19Impressive work, love seeing tools that boost local LLM reliability without touching the model itself
Re: Show HN: Forge – Guardrails take an 8B model from 53% to 99% on agentic tasks
#20Hey I'm really impressed and hoping to connect. I followed you on X just now, is that a decent place to shoot you a DM? I don't want anything from you, we just seem to be working on similar things (I'm working on our internal agent harness here, at a healthcare startup).
Neat! Historically I've been most active on LinkedIn but the AI community seems very X-leaning so I'll make sure to pay closer attention there. Good luck with the harness, happy to connect!