Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Forge – Guardrails take an 8B model from 53% to 99% on agentic tasks

github.com

Post reply on HN
Print view View original

71–80 of 292 posts

Re: Show HN: Forge – Guardrails take an 8B model from 53% to 99% on agentic tasks

#71

Post by jonnyasmar »

zambelli wrote: post #69
jonnyasmar wrote: post #67

[flagged]

That's where frontier pulls ahead for sure, at least on the big frontier models - though I haven't formalized those findings because...time. Necessary disclaimer, forge isn't concerned, technically, with model quality, just execution of tool calls. Now for the actual answer... What I found to be the limiting factor with small models in the 14B range was "effective attention". Beyond a certain point, still well within…

The "effective attention" framing nails what I keep noticing too. Sonnet's official context is huge in principle, but in a real coding session where the agent is reading 30+ files, running grep, processing test output, emitting diffs — somewhere around 60-80k effective tokens I can feel it start to "skim" earlier context rather than reason over it. The thing it forgot isn't out of window; it's just not weighted highly enough anymore.

The tool-call history collapse is a problem I'd pay real money to have solved cleanly. My crude manual version: keep the function calls but drop or summarize the responses for anything older than ~15 turns. Most of the "what was I doing" signal lives in the calls, not the outputs. Letting the model itself mark "I'm done with that thread, compress the responses" feels like the right abstraction, but I haven't seen anyone ship it well yet.

A per-model "compaction aggressiveness" knob in Forge could be interesting — the small-model effective-attention cliff might respond to earlier/heavier trimming.

Re: Show HN: Forge – Guardrails take an 8B model from 53% to 99% on agentic tasks

#72

Post by zambelli »

jonnyasmar wrote: post #71
zambelli wrote: post #69

Earlier quoted context omitted.

That's where frontier pulls ahead for sure, at least on the big frontier models - though I haven't formalized those findings because...time. Necessary disclaimer, forge isn't concerned, technically, with model quality, just execution of tool calls. Now for the actual answer... What I found to be the limiting factor with small models in the 14B range was "effective attention". Beyond a certain point, still well within…

The "effective attention" framing nails what I keep noticing too. Sonnet's official context is huge in principle, but in a real coding session where the agent is reading 30+ files, running grep, processing test output, emitting diffs — somewhere around 60-80k effective tokens I can feel it start to "skim" earlier context rather than reason over it. The thing it forgot isn't out of window; it's just not weighted highl…

Forge does have tiered compaction, and it's configurable! Defaults are currently probably a bit on the high side for catching effective attention, but that might be a part of the code that interests you the most.

src/forge/context/ - specifically TieredCompact in strategies.py. That's the furthest I took it. The tool-call collapse in particular has been useful in agentic coding, but I haven't formalized/generalized it yet. I think within forge it'll be a callable tool that will rely on the model knowing when to trigger it (as you said - "I'm done with the task, can collapse"). That's the part I need to abstract out of my bespoke implementation.

Re: Show HN: Forge – Guardrails take an 8B model from 53% to 99% on agentic tasks

#73

Post by zambelli »

trollbridge wrote: post #57
zambelli wrote: post #46

Earlier quoted context omitted.

Yes it does! I haven't published those evals yet, but I'm actually running 24-35B class models on a custom coding harness built on forge (even 120B class recently). I just need more GPU wall clock time to get more evals done. ETA is...a few weeks? Got distracted by the coding harness. But the results are the same. Reforged models do better than bare, even at those sizes. As for published results, I ran forge on Anthr…

Exactly what I was thinking - even on frontier or near-frontier models I still see my agents get stuck in these pointless loops where it's very obvious to me what they need to do to get "unstuck".

Yeah, it's a useful framework even with frontier. And it definitely lifts "cheap" frontier models like Haiku into more solid territory. I haven't done a ton of forge integrations into frontier (like pointing claude code into proxy mode) yet, but if you run into any issues let me know!

Re: Show HN: Forge – Guardrails take an 8B model from 53% to 99% on agentic tasks

#74

Post by jonnyasmar »

zambelli wrote: post #72
jonnyasmar wrote: post #71

Earlier quoted context omitted.

The "effective attention" framing nails what I keep noticing too. Sonnet's official context is huge in principle, but in a real coding session where the agent is reading 30+ files, running grep, processing test output, emitting diffs — somewhere around 60-80k effective tokens I can feel it start to "skim" earlier context rather than reason over it. The thing it forgot isn't out of window; it's just not weighted highl…

Forge does have tiered compaction, and it's configurable! Defaults are currently probably a bit on the high side for catching effective attention, but that might be a part of the code that interests you the most. src/forge/context/ - specifically TieredCompact in strategies.py. That's the furthest I took it. The tool-call collapse in particular has been useful in agentic coding, but I haven't formalized/generalized i…

The "model triggers it" pattern is exactly the right shape, but there's a subtle failure mode in it: models are notoriously bad at perceiving their own context pressure. Asking "are you done with that thread?" lands well; asking "would compacting now help you?" doesn't, because the model lacks a reliable internal signal for "I'm starting to skim." You almost have to tie the compaction trigger to task-shape signals (file closed, test passed, agent reports a milestone hit) rather than self-assessment.

Going to actually go read TieredCompact tonight — curious whether you've ended up tying triggers to task signals or kept them on model self-report.

Re: Show HN: Forge – Guardrails take an 8B model from 53% to 99% on agentic tasks

#75

Post by [deleted] »

jonnyasmar wrote: post #71
zambelli wrote: post #69

Earlier quoted context omitted.

That's where frontier pulls ahead for sure, at least on the big frontier models - though I haven't formalized those findings because...time. Necessary disclaimer, forge isn't concerned, technically, with model quality, just execution of tool calls. Now for the actual answer... What I found to be the limiting factor with small models in the 14B range was "effective attention". Beyond a certain point, still well within…

The "effective attention" framing nails what I keep noticing too. Sonnet's official context is huge in principle, but in a real coding session where the agent is reading 30+ files, running grep, processing test output, emitting diffs — somewhere around 60-80k effective tokens I can feel it start to "skim" earlier context rather than reason over it. The thing it forgot isn't out of window; it's just not weighted highl…

[deleted]

Re: Show HN: Forge – Guardrails take an 8B model from 53% to 99% on agentic tasks

#76

Post by zambelli »

zambelli wrote: post #72
jonnyasmar wrote: post #71

Earlier quoted context omitted.

The "effective attention" framing nails what I keep noticing too. Sonnet's official context is huge in principle, but in a real coding session where the agent is reading 30+ files, running grep, processing test output, emitting diffs — somewhere around 60-80k effective tokens I can feel it start to "skim" earlier context rather than reason over it. The thing it forgot isn't out of window; it's just not weighted highl…

Forge does have tiered compaction, and it's configurable! Defaults are currently probably a bit on the high side for catching effective attention, but that might be a part of the code that interests you the most. src/forge/context/ - specifically TieredCompact in strategies.py. That's the furthest I took it. The tool-call collapse in particular has been useful in agentic coding, but I haven't formalized/generalized i…

At the moment TieredCompact is naive. It uses context thresholds the consumer determines and fires when those thresholds are hit. It just does different things at different threshold levels.

Your idea of using task shape to dynamically set those thresholds (or even move to model-triggered) I think is the key but is a trickier implementation. That's what I haven't gotten around to yet.

Definitely on my todo list but happy to check out a PR if you have something in mind.

Some additional info on my current public hack is also at: https://github.com/antoinezambelli/forge/blob/main/docs/USER...

Re: Show HN: Forge – Guardrails take an 8B model from 53% to 99% on agentic tasks

#77

Post by jonnyasmar »

zambelli wrote: post #76
zambelli wrote: post #72

Earlier quoted context omitted.

Forge does have tiered compaction, and it's configurable! Defaults are currently probably a bit on the high side for catching effective attention, but that might be a part of the code that interests you the most. src/forge/context/ - specifically TieredCompact in strategies.py. That's the furthest I took it. The tool-call collapse in particular has been useful in agentic coding, but I haven't formalized/generalized i…

At the moment TieredCompact is naive. It uses context thresholds the consumer determines and fires when those thresholds are hit. It just does different things at different threshold levels. Your idea of using task shape to dynamically set those thresholds (or even move to model-triggered) I think is the key but is a trickier implementation. That's what I haven't gotten around to yet. Definitely on my todo list but h…

Honestly probably not a PR from me right now — I'm in the middle of shipping something else — but the design idea I keep returning to is splitting the trigger into two signals:

1. Runtime-computed "context pressure" — tokens-since-last-compaction, depth of tool-call nesting, response/call ratio in recent turns. The runtime computes this; the model never sees it.

2. Model-emitted "natural breakpoint" — a tool call the model fires when it perceives it's done with a thread (file closed, task complete, branch abandoned).

Compaction fires on the AND of both. Keeps the model from compacting mid-reasoning-chain, and keeps the runtime from waiting until 90% context for the model to notice on its own.

Re: Show HN: Forge – Guardrails take an 8B model from 53% to 99% on agentic tasks

#78

Post by seemaze »

> One thing I really didn't expect: the serving backend matters. Same Mistral-Nemo 12B weights produce 7% accuracy on llama-server with native function calling and 83% on Llamafile in prompt mode.

I thought Llamafile was just a model and llama.cpp bundled in to a single binary - is this the difference between Llamafile injecting a default sysmtem prompt vs hitting the raw llama-server endpoint with no harness?

That seems like comparing apples to apple pie, there's some ingredients missing.

Re: Show HN: Forge – Guardrails take an 8B model from 53% to 99% on agentic tasks

#79

Post by imachine1980_ »

seemaze wrote: post #78

> One thing I really didn't expect: the serving backend matters. Same Mistral-Nemo 12B weights produce 7% accuracy on llama-server with native function calling and 83% on Llamafile in prompt mode. I thought Llamafile was just a model and llama.cpp bundled in to a single binary - is this the difference between Llamafile injecting a default sysmtem prompt vs hitting the raw llama-server endpoint with no harness? That s…

I wouldn't expect such difference

Re: Show HN: Forge – Guardrails take an 8B model from 53% to 99% on agentic tasks

#80

Post by jedisct1 »

Interesting!

The https://swival.dev harness already has retry nudges, step enforcement, error recovery, context awareness, etc. to try to support small models as much as possible.

Curious to see how it compares with forge, and if both could be combined.

Post reply on HN