Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Forge – Guardrails take an 8B model from 53% to 99% on agentic tasks

github.com

Post reply on HN
Print view View original

121–130 of 292 posts

Re: Show HN: Forge – Guardrails take an 8B model from 53% to 99% on agentic tasks

#121

Post by jvalencia »

lwansbrough wrote: post #96

Had a couple thoughts in this realm, and am working them into my own harness. Curious to see what others think. I'm not sure if this is generalizable, as my harness is fairly specialized: - Breaking down a problem into a planned execution, with executing agent providing the initial plan which includes explicit objectives such as which tools it calls and what it would consider to be a successful execution. - The harne…

I've been writing my own, out of curiosity, with gemma4. I've been surprised how far I'm getting.

Re: Show HN: Forge – Guardrails take an 8B model from 53% to 99% on agentic tasks

#122

Post by bglusman »

zambelli wrote: post #108
esperent wrote: post #98

Earlier quoted context omitted.

I've just read through your readme and I have zero clue what this does. Something about proxying model calls and applying "policies" to them? But what kind of things does it actually do, what benefits are there? That should be at the top of the readme.

I'm sorry to hear that! I'll take a fresh look at docs in my upcoming release. In a nutshell, it applies guardrails around LLM calls to make them more reliable - specifically small models but works on all: "on multi-step agentic workflows through guardrails (rescue parsing, retry nudges, step enforcement) and context management (VRAM-aware budgets, tiered compaction).". It'll try to parse malformed tool calls, it'll…

I think that comment was aimed at my Wardwright link, not Forge, given mention of policies and proxying model calls! I think your docs are in much better shape ;-)

Re: Show HN: Forge – Guardrails take an 8B model from 53% to 99% on agentic tasks

#123

Post by zambelli »

jvalencia wrote: post #121
lwansbrough wrote: post #96

Had a couple thoughts in this realm, and am working them into my own harness. Curious to see what others think. I'm not sure if this is generalizable, as my harness is fairly specialized: - Breaking down a problem into a planned execution, with executing agent providing the initial plan which includes explicit objectives such as which tools it calls and what it would consider to be a successful execution. - The harne…

I've been writing my own, out of curiosity, with gemma4. I've been surprised how far I'm getting.

Very cool! Hopefully you'll share it someday!

Re: Show HN: Forge – Guardrails take an 8B model from 53% to 99% on agentic tasks

#124

Post by zambelli »

bglusman wrote: post #122
zambelli wrote: post #108

Earlier quoted context omitted.

I'm sorry to hear that! I'll take a fresh look at docs in my upcoming release. In a nutshell, it applies guardrails around LLM calls to make them more reliable - specifically small models but works on all: "on multi-step agentic workflows through guardrails (rescue parsing, retry nudges, step enforcement) and context management (VRAM-aware budgets, tiered compaction).". It'll try to parse malformed tool calls, it'll…

I think that comment was aimed at my Wardwright link, not Forge, given mention of policies and proxying model calls! I think your docs are in much better shape ;-)

lol - my bad! but thanks!

Re: Show HN: Forge – Guardrails take an 8B model from 53% to 99% on agentic tasks

#125

Post by Karuma »

snovv_crash wrote: post #22

I get a strong LLM smell in your description. If you couldn't bother to write it, why should I bother to read it?

Thank you for mentioning it. Too bad you got downvoted to hell as usual when anybody dares to do it.

The original post and every comment by OP is so full of AI slop ("the biggest surprise!", "one thing I didn't expect!", "the biggest challenge!", etc. etc.") that is absolutely painful to read. I still can't believe most people (especially here on HN, I thought we were a bit better than this) can't notice all this stuff.

What's much worse, it's that all these people posting this useless slop are so dishonest ("I definitely use LLMs to help write things - but this is my draft!") that it makes me really nauseous... This is the worst time to be an internet user if you have more than 2 points of IQ.

Re: Show HN: Forge – Guardrails take an 8B model from 53% to 99% on agentic tasks

#126

Post by fc417fc802 »

arijun wrote: post #95
henry2023 wrote: post #93

Earlier quoted context omitted.

Generated comments are not allowed. https://news.ycombinator.com/newsguidelines.html#generated https://news.ycombinator.com/item?id=47340079

Why do you think their comment is AI generated? I didn’t get that from it but I’m no expert.

The general tone (it just feels like it's an LLM) but also check the account history. It's a 2018 account that had never commented until today's flood of suspicious comments.

Re: Show HN: Forge – Guardrails take an 8B model from 53% to 99% on agentic tasks

#127

Post by trollbridge »

zambelli wrote: post #73
trollbridge wrote: post #57

Earlier quoted context omitted.

Exactly what I was thinking - even on frontier or near-frontier models I still see my agents get stuck in these pointless loops where it's very obvious to me what they need to do to get "unstuck".

Yeah, it's a useful framework even with frontier. And it definitely lifts "cheap" frontier models like Haiku into more solid territory. I haven't done a ton of forge integrations into frontier (like pointing claude code into proxy mode) yet, but if you run into any issues let me know!

And we're off! It's working great with DeepSeek V4, although DeepSeek V4 Pro tends not to really run into problems anyway being near-frontier, but I definitely see improvement with Flash.

Re: Show HN: Forge – Guardrails take an 8B model from 53% to 99% on agentic tasks

#128

Post by trollbridge »

zambelli wrote: post #94
simonw wrote: post #92

This is a neat project, but the description made me realize that I don't actually know what the term "guardrails" means. ... which lead me to realize that it's one of those terms with multiple meanings - like "agent" or even "AI" itself - but where people who use it may not be aware of how many different definitions are floating around. In this project it refers to validating tool calls - fixing invalid tool response…

That's a fair point, and frankly something that might not age well in my docs one day. I genuinely don't know what the industry will standardize on when it comes to the use of the term "guardrails". I've seen the sec definitions as well. You're 100% right about how I meant it and what it means within Forge though, but it's something that might lead to doc changes as things evolve.

I'm thinking of it like a guardrail that keeps your car from driving off the edge of a road, but in this case, it keeps your tool calls from driving off a cliff.

Re: Show HN: Forge – Guardrails take an 8B model from 53% to 99% on agentic tasks

#129

Post by zambelli »

trollbridge wrote: post #127
zambelli wrote: post #73

Earlier quoted context omitted.

Yeah, it's a useful framework even with frontier. And it definitely lifts "cheap" frontier models like Haiku into more solid territory. I haven't done a ton of forge integrations into frontier (like pointing claude code into proxy mode) yet, but if you run into any issues let me know!

And we're off! It's working great with DeepSeek V4, although DeepSeek V4 Pro tends not to really run into problems anyway being near-frontier, but I definitely see improvement with Flash.

That was fast! It's great to hear it's working well :)

Did you notice any particular guardrails firing? Always curious about things I haven't tested on - especially if it has a different shape.

Post reply on HN