Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Forge – Guardrails take an 8B model from 53% to 99% on agentic tasks

github.com

Post reply on HN
Print view View original

161–170 of 292 posts

Re: Show HN: Forge – Guardrails take an 8B model from 53% to 99% on agentic tasks

#161

Post by zambelli »

digitaltrees wrote: post #159

It’s really strange to see a project I really fundamentally agree from the same author of a project I fundamentally disagree with. How is it that you want to remove watermarks from AI generated images while also making AI more a more reliable partner? I am not trying to be combative or accusatory, just am curious about your world view an open to an argument that removing the origin of AI generated images isn’t an exi…

I think you have me confused with someone else. I haven't worked on any AI watermark removal project.

What project are you referring to specifically?

Re: Show HN: Forge – Guardrails take an 8B model from 53% to 99% on agentic tasks

#162

Post by killing_time »

zambelli wrote: post #70
koolba wrote: post #59

Earlier quoted context omitted.

A thousand monkeys on a thousand typewriters…

That is the whole challenge, actually! A new metric I'm going to dogfood into forge is ETTWS - estimated time to working solution. A simple retry loop around your whole workflow could, in some cases, be all you need . But it could mean many blind attempts to get through a workflow successfully. And hopefully there isn't a payment step partway through! The fewer hard errors nix the whole workflow, the lower your ETTWS…

Is it strange that I immediately interpreted ETTWS to be Estimated Time To William Shakespeare?

Re: Show HN: Forge – Guardrails take an 8B model from 53% to 99% on agentic tasks

#164

Post by zambelli »

Jayakumark wrote: post #163

Probably in same league https://github.com/Doorman11991/smallcode

I think there's certainly overlap there - and I love to see small local models being leveraged!

I do think there's some differences though. The biggest one being that forge isn't a coding harness, it's a guardrail primitive, really. Applicable to any tool-calling workflow.

As for the errors, are you nudging or passing errors back or swallowing them completely? Love the 2-stage routing though, neat!

Re: Show HN: Forge – Guardrails take an 8B model from 53% to 99% on agentic tasks

#165

Post by zambelli »

_fizz_buzz_ wrote: post #158

So, I experimented a little bit with smaller models and the problem I faced is that it would simply not call a tool that is available, but instead just describe the tool. Is this something that Forge can help with?

Within limits, yes. Forge has escalating nudges that will tell the model effectively "stop responding with text, you MUST call a tool" vibes. If the model is emitting something like "ok, let me call the tool: [valid json tool call in the middle of prose]" then we catch it with rescue parsing.

But at the end of the day, if the model keeps responding with text, there's nothing forge can do. I've run into that failure mode for sure, even with forge.

That works well enough for all the models shown in the eval here: relatively modern 8B+ models.

But some of the older generation (mistral 7b, that sort of thing) still can't be reliably used in something like a production setting.

Re: Show HN: Forge – Guardrails take an 8B model from 53% to 99% on agentic tasks

#169

Post by zambelli »

momo26 wrote: post #167

I was wondering that will modifying prompts or contracting the context also impact the performance? It may mistake the original meaning, and these steps also need help from external LLM.

Forge doesn't modify the prompt, it just injects information into the conversation as if it was a conversation turn. Over many turns - it can degrade the model (a concept I'm calling "effective attention"). But that requires serious context growth that really only becomes relevant for long-running agentic coding tasks in my experience. Still, it's possible.

Context compaction can also affect the outcome - I have eval scenarios for that as well but not in the published set, only in the repo. For those, I'd say "it's better than nothing". If you hit max context, the whole thing will barf or OOM the rig or something like that. So compaction degrades performance versus some theoretical ideal where you never need to, certainly. But it's better than a hard failure. Eval on those scenarios showed increasing degradation depending on severity of compaction. I view the auto-compaction as insurance. I never give the models tasks that will require that much context, but if it ends up getting there then the run might be saved.

Re: Show HN: Forge – Guardrails take an 8B model from 53% to 99% on agentic tasks

#170

Post by digitaltrees »

zambelli wrote: post #161
digitaltrees wrote: post #159

It’s really strange to see a project I really fundamentally agree from the same author of a project I fundamentally disagree with. How is it that you want to remove watermarks from AI generated images while also making AI more a more reliable partner? I am not trying to be combative or accusatory, just am curious about your world view an open to an argument that removing the origin of AI generated images isn’t an exi…

I think you have me confused with someone else. I haven't worked on any AI watermark removal project. What project are you referring to specifically?

You’re right. I followed another hacker news thread to the git repo of the watermark removal project and saw your name and seem to have wrongly connected you as an author of both projects.

Seriously awesome concept to what you did build I will test it out.

If you’re interested, I have sponsored research on AI reliability with Duke University (my graduate Alma mater) and there is an active research project this might be a good fit for if your interested in participating.

Post reply on HN