Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Forge – Guardrails take an 8B model from 53% to 99% on agentic tasks

github.com

Post reply on HN
Print view View original

221–230 of 292 posts

Re: Show HN: Forge – Guardrails take an 8B model from 53% to 99% on agentic tasks

#221

Post by zambelli »

MWil wrote: post #198
zambelli wrote: post #105

Earlier quoted context omitted.

Do you mean catching errors as tokens stream back versus waiting for the full message? If so, then no I hadn't looked into that. This was mostly geared towards local models so token cost isn't really a big deal, though latency might be. And if you didn't mean that then please elaborate :)

No, more like not waiting for drift/deviation to hit something load bearing or god forbid go on hitting unnoticed over time. Let it hit something trivial that is constantly being monitored cheaply. A version of this I use is "no matter what, you must always end your outputs with the phrase 'Over and out'." Once it stops doing this with outputs, even if I haven't noticed any load-bearing drift or issue elsewhere, I im…

Ohhhh, that's much more interesting. I haven't looked into that at all, but now I'm curious. I'd need to think way more about how to layer that into forge, but the principle could likely be applied somewhere. I get it now.

Re: Show HN: Forge – Guardrails take an 8B model from 53% to 99% on agentic tasks

#222

Post by Aurornis »

Zetaphor wrote: post #206
Aurornis wrote: post #145

Earlier quoted context omitted.

The problem is that you get similar quality as if you gave a junior unlimited time to work on a problem and told them to keep trying different things until the goal is reached. Even the SOTA models have this problem when the work is complicated enough. The problem is amplified more with the small models.

There's a lot of valuable things that can be done in that range, especially when token costs aren't a concern. Not every problem requires SOTA

> especially when token costs aren't a concern. Not every problem requires SOTA

If token costs aren’t a concern I’m using SOTA for everything.

Even SOTA gets it wrong and hallucinates, but at a lower rate. I don’t want to waste my time.

Re: Show HN: Forge – Guardrails take an 8B model from 53% to 99% on agentic tasks

#223

Post by forlorn_mammoth »

cornholio wrote: post #55
Escapade5160 wrote: post #49

I've been saying for a while that given a proper harness, small local models can perform incredibly well. When you have a system that can try everything, it will eventually get it right as long as you can prevent it from getting it wrong in the meantime.

If I understood correctly, the model will get it right because it knows when it isn't right.

the missile knows where it is because it knows where it isn't

Re: Show HN: Forge – Guardrails take an 8B model from 53% to 99% on agentic tasks

#225

Post by anentropic »

zambelli wrote: post #204
1dom wrote: post #183

I like work in this area, and this is really helpful, thanks. I actively avoid cloud based LLMs and mainly use 4b - 30a3b param local models. This means I don't really have a good grasp of SOTA LLM performance or accuracy, but I know what to expect when dealing with local models, and where the pain points are. I've only skimmed the post and read the abstract and in some places you make a nod to how simple tweaks can…

Hi! Latency is definitely a factor in any system, and the dashboard and paper do report elapsed time - but at the workflow level. On a per-call basis, the wrappers are pure python ifs and such, measured in ms easily, and frankly negligible compared to the LLM call itself which will be on the order of magnitude seconds. Where timing gets interesting is that forge will slow down workflows because the retries mean you d…

> On a per-call basis, the wrappers are pure python ifs and such, measured in ms easily

Ah that's good to know

when I first saw this posted yesterday I was wondering that, kind of assumed maybe it was doing extra LLM calls to make judgements

Re: Show HN: Forge – Guardrails take an 8B model from 53% to 99% on agentic tasks

#226

Post by zambelli »

faizshah wrote: post #224

If anyone else couldn’t find the working paper link (the readme and conf link didn’t work for me) it’s this one here: https://github.com/antoinezambelli/forge/blob/main/docs/forg...

Thank you! I've been trying to catch those replies and redirect people, but hopefully your comment be upvoted for others. Very embarrassing to put up the post with the wrong link lol.

Re: Show HN: Forge – Guardrails take an 8B model from 53% to 99% on agentic tasks

#228

Post by alsetmusic »

Sounds like an implementation of the discussion[0] spawned by this[1] article. I've been thinking about the best way to implement such a system ever since seeing that. I'm going to try this out.

0. https://news.ycombinator.com/item?id=48051562

1. https://bsuh.bearblog.dev/agents-need-control-flow/

Post reply on HN