Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Forge – Guardrails take an 8B model from 53% to 99% on agentic tasks

github.com

Post reply on HN
Print view View original

231–240 of 292 posts

Re: Show HN: Forge – Guardrails take an 8B model from 53% to 99% on agentic tasks

#231

Post by Mithriil »

killing_time wrote: post #162
zambelli wrote: post #70

Earlier quoted context omitted.

That is the whole challenge, actually! A new metric I'm going to dogfood into forge is ETTWS - estimated time to working solution. A simple retry loop around your whole workflow could, in some cases, be all you need . But it could mean many blind attempts to get through a workflow successfully. And hopefully there isn't a payment step partway through! The fewer hard errors nix the whole workflow, the lower your ETTWS…

Is it strange that I immediately interpreted ETTWS to be Estimated Time To William Shakespeare?

It's relevant to the "thousand monkeys on a thousand typewriters".

Re: Show HN: Forge – Guardrails take an 8B model from 53% to 99% on agentic tasks

#232

Post by c7b »

zambelli wrote: post #208
c7b wrote: post #177

Thanks for building what I'd hoped to find the time to build (and much better than what I would have made)! One question: do you think there is room for parallelization here, eg in the retry loop? Local models generally can handle a limited number (~ 2 digits) of concurrent requests pretty well, even on consumer hardware, which can give >10x boosts in the effective number of token/s. I've been thinking for a while ab…

Interesting - so you're thinking give the model two parallel shots at the tool call and take the winner if there is one, or fallback to retry if not? That would certainly work in theory, but I'm not as familiar with parallel calls. - If you mean the model calls the tool twice, identically, in a batch call - that would work fine and Forge handles batch calls, but many small models wouldn't think to do that so you'd ha…

So as an oversimplified PoC, I get:

    llama-parallel -m ~/models/Qwen3.5-4B-Q8_0.gguf -ns 4 -p "Fix this Python code, answer with code only: prnt('Hello World)" -pps

    llama_perf_context_print:        load time =    1181.90 ms
    llama_perf_context_print: prompt eval time =     190.57 ms /   374 tokens (    0.51 ms per token,  1962.49 tokens per second)
    llama_perf_context_print:        eval time =    3612.25 ms /   159 runs   (   22.72 ms per token,    44.02 tokens per second)
    llama_perf_context_print:       total time =    4302.84 ms /   533 tokens
    llama_perf_context_print:    graphs reused =        155
and four answers (3 of which are immediately usable), with -ns 1 I get :

    llama_perf_context_print:        load time =    1185.61 ms
    llama_perf_context_print: prompt eval time =     187.55 ms /   305 tokens (    0.61 ms per token,  1626.27 tokens per second)
    llama_perf_context_print:        eval time =     158.92 ms /     7 runs   (   22.70 ms per token,    44.05 tokens per second)
    llama_perf_context_print:       total time =     468.85 ms /   312 tokens
    llama_perf_context_print:    graphs reused =          6
Now this is probably not the right way to use it, you should probably also use vLLM instead and it's also not a good model to use for this. But there is a real effect here that others have demonstrated, that the GPU is apparently not always maxed out while handling a single request, so sending concurrent requests can yield substantial parallelization benefits. The idea with this application would be something like this: send off the same query in parallel requests, triggering parallel tool calls, and then filter the results (filter out all failing ones, rank the rest by some simple metric of code complexity). There are probably better applications as well, I'm basically just thinking what kinds of tasks could benefit from parallelization.

Re: Show HN: Forge – Guardrails take an 8B model from 53% to 99% on agentic tasks

#233

Post by zambelli »

alsetmusic wrote: post #228

Sounds like an implementation of the discussion[0] spawned by this[1] article. I've been thinking about the best way to implement such a system ever since seeing that. I'm going to try this out. 0. https://news.ycombinator.com/item?id=48051562 1. https://bsuh.bearblog.dev/agents-need-control-flow/

[dead]

Re: Show HN: Forge – Guardrails take an 8B model from 53% to 99% on agentic tasks

#234

Post by zambelli »

sfifs wrote: post #227

Very nice! I also saw you have a vllm branch and I validated it works on my system. There is bugfix which sent you a PR for to auto-discover served-model-name which vllm hard validates for.

Oh, awesome! I'll take a look.

Re: Show HN: Forge – Guardrails take an 8B model from 53% to 99% on agentic tasks

#235

Post by 1dom »

JKCalhoun wrote: post #207
1dom wrote: post #183

I like work in this area, and this is really helpful, thanks. I actively avoid cloud based LLMs and mainly use 4b - 30a3b param local models. This means I don't really have a good grasp of SOTA LLM performance or accuracy, but I know what to expect when dealing with local models, and where the pain points are. I've only skimmed the post and read the abstract and in some places you make a nod to how simple tweaks can…

"I actively avoid cloud based LLMs… This means I don't really have a good grasp of SOTA LLM performance or accuracy… …but then all of your metrics and data seem to focus 100% on accuracy. You need to address speed." I wonder, if you were to use cloud-based LLMs more often, you might find that accuracy (fidelity?) is indeed more more lacking in your local models. You can always just throw hardware at your speed proble…

I agree accuracy isn't maybe the best word here, I used it as it was used in the original post, mainly a as a catchall for "everything but speed", so fidelity, perplexity, etc.

I also agree that if I spent more time using cloud based LLMs, I would very much find local LLMs less capable and useful. Comparison is the thief of joy though, and I'd rather feel blissfully ignorance towards SOTA LLMs rather than a dependence on them.

Before taking a local focus approach, LLMs increasingly left me feeling a mixture of FOMO, sadness and futility towards the future of software and tech. I assume it's 100% a me problem, but it has it's benefits:)

Re: Show HN: Forge – Guardrails take an 8B model from 53% to 99% on agentic tasks

#236

Post by mrothroc »

simonw wrote: post #92

This is a neat project, but the description made me realize that I don't actually know what the term "guardrails" means. ... which lead me to realize that it's one of those terms with multiple meanings - like "agent" or even "AI" itself - but where people who use it may not be aware of how many different definitions are floating around. In this project it refers to validating tool calls - fixing invalid tool response…

Yes, "guardrails" is a squishy term. But it gets clearer if you ask what transition is being guarded.

Some of this is inside the model, like topic refusals. Forge sits at the tool call level.

My personal workflow uses guardrails at the SDLC level: I have a standard pipeline (plan, design, code, build, test). I use gates between each stage, and the right composition leads to a much higher quality in the final product.

Also worth mentioning that gate failures are given to the agent that produced the artifact, so it has a chance to fix it. That means that I don't have to review obviously wrong output.

Re: Show HN: Forge – Guardrails take an 8B model from 53% to 99% on agentic tasks

#237

Post by zambelli »

anentropic wrote: post #225
zambelli wrote: post #204

Earlier quoted context omitted.

Hi! Latency is definitely a factor in any system, and the dashboard and paper do report elapsed time - but at the workflow level. On a per-call basis, the wrappers are pure python ifs and such, measured in ms easily, and frankly negligible compared to the LLM call itself which will be on the order of magnitude seconds. Where timing gets interesting is that forge will slow down workflows because the retries mean you d…

> On a per-call basis, the wrappers are pure python ifs and such, measured in ms easily Ah that's good to know when I first saw this posted yesterday I was wondering that, kind of assumed maybe it was doing extra LLM calls to make judgements

Retry nudges do generate an extra LLM call, and those average extra calls time impacts are captured in the eval data.

But that's the difference between the call failing and succeeding (eventually).

On successful calls the presence of forge should be unnoticeable.

Re: Show HN: Forge – Guardrails take an 8B model from 53% to 99% on agentic tasks

#240

Post by tmjdev »

zambelli wrote: post #210
monster_truck wrote: post #179

Hey this genuinely _fucks_, you're a legend. You can even get stupid good results from the 1 bit bonsai models! Plays v nice with lmstudio It's now completely reasonable to throw a 7900XTX in a spare rig, put it in the basement, give it an absurd goal, and forget about it.

Thanks! Did you try it with lmstudio? I actually never tried it with that. Only published ollama, llamfile, llama.cpp native/prompt - and unofficially tested vLLM, but never lmstudio.

I couldn't get it working with lmstudio. I have bonsai-8B running with llama-cpp and am attempting to build a harness for it. Looking good so far, I just got it started but Forge made tool calling work pretty quickly!
Post reply on HN