This seems similar to what I done using llama.cpp's "Grammar constrained generation" for my local agents. But using that instead of catching and retrying it is just literally impossible for the LLM to generate something that doesn't match a specific schema of tool choices. It is amazing how much better small models can be when you reduce the problem space to only grammatically correct answers.
Show HN: Forge – Guardrails take an 8B model from 53% to 99% on agentic tasks
241–250 of 292 posts
Re: Show HN: Forge – Guardrails take an 8B model from 53% to 99% on agentic tasks
#242Earlier quoted context omitted.
There's a lot of valuable things that can be done in that range, especially when token costs aren't a concern. Not every problem requires SOTA
> especially when token costs aren't a concern. Not every problem requires SOTA If token costs aren’t a concern I’m using SOTA for everything. Even SOTA gets it wrong and hallucinates, but at a lower rate. I don’t want to waste my time.
I believe they mean token costs aren't a concern when you're not paying for a SOTA model via API, and are instead running local models.
Infinite monkeys on infinite typewriters, and all that.
Re: Show HN: Forge – Guardrails take an 8B model from 53% to 99% on agentic tasks
#243This seems similar to what I done using llama.cpp's "Grammar constrained generation" for my local agents. But using that instead of catching and retrying it is just literally impossible for the LLM to generate something that doesn't match a specific schema of tool choices. It is amazing how much better small models can be when you reduce the problem space to only grammatically correct answers.
Interesting, catching the problem upstream, effectively. How did you enforce the grammar?
Re: Show HN: Forge – Guardrails take an 8B model from 53% to 99% on agentic tasks
#244I was hoping it would work with vLLM (openai compatible) to test it, does anyone know a similar proxy for local coding models?
Re: Show HN: Forge – Guardrails take an 8B model from 53% to 99% on agentic tasks
#245[flagged]
Re: Show HN: Forge – Guardrails take an 8B model from 53% to 99% on agentic tasks
#246I like work in this area, and this is really helpful, thanks. I actively avoid cloud based LLMs and mainly use 4b - 30a3b param local models. This means I don't really have a good grasp of SOTA LLM performance or accuracy, but I know what to expect when dealing with local models, and where the pain points are. I've only skimmed the post and read the abstract and in some places you make a nod to how simple tweaks can…
Hi! Latency is definitely a factor in any system, and the dashboard and paper do report elapsed time - but at the workflow level. On a per-call basis, the wrappers are pure python ifs and such, measured in ms easily, and frankly negligible compared to the LLM call itself which will be on the order of magnitude seconds. Where timing gets interesting is that forge will slow down workflows because the retries mean you d…
Hi! Thanks for the response. Like I mentioned, I only skimmed, and it sounds like there's more to it than I understand, so I'll take a deeper look and see how it feels in practice.
> Where timing gets interesting is that forge will slow down workflows because the retries mean you don't error right away. Bare runs were failing fast in my experience. But on a per-call basis there's very little overhead.
> I haven't detailed it simply because the order of magnitude of a single LLM call is so much higher than all the overhead put together.
Yeah, that makes sense and seems fair. The sort of delays are almost and inevitability, you're not trying to improve speed, but by improving reliability, it can obviously increase overall throughput.
Having watched the demo video too now, automating retries etc would be helpful for me. It's impressive to see how quick the models run on better hardware, and the performance improvements are impressive, even if the overall run takes longer sometimes because it does more correct things. Thanks again!
Re: Show HN: Forge – Guardrails take an 8B model from 53% to 99% on agentic tasks
#247This seems similar to what I done using llama.cpp's "Grammar constrained generation" for my local agents. But using that instead of catching and retrying it is just literally impossible for the LLM to generate something that doesn't match a specific schema of tool choices. It is amazing how much better small models can be when you reduce the problem space to only grammatically correct answers.
Interesting, catching the problem upstream, effectively. How did you enforce the grammar?
https://github.com/ggml-org/llama.cpp/blob/master/grammars/R...
llama.cpp supports grammar limiting using either GBNF or json schema (It just translate it to GBNF behind the scenes I think). So I have my harness generate a tool schema on the fly (based on what tools are possible for the current task) and pass it in at request time.
Re: Show HN: Forge – Guardrails take an 8B model from 53% to 99% on agentic tasks
#248Earlier quoted context omitted.
Thanks! Did you try it with lmstudio? I actually never tried it with that. Only published ollama, llamfile, llama.cpp native/prompt - and unofficially tested vLLM, but never lmstudio.
I couldn't get it working with lmstudio. I have bonsai-8B running with llama-cpp and am attempting to build a harness for it. Looking good so far, I just got it started but Forge made tool calling work pretty quickly!
Very cool! I'll try to get an issue open on lmstudio support and add it to the backlog.
Re: Show HN: Forge – Guardrails take an 8B model from 53% to 99% on agentic tasks
#249Earlier quoted context omitted.
Interesting, catching the problem upstream, effectively. How did you enforce the grammar?
https://github.com/ggml-org/llama.cpp/blob/master/grammars/R... llama.cpp supports grammar limiting using either GBNF or json schema (It just translate it to GBNF behind the scenes I think). So I have my harness generate a tool schema on the fly (based on what tools are possible for the current task) and pass it in at request time.
Oh, interesting - thanks for the link. I really haven't explored this but it should slot in fairly easily I think? Gotta dig into it more.
Re: Show HN: Forge – Guardrails take an 8B model from 53% to 99% on agentic tasks
#250Earlier quoted context omitted.
> especially when token costs aren't a concern. Not every problem requires SOTA If token costs aren’t a concern I’m using SOTA for everything. Even SOTA gets it wrong and hallucinates, but at a lower rate. I don’t want to waste my time.
I believe they mean token costs aren't a concern when you're not paying for a SOTA model via API, and are instead running local models. Infinite monkeys on infinite typewriters, and all that.
Correct, I have local hardware, not infinite money.