how do you know your harness design isn’t just overfitting on your test set?
Show HN: Forge – Guardrails take an 8B model from 53% to 99% on agentic tasks
201–210 of 292 posts
Re: Show HN: Forge – Guardrails take an 8B model from 53% to 99% on agentic tasks
#202[flagged]
Re: Show HN: Forge – Guardrails take an 8B model from 53% to 99% on agentic tasks
#203Very cool work ! I'm running harness system myself and could measure improvement of token use of 2x to 10x on gsm8k only by running a math harness - i'm confident the future is bright for people who will know how to sell tech that is appropriately scaled to one's need. We absolutely do not need to run Claude 123 for most tasks and we better prepare for the rag-pull !
A while back when the latest Big Model came out, very impressive benchmarks, I tested it on some coding tasks.
I gave it 3 simple changes to make. It did it perfectly.
Then I tried with a much smaller model. It also did it perfectly, except 3x faster and 9x cheaper.
I used to think "best model" was what's at the top of the benchmarks, but for most tasks that just means you're going to wait longer and pay more money. The right model depends on the job.
(Also, speed itself is a feature -- when you get the really fast models, it enables a kind of real-time interactive usage that is otherwise not possible in the "alt tab and hope it's done" workflow.)
Re: Show HN: Forge – Guardrails take an 8B model from 53% to 99% on agentic tasks
#204I like work in this area, and this is really helpful, thanks. I actively avoid cloud based LLMs and mainly use 4b - 30a3b param local models. This means I don't really have a good grasp of SOTA LLM performance or accuracy, but I know what to expect when dealing with local models, and where the pain points are. I've only skimmed the post and read the abstract and in some places you make a nod to how simple tweaks can…
Hi! Latency is definitely a factor in any system, and the dashboard and paper do report elapsed time - but at the workflow level.
On a per-call basis, the wrappers are pure python ifs and such, measured in ms easily, and frankly negligible compared to the LLM call itself which will be on the order of magnitude seconds.
Where timing gets interesting is that forge will slow down workflows because the retries mean you don't error right away. Bare runs were failing fast in my experience. But on a per-call basis there's very little overhead.
I haven't detailed it simply because the order of magnitude of a single LLM call is so much higher than all the overhead put together.
Re: Show HN: Forge – Guardrails take an 8B model from 53% to 99% on agentic tasks
#205When it comes to the business logic of production use, this particular failure type is less obvious compared to benchmarking tasks. Benchmarking involves having the answer already known — it helps detect mismatches easily. Business logic pipeline does not. If LLM gives out a valid output that happens to be semantically incorrect, the pipeline goes through. There is no mistake to catch. Created a dedupe pipeline where…
Definitely, there's several failure modes and Forge doesn't address all of them. This is just one tool in the toolbox to getting things stable enough for production use at reduced costs.
Forge sits one level lower - in my mind - than a gate which would sit more at the workflow level. Perfectly complementary.
Re: Show HN: Forge – Guardrails take an 8B model from 53% to 99% on agentic tasks
#206I've been saying for a while that given a proper harness, small local models can perform incredibly well. When you have a system that can try everything, it will eventually get it right as long as you can prevent it from getting it wrong in the meantime.
The problem is that you get similar quality as if you gave a junior unlimited time to work on a problem and told them to keep trying different things until the goal is reached. Even the SOTA models have this problem when the work is complicated enough. The problem is amplified more with the small models.
There's a lot of valuable things that can be done in that range, especially when token costs aren't a concern. Not every problem requires SOTA
Re: Show HN: Forge – Guardrails take an 8B model from 53% to 99% on agentic tasks
#207I like work in this area, and this is really helpful, thanks. I actively avoid cloud based LLMs and mainly use 4b - 30a3b param local models. This means I don't really have a good grasp of SOTA LLM performance or accuracy, but I know what to expect when dealing with local models, and where the pain points are. I've only skimmed the post and read the abstract and in some places you make a nod to how simple tweaks can…
"I actively avoid cloud based LLMs… This means I don't really have a good grasp of SOTA LLM performance or accuracy…
…but then all of your metrics and data seem to focus 100% on accuracy. You need to address speed."
I wonder, if you were to use cloud-based LLMs more often, you might find that accuracy (fidelity?) is indeed more more lacking in your local models.
You can always just throw hardware at your speed problems after all.
Re: Show HN: Forge – Guardrails take an 8B model from 53% to 99% on agentic tasks
#208Thanks for building what I'd hoped to find the time to build (and much better than what I would have made)! One question: do you think there is room for parallelization here, eg in the retry loop? Local models generally can handle a limited number (~ 2 digits) of concurrent requests pretty well, even on consumer hardware, which can give >10x boosts in the effective number of token/s. I've been thinking for a while ab…
Interesting - so you're thinking give the model two parallel shots at the tool call and take the winner if there is one, or fallback to retry if not?
That would certainly work in theory, but I'm not as familiar with parallel calls.
- If you mean the model calls the tool twice, identically, in a batch call - that would work fine and Forge handles batch calls, but many small models wouldn't think to do that so you'd have to explicitly prompt it to do so.
- If you mean ask the LLM twice to call the tool and look at both answers, my only concern would be latency from doing 2 calls instead of 1.
- Unless you're truly running 2 instances of the model and aren't memory-bandwidth bound, then yes running parallel workflows would likely help. Especially if you could have them compare notes at certain steps or something.
But I haven't explored this much at all so if you're thinking of something else, let me know!
Re: Show HN: Forge – Guardrails take an 8B model from 53% to 99% on agentic tasks
#209Earlier quoted context omitted.
I genuinely don't, sadly. I'm a mathematician originally, evolved organically into ML then AI - but I never really was a SWE. I feel like there's some backend decoding or chat template thing going on at a much lower level than what I'm best at. Maybe it's injecting headers or something that eventually compounds to model confusion? I really have no idea. I really hope folks better than me at backend stuff take a look…
Hey, this is most probably related to the chat template or the reasoning parser or the tool call parser or also things like kv cache quantization and possibly other params that affect results like the regular top k top p and all of that, the backend often sets its own defaults or the lack of them. It’s best to have all these under control if possible. I wonder regarding this project have you been testing it on real w…
Yes I've now used it "in the wild" for a handful of use-cases. I still run into the backend thing even when declaring params though, which is odd to me. But there might be params not typically passed in with the model that backends are setting. Again, really not my area of expertise.
As for consumers, I've done a home assistant, an agentic coding harness, and an autonomous engineering project (still in flight).
Re: Show HN: Forge – Guardrails take an 8B model from 53% to 99% on agentic tasks
#210Hey this genuinely _fucks_, you're a legend. You can even get stupid good results from the 1 bit bonsai models! Plays v nice with lmstudio It's now completely reasonable to throw a 7900XTX in a spare rig, put it in the basement, give it an absurd goal, and forget about it.
Thanks! Did you try it with lmstudio? I actually never tried it with that. Only published ollama, llamfile, llama.cpp native/prompt - and unofficially tested vLLM, but never lmstudio.