[flagged]
Show HN: Forge – Guardrails take an 8B model from 53% to 99% on agentic tasks
191–200 of 292 posts
Re: Show HN: Forge – Guardrails take an 8B model from 53% to 99% on agentic tasks
#192Nice work, thank you very much! Did you develop that as part of your work at TI or on your own time?
Re: Show HN: Forge – Guardrails take an 8B model from 53% to 99% on agentic tasks
#193[flagged]
Re: Show HN: Forge – Guardrails take an 8B model from 53% to 99% on agentic tasks
#194I like work in this area, and this is really helpful, thanks. I actively avoid cloud based LLMs and mainly use 4b - 30a3b param local models. This means I don't really have a good grasp of SOTA LLM performance or accuracy, but I know what to expect when dealing with local models, and where the pain points are. I've only skimmed the post and read the abstract and in some places you make a nod to how simple tweaks can…
what does "30a3b" mean?
Re: Show HN: Forge – Guardrails take an 8B model from 53% to 99% on agentic tasks
#195Earlier quoted context omitted.
Generated comments are not allowed. https://news.ycombinator.com/newsguidelines.html#generated https://news.ycombinator.com/item?id=47340079
Why do you think their comment is AI generated? I didn’t get that from it but I’m no expert.
Stylistic tells: "The tool-call ambiguity point", "—", "the negative space", "The retry-nudge layer", "the right shape", "→", "context drift"
Correctness tells: find exits with 0 when no matches were found, not 1. LLMs do get confused about tool call results sometimes but it's nowhere near as bad as needing "[manual corrections] multiple times an hour".
Contextual tells: see their account history and other comments.
Re: Show HN: Forge – Guardrails take an 8B model from 53% to 99% on agentic tasks
#196I get a strong LLM smell in your description. If you couldn't bother to write it, why should I bother to read it?
Thank you for mentioning it. Too bad you got downvoted to hell as usual when anybody dares to do it. The original post and every comment by OP is so full of AI slop ("the biggest surprise!", "one thing I didn't expect!", "the biggest challenge!", etc. etc.") that is absolutely painful to read. I still can't believe most people (especially here on HN, I thought we were a bit better than this) can't notice all this stu…
Just saying you’re not alone, very surprised by the reception given how brutally sloppified the OP is.
Interesting problems space but I hope the author just gives dot points next time rather than bloating it and losing most of its meaning.
Re: Show HN: Forge – Guardrails take an 8B model from 53% to 99% on agentic tasks
#197I like work in this area, and this is really helpful, thanks. I actively avoid cloud based LLMs and mainly use 4b - 30a3b param local models. This means I don't really have a good grasp of SOTA LLM performance or accuracy, but I know what to expect when dealing with local models, and where the pain points are. I've only skimmed the post and read the abstract and in some places you make a nod to how simple tweaks can…
what does "30a3b" mean?
Guess: 30B MOA with 3B active
Re: Show HN: Forge – Guardrails take an 8B model from 53% to 99% on agentic tasks
#198have you considered implementing the addition of a leading canary sentinel that fires at the earliest/cheapest possible point instead of only on lag of some actual load-bearing constraint violation?
Do you mean catching errors as tokens stream back versus waiting for the full message? If so, then no I hadn't looked into that. This was mostly geared towards local models so token cost isn't really a big deal, though latency might be. And if you didn't mean that then please elaborate :)
No, more like not waiting for drift/deviation to hit something load bearing or god forbid go on hitting unnoticed over time. Let it hit something trivial that is constantly being monitored cheaply.
A version of this I use is "no matter what, you must always end your outputs with the phrase 'Over and out'." Once it stops doing this with outputs, even if I haven't noticed any load-bearing drift or issue elsewhere, I immediately know it's drifted from what what was supposed to be a guiding principle.
Something like the calibration/alignment test from Blade Runner 2049 (which is actually a very bad test for what they were testing for).
Re: Show HN: Forge – Guardrails take an 8B model from 53% to 99% on agentic tasks
#199I like work in this area, and this is really helpful, thanks. I actively avoid cloud based LLMs and mainly use 4b - 30a3b param local models. This means I don't really have a good grasp of SOTA LLM performance or accuracy, but I know what to expect when dealing with local models, and where the pain points are. I've only skimmed the post and read the abstract and in some places you make a nod to how simple tweaks can…
what does "30a3b" mean?
Yup, confirming what pamcake said, 30b with 3b active.
I have a laptop with a broken screen and an RTX2060 at my disposal. I can run 12b - 14b dense usably, just, although I think 4b - 8b dense models give me the best tradeoff of speed and usefulness.
Larger MOE models with more parameters (20b+) but fewer active (2 - 3b) are sometimes a little bit slower, but are often far more capable.
Re: Show HN: Forge – Guardrails take an 8B model from 53% to 99% on agentic tasks
#200When it comes to the business logic of production use, this particular failure type is less obvious compared to benchmarking tasks. Benchmarking involves having the answer already known — it helps detect mismatches easily. Business logic pipeline does not. If LLM gives out a valid output that happens to be semantically incorrect, the pipeline goes through. There is no mistake to catch.
Created a dedupe pipeline where an LLM decides whether two feature requests are similar enough to merge. Occasional mistakes in terms of false positives — valid JSON structure, but incorrectly assessed similarity. In this case, it didn’t help to implement the retry technique. The solution was implementing a deterministic gate validating the output of the model based on its semantic similarity score calculated separately.
The reason why recovery works only with the help of additional tools when the error rate is at zero percent becomes clear: the LLM does not recognize the fact that it made a mistake. The guardrail becomes necessary for that — the retry is just one way of implementing the guardrail concept.