Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Forge – Guardrails take an 8B model from 53% to 99% on agentic tasks

github.com

Post reply on HN
Print view View original

271–280 of 292 posts

Re: Show HN: Forge – Guardrails take an 8B model from 53% to 99% on agentic tasks

#271

Post by RichardFF991 »

andai wrote: post #203
6r17 wrote: post #30

Very cool work ! I'm running harness system myself and could measure improvement of token use of 2x to 10x on gsm8k only by running a math harness - i'm confident the future is bright for people who will know how to sell tech that is appropriately scaled to one's need. We absolutely do not need to run Claude 123 for most tasks and we better prepare for the rag-pull !

A while back when the latest Big Model came out, very impressive benchmarks, I tested it on some coding tasks. I gave it 3 simple changes to make. It did it perfectly. Then I tried with a much smaller model. It also did it perfectly, except 3x faster and 9x cheaper. I used to think "best model" was what's at the top of the benchmarks, but for most tasks that just means you're going to wait longer and pay more money.…

[flagged]

Re: Show HN: Forge – Guardrails take an 8B model from 53% to 99% on agentic tasks

#273

Post by coip »

Aurornis wrote: post #145
Escapade5160 wrote: post #49

I've been saying for a while that given a proper harness, small local models can perform incredibly well. When you have a system that can try everything, it will eventually get it right as long as you can prevent it from getting it wrong in the meantime.

The problem is that you get similar quality as if you gave a junior unlimited time to work on a problem and told them to keep trying different things until the goal is reached. Even the SOTA models have this problem when the work is complicated enough. The problem is amplified more with the small models.

One important facet of this is it’s not far from “giving unlimited juniors unlimited time…”

Where the limits are set by hardware for agentic execution (compute/network/storage) && inference speed

Re: Show HN: Forge – Guardrails take an 8B model from 53% to 99% on agentic tasks

#274

Post by zambelli »

mrothroc wrote: post #270
zambelli wrote: post #251

Earlier quoted context omitted.

Nice symmetry with tool call failures being sent to LLM that made the call without bugging the user. The artifact-generating entity gets the error back, effectively. 100% correct, and stackable. Could have topic refusal in LLM training itself, forge in tool call alter, and sdlc gates at the workflow level.

Definitely stacks. The thing that made it clear for me was being explicit about the stages, and where/what you can verify with a guardrail, or gate. I wrote up the framework I use here: https://michael.roth.rocks/research/trust-topology/ Being explicit about the space between the stages is critical, because that's your enforcement point.

This is a really neat writeup, and the empirical data for coding agents is super useful. Will take a closer read and see if there's anything I easily lift into my harness!

Re: Show HN: Forge – Guardrails take an 8B model from 53% to 99% on agentic tasks

#276

Post by zambelli »

zambelli wrote: post #2

Happy to answer questions about the eval methodology, the backend findings, or anything in the repo. I'll be around.

Thanks to everyone for the great discussion! v0.7.0 is out now. It was in flight when this landed - changes tool error channel based on dogfooding observations with some larger models, eval re-run (numbers shift but within CI), and most importantly docs updated! I hope they're clearer now.

Re: Show HN: Forge – Guardrails take an 8B model from 53% to 99% on agentic tasks

#277

Post by mrothroc »

zambelli wrote: post #274
mrothroc wrote: post #270

Earlier quoted context omitted.

Definitely stacks. The thing that made it clear for me was being explicit about the stages, and where/what you can verify with a guardrail, or gate. I wrote up the framework I use here: https://michael.roth.rocks/research/trust-topology/ Being explicit about the space between the stages is critical, because that's your enforcement point.

This is a really neat writeup, and the empirical data for coding agents is super useful. Will take a closer read and see if there's anything I easily lift into my harness!

Thanks, glad you find it useful! Feel free to ping me if you have any questions.

Re: Show HN: Forge – Guardrails take an 8B model from 53% to 99% on agentic tasks

#280

Post by IIsi50MHz »

zambelli wrote: post #23
snovv_crash wrote: post #22

I get a strong LLM smell in your description. If you couldn't bother to write it, why should I bother to read it?

I definitely use LLMs to help write things - but this is my draft! Maybe I've been spending too much time reading the evals and I now sound like an LLM... Either way, here I am - happy to answer any questions!

I do not hold with the detractors. I've personally known people who write like this since before LLMs were a thing. I rarely use LLMs, and come from the days of "expert systems" and the tail end of when universities had hundreds of typewriters in a room.

So perhaps, that's my bias: towards a former reality.

I often find myself wondering at a random HN commenter's flaming of a post for being full of AI slop, when the accused reads like normalspeak to my half-old eyes.

Post reply on HN