21–30 of 292 posts
Impressive work, love seeing tools that boost local LLM reliability without touching the model itself
I'm in the same boat, tuning models wasn't super interesting, though I might do a focused spike on behavior -focused fine tuning. But the harness matters almost more than the model in many cases.
I get a strong LLM smell in your description. If you couldn't bother to write it, why should I bother to read it?
Maybe I've been spending too much time reading the evals and I now sound like an LLM...
Either way, here I am - happy to answer any questions!
I get a strong LLM smell in your description. If you couldn't bother to write it, why should I bother to read it?
Happy to answer questions about the eval methodology, the backend findings, or anything in the repo. I'll be around.
I run small models at home, so I'm very curious.
Earlier quoted context omitted.
Ah, so it a "smart" retry mechanism?
I'd like to think so! ;). It has some brains, but the key insight was to send the model domain-agnostic nudges. I don't need to know what you're trying to do, the LLM already knows, I just need to nudge it back on the structural side: text response vs tool call, arg mismatch, etc. and let its knowledge of the context fill in the blanks (otherwise I'd need a massive library of every possible failure mode). The other i…
I get a strong LLM smell in your description. If you couldn't bother to write it, why should I bother to read it?
If you are so outright against using AI, why would he care if you read his article about AI?
If you're generating AI text you shouldn't expect humans that you aren't paying to bother reading it, purely out of politeness. Brian Cantrill has a great piece on this: https://rfd.shared.oxide.computer/rfd/0576
Interesting point about backend variance. Do you think serving layer should become part of standard LLM eval reporting?
I get a strong LLM smell in your description. If you couldn't bother to write it, why should I bother to read it?
I definitely use LLMs to help write things - but this is my draft! Maybe I've been spending too much time reading the evals and I now sound like an LLM... Either way, here I am - happy to answer any questions!
I play with local models a lot but also have limited time and the conciseness, polish and human indication in presentation has become a major quality indicator. I've wasted too much time with slop projects or people's LLM-induced delusions and now take a pretty strict line on what I'm willing to spend my time on. Even if this ends up with some false positives, there's just so much happening these days it doesn't really matter...
Best of luck with Forge!