611–620 of 620 posts
Earlier quoted context omitted.
Curious what model you're using that works well on a 16GB card? I very much want to use my 5080 for inference, but everything I've tried so far has either just not been good enough or painfully slow.
gemma 4 12b
The harness was initially built to lift the quality of Qwen 3.6 27B, and then I expanded it to any model out there.
As many here have said, harness is critical, and it can do wonders to your model if you do it right. TSForge, the harness I built IMO, is the best harness on the market for writing full stack TS apps, as I have incorporated all the best practices, tools, etc. surrounding the stack.
If you don't believe me, give it a shot for a minute, I have no doubts that you'll agree.
Earlier quoted context omitted.
Not a harness issue. The harness (pi in my case) passes back the cot for all previous turns. The jinja template is what renders the openai-format request sent by the harness, into the actual string of text that will be tokenized and fed to the model. For models without preserve thinking support, the jinja template drops the reasoning from all but the current turn. Here is the default jinja for Gemma 4: https://huggin…
my heavily modified test bed for of oh my pi fixes this
Anyhow, I've heard mixed results on any method of supplying reasoning traces beyond the current turn to models not trained on them. For some models, I've heard that it works fine this way, for others I've heard it degrades performance. But I don't know of anyone who has any kind of reliable benchmark for how well this works.
Earlier quoted context omitted.
There is no Claude 4 Opus model... It's a series of model, of which the strongest is Opus 4.8, and Qwen 3.6 35B-A3b gets 51.5% on Swe-bench pro to Opus 4.8's 69.2%
Huh? There is a Claude 4 Opus. It was released about a year ago. It is retired by now, in fact, just retired yesterday: https://platform.claude.com/docs/en/about-claude/model-depre... But it is still available on Google Vertex according to OpenRouter (though it's possible that info is just out of date, it's currently quoting 3tps which is unusably slow): https://openrouter.ai/anthropic/claude-opus-4
Earlier quoted context omitted.
thats a harness issue not a model issue. eg i have my own reasoninf harness that forced persisted cot
Would you mind sharing your harness for reasoning?
use that to install assuming you have whatevers needed to set up.
ive a much fancier next gen thing i hope to make available as a sass mid to late summer, but if you have any feedback or questions on mine do reach out
Always a bit disappointed in the details in these kinds of threads. When you do get answers, they're never specific enough to try out on your own. It'll be something like "I use Qwen 3.5 and get great results!" OK but what quantization are you using? What llama parameters? What context size? What GPU are you running it on, and how much VRAM does it have? Are you hosting it on a separate box, or running it locally on…
I have good results with this setup: Hardware: - GPU: AMD 7900xtx, 24gb vram - CPU: AMD 5950x, AM4 - RAM: 64gb DDR4 3600 Software: - OS: Bazzite (atomic fedora - this machine is running Steam "big picture" mode on my TV when not in use for LLM tasks) - Virtualization: Podman Quadlets, which allows me to run container images as managed systemd units - Network: tailscale - Inference: llama.cpp vulkan (better performanc…
I've had some success with local models by chaining "agents" together in a workflow. Each agent has a different prompt and uses a different ollama model based on what their role is. The project manager, schema agent(qwen3:14b), etc. doesn't use the same model as the coding agent (qwen2.5-coder:7b). Between each step is an orchestrator and with a Playwright task which attempts to surface errors to the agent who introd…
I wish someone would do a benchmark and competition for this kind of work flow so we could figure out what works well. Like "Here's this consumer grade GPU. Using only this GPU but with whatever models and workflow you want, see how well you can do on xyz benchmark." Contestants would be given like 1 hour max and scored based on % of questions answered, % of questions correct and total time to finish. Like "The Local…
Earlier quoted context omitted.
How are you sandboxing your Pi coding harness? Directly only mounting certain folders, using capabilities to kill the network and not giving it all your shell env vars, that sort of thing? Or do you use a tool?
I'm using firejail to sandbox Opencode, for security and to keep the agents from personal data. I documented it in my blog [1]. [1] https://blog.fidelramos.net/software/how-i-sandbox-ai-agents...