Earlier quoted context omitted.
I have on Qwen3.8 27B. Results are close enough across multiple runs.
So it's not reproducible.
This is inevitable unfortunately with LLMs.
I tested 10 model/harness combinations on the same Three.js task
71–80 of 84 posts
Re: I tested 10 model/harness combinations on the same Three.js task
#72I love everything about OpenCode except for the stuff it outputs. On paper it has everything I want in a harness and more, but it tends to struggle to deliver the desired outcome. Last I used it the context was massive and tool calls were reliably unreliable. Is it worth revisiting as a daily driver?
Why not ohmypi?
Cache rate for oh-my-pi is significantly lower than opencode's in your results.
Re: I tested 10 model/harness combinations on the same Three.js task
#73Would be good to include Fable and Opus. Based on the results with Astra vs GLM (not even in the same ballpark), these are probably the realistic alternatives to consider.
Yes, I'd be really interested to see Fable / Opus / Sonnet.
GLM 5.3 Flash Max had an interesting showing. Its Codex version was bad [0], and it completed in 9 minutes. The OpenCode version was much richer [1] and more detailed, completed in 20 minutes. And the OMP version was arguably the most complete [2], completing in 30 minutes.
This is probably the strongest argument for the effect of a harness, and I'd be interested to learn the differences in the prompts and tools between these three.
[0]: https://alvins82.github.io/hangar-harness-model-tests/hangar...
[1]: https://alvins82.github.io/hangar-harness-model-tests/hangar...
[2]: https://alvins82.github.io/hangar-harness-model-tests/hangar...
Re: I tested 10 model/harness combinations on the same Three.js task
#74Earlier quoted context omitted.
Can you elaborate on this? how would you work on a larger THREE project? would you create models independently?
> how would you work on a larger THREE project? A larger THREE.js project starts to look more and more like a game engine, so you pick and choose the parts you need. There's a ton of open source libs, most of the heavy components the big players use are open source, things like physics, mesh optimization. Key AI-specific parts might be: - a harness (so the agent can drive the thing) - authoring pipeline (so you can b…
Does anyone have an efficient, reasonably designed three.js or other web oriented mini game engine? Low poly but modern rendering effects should be a good niche for hobbyist stuff but hard to find.
Re: I tested 10 model/harness combinations on the same Three.js task
#75Would be good to include Fable and Opus. Based on the results with Astra vs GLM (not even in the same ballpark), these are probably the realistic alternatives to consider.
Yes, I'd be really interested to see Fable / Opus / Sonnet. GLM 5.3 Flash Max had an interesting showing. Its Codex version was bad [0], and it completed in 9 minutes. The OpenCode version was much richer [1] and more detailed, completed in 20 minutes. And the OMP version was arguably the most complete [2], completing in 30 minutes. This is probably the strongest argument for the effect of a harness, and I'd be inter…
I have posted all the transcripts on github so you can point your AI at it and prompt away. https://github.com/alvins82/hangar-harness-model-tests/
Re: I tested 10 model/harness combinations on the same Three.js task
#76Would be good to include Fable and Opus. Based on the results with Astra vs GLM (not even in the same ballpark), these are probably the realistic alternatives to consider.
Yes, I'd be really interested to see Fable / Opus / Sonnet. GLM 5.3 Flash Max had an interesting showing. Its Codex version was bad [0], and it completed in 9 minutes. The OpenCode version was much richer [1] and more detailed, completed in 20 minutes. And the OMP version was arguably the most complete [2], completing in 30 minutes. This is probably the strongest argument for the effect of a harness, and I'd be inter…
[deleted]
Re: I tested 10 model/harness combinations on the same Three.js task
#77I wish there was another column with the estimated cost for each, with a specific date. Ideally also finding somehow (not sure what would be the right away) what is publicly available before running the test. It's quite a different outcome if there are competitions, e.g. js13k, live code examples from books, even templates, on specific that topic. Visually here the results looks very very similar to the point that I…
[dead]
Re: I tested 10 model/harness combinations on the same Three.js task
#78I wish there was another column with the estimated cost for each, with a specific date. Ideally also finding somehow (not sure what would be the right away) what is publicly available before running the test. It's quite a different outcome if there are competitions, e.g. js13k, live code examples from books, even templates, on specific that topic. Visually here the results looks very very similar to the point that I…
Added costs.
Re: I tested 10 model/harness combinations on the same Three.js task
#79I’m always confused, are all these shapes programmatically generated or are they downloaded from some source? Also I think Astra looks the best and has the best functionality. Also shocked how much better GLM is on the Non Codex harnesses. Didn’t think it would make such a difference. Would be nice if you could include cost in the table
Added costs.
Re: I tested 10 model/harness combinations on the same Three.js task
#80Damn its unreal how much Astra is better than anything else :( Would be cool to see total cost
Added costs.