Nice. But these tests raise a question what results are reproducible, the final visual, time, tool calling or it's mostly noise. Like, have you tried same combination or model and harness multiple times?
I tested 10 model/harness combinations on the same Three.js task
11–20 of 75 posts
Re: I tested 10 model/harness combinations on the same Three.js task
#12What I'm missing in all of those one-shot tests:
How different are the results between multiple runs of the same setup?
Re: I tested 10 model/harness combinations on the same Three.js task
#13Refreshing to see a post that gets straight to the point and not hide the data behind pages of prose.
Re: I tested 10 model/harness combinations on the same Three.js task
#14The Astra version seems to have used three.js r170, which is from October 2024. Sol used an even earlier version. GLM's code used the latest version, but I think it's just getting three.js@latest from jsdelivr so it's unlikely to be writing code against that version. Qwen on OpenCode also fetches from jsdelivr, but using a pinned version at r160.
I don't think any of these examples are using things like tone mapping so they're stuck in sRGB (AgX or ACES look much better), they're not using the node materials (good for programmatic texture implementation), and they're not doing anything cool like baking shadow environments or using post-processing effect.
They're nice, but I think they're showing how far behind AI models are on this sort of project rather than how good they are.
Re: I tested 10 model/harness combinations on the same Three.js task
#15I love everything about OpenCode except for the stuff it outputs. On paper it has everything I want in a harness and more, but it tends to struggle to deliver the desired outcome. Last I used it the context was massive and tool calls were reliably unreliable. Is it worth revisiting as a daily driver?
Re: I tested 10 model/harness combinations on the same Three.js task
#16[dead]
Re: I tested 10 model/harness combinations on the same Three.js task
#17I wish there was another column with the estimated cost for each, with a specific date. Ideally also finding somehow (not sure what would be the right away) what is publicly available before running the test. It's quite a different outcome if there are competitions, e.g. js13k, live code examples from books, even templates, on specific that topic. Visually here the results looks very very similar to the point that I…
I've just pushed the transcripts of each if that helps. https://github.com/alvins82/hangar-harness-model-tests
Re: I tested 10 model/harness combinations on the same Three.js task
#18I love everything about OpenCode except for the stuff it outputs. On paper it has everything I want in a harness and more, but it tends to struggle to deliver the desired outcome. Last I used it the context was massive and tool calls were reliably unreliable. Is it worth revisiting as a daily driver?
Why not ohmypi?
Re: I tested 10 model/harness combinations on the same Three.js task
#19Nice. But these tests raise a question what results are reproducible, the final visual, time, tool calling or it's mostly noise. Like, have you tried same combination or model and harness multiple times?
I have on Qwen3.8 27B. Results are close enough across multiple runs.
Re: I tested 10 model/harness combinations on the same Three.js task
#20What I'm missing in all of those one-shot tests: How different are the results between multiple runs of the same setup?
I have done multiple runs on Qwen3.8 27B. Results are close enough across multiple runs qualitatively.