Earlier quoted context omitted.
OpenRouter should be penalising or banning for this.
They have an "exacto" category with providers they supposedly verified
That’s only for tool use.
GLM-5.2 is the new leading open weights model on Artificial Analysis
241–250 of 476 posts
Re: GLM-5.2 is the new leading open weights model on Artificial Analysis
#242Just ran and scored 63 3d model generations (via code) across high and no reasoning. 3D Modeling benchmark quickly shows spatial, logic and code performance of the model so I think it's a very good indicator of the quality. Here are the results compared to Gemini 3.5 Flash: Model + config CodeErr/gen Cost/gen Median time Quality gemini-3.5-flash, low 0.71 $0.18 68s baseline GLM 5.2, reasoning high 0.61 $0.18 289s -6.…
Would you be able to run it against Gemini Flash (not Lite) 3.0, high thinking?
Re: GLM-5.2 is the new leading open weights model on Artificial Analysis
#243Just ran and scored 63 3d model generations (via code) across high and no reasoning. 3D Modeling benchmark quickly shows spatial, logic and code performance of the model so I think it's a very good indicator of the quality. Here are the results compared to Gemini 3.5 Flash: Model + config CodeErr/gen Cost/gen Median time Quality gemini-3.5-flash, low 0.71 $0.18 68s baseline GLM 5.2, reasoning high 0.61 $0.18 289s -6.…
Very interested in this! Can you share more about the modelling method (eg, three js?), the task list, and outputs here? I think there's probably some good juice to squeeze in terms of spacial awareness by doing a benchmark something like - give 3d modelling task - render and snapshot from a variety of angles - feed to third-party vision model for a "what is this" type query - grade on end-to-end accuracy Bonus point…
I don't have the eval results live yet, so I cannot share them yet.
I was benchmarking using a soon to be released new version of my AI CAD modeling software[0]. It's basically an agent that has access to tools that can execute build123d scripts, get sculpted models, blender to combine sculpts + parametric models, tools to inspect the model (visually and with code), search datasheets, ...
I tried what you recommend a while ago (asking an AI to evaluate using different angles) and the AI evaluations were extremely bad - barely any correlation to what I scored. Things have gotten better, but I don't trust it enough yet.
Here is how I score adherence (and how AI did as well, but I tried methods where it would just give back a boolean "pass" or not):
dragon-bottle-stopper
editing-param-mid-conv
editing-parametric-enclosure
editing-swap-material-param
editing-text-edit-cube
multi-turn-bird-house
multi-turn-dice-tower
multi-turn-modular-planter
multi-turn-phone-stand
multi-turn-shelf
one-shot-bookend
one-shot-cable-clip
one-shot-chess-queen
one-shot-coaster
one-shot-coffee-cup
one-shot-dog-tag
one-shot-dragon-figurine
one-shot-hex-bracket
one-shot-keychain-fob
one-shot-low-poly-tree
one-shot-pegboard-hook
one-shot-pi4-case
one-shot-threaded-jar
Re: GLM-5.2 is the new leading open weights model on Artificial Analysis
#244Earlier quoted context omitted.
To answer the question in your first sentence - because it's VERY computationally (ha) expensive as a human being to keep up with all the options. It's also very hard to figure out how to run a model like this. There's no installer . If you really really care, which 99% of people do not, you have to google a guide, and then find out it's out of date... I've tried a number of these, and the learning curve is very stee…
> There's no installer. There's ZCode ( https://zcode.z.ai ). Which is like the Codex App. That's as "easy" as it is for non-devs that you're complaining about.
I'm not complaining about anything. I'm answering a question.
Re: GLM-5.2 is the new leading open weights model on Artificial Analysis
#245Just ran and scored 63 3d model generations (via code) across high and no reasoning. 3D Modeling benchmark quickly shows spatial, logic and code performance of the model so I think it's a very good indicator of the quality. Here are the results compared to Gemini 3.5 Flash: Model + config CodeErr/gen Cost/gen Median time Quality gemini-3.5-flash, low 0.71 $0.18 68s baseline GLM 5.2, reasoning high 0.61 $0.18 289s -6.…
Would you be able to run it against Gemini Flash (not Lite) 3.0, high thinking?
Absolutely. Running it now, will update this comment in about 30 mins.
Edit: Surprisingly very good results with 3.0 flash with high thinking.
Cost: $0.06
Duration: 3.22 min
Code Errors: 1.3 per attempts (meaning on average it had to retry 1.3 times)
Adherence was on par with 3.5 flash Low thinking
Re: GLM-5.2 is the new leading open weights model on Artificial Analysis
#246I tried it today through Openrouter and the API is atrocious. I got multiple rate limit and random errors every turn. Somebody wrote [1]; "I am never touching Minimax or GLM again. Their APIs had constant outages and I had to restart my runs multiple times — after burning money on the runs that failed midway." and I 100% agree. The model might be good, but if the API is so bad, it's effectively useless. [1]: https://…
Give it a few days and additional provider will be up and available on OpenRouter. Then the game of figuring out who’s not nuking the weights and neutering the quantization begins.
Re: GLM-5.2 is the new leading open weights model on Artificial Analysis
#247Earlier quoted context omitted.
The entire point of this post is that it's open weights, you can run it yourself and don't have to deal with the API issues. You really do have that choice.
You could subscribe to Anthropic/OpenAI for the rest of your life for the cost it would take to host GLM5.2 locally - you need 1.5TB of VRAM just for the weights
You don't need that much VRAM unless you're targeting a high-performance deployment that's intended to scale far beyond local use. For a lower-throughput case, you can keep the model weights on SSD at very low cost and stream them in for inference. This could actually scale reasonably well if you have something as simple as a previous-gen HEDT with a decent amount of PCIe lanes to host fast storage from.
Re: GLM-5.2 is the new leading open weights model on Artificial Analysis
#248Earlier quoted context omitted.
Would you be able to run it against Gemini Flash (not Lite) 3.0, high thinking?
Absolutely. Running it now, will update this comment in about 30 mins. Edit: Surprisingly very good results with 3.0 flash with high thinking. Cost: $0.06 Duration: 3.22 min Code Errors: 1.3 per attempts (meaning on average it had to retry 1.3 times) Adherence was on par with 3.5 flash Low thinking
Thanks! I’ve still been using 3.0 a lot, the price-to-performance ratio absolutely kills compared to Google’s other and newer offerings.
Re: GLM-5.2 is the new leading open weights model on Artificial Analysis
#249Correct me if I'm wrong, but neither DeepSeek nor GLM have image input modality. This makes them less useful when looking at UIs, photos, screenshots, etc. doesn't it? Or do they have alternate ways of doing so?
They do not and it sucks for certain tasks. It also means that if they actually trained with vision, they'd be on par with Anthropic models as vision seems to improve model performance across the board even for non-vision tasks.
it helps giving them a cli vision tool (curl to openrouter vision model for example)