Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GLM 5.2 beats Claude in our benchmarks

semgrep.dev

Post reply on HN
Print view View original

221–230 of 559 posts

Re: GLM 5.2 beats Claude in our benchmarks

#221

Post by jklmnopqrstuvw »

pimeys wrote: post #59

I have taken another look on these open models after the fiasco of Fable and GPT 5.6 this weekend and... GLM-5.2 truly is a good workhorse model for daily programming. I consider myself a heavy user of LLMs and a seasoned developer. A typical session for me with GPT is usually over a hundred dollars... This weekend I programmed a matrix bot with encryption and a Rust agent with some tools. Because I need one and Open…

> A typical session for me with GPT is usually over a hundred dollars.

I don't think a $100 session is "typical". I use GPT for months. $20/m plus plan is enough for my daily work.

Re: GLM 5.2 beats Claude in our benchmarks

#222

Post by Madmallard »

gertlabs wrote: post #172
pimeys wrote: post #59

I have taken another look on these open models after the fiasco of Fable and GPT 5.6 this weekend and... GLM-5.2 truly is a good workhorse model for daily programming. I consider myself a heavy user of LLMs and a seasoned developer. A typical session for me with GPT is usually over a hundred dollars... This weekend I programmed a matrix bot with encryption and a Rust agent with some tools. Because I need one and Open…

GLM 5.2 is a great model, but if you only want to use the best model available, it isn't there yet. Every lab releases models that memorize benchmark answers, both intentionally and unintentionally. But we consistently find that models from Chinese labs have a wider gap between public benchmarks and our evaluations, which we designed to be less vulnerable to benchmaxxing. In multi-agent coding environments, GLM 5.2 i…

Notice the website url is the same name as the commentor.

Notice he's using "trust me bro" benchmarks.

Can we just remove all the motivated speech on HN? This is just not trustworthy information at all and obviously is incentivized.

Everyone is grinding and marketing nobody is actually discussing anything for real.

Re: GLM 5.2 beats Claude in our benchmarks

#223

Post by Paradigm2020 »

lenerdenator wrote: post #209
chillfox wrote: post #201

Earlier quoted context omitted.

So... how's that any different from using American stuff for those of us in the rest of the world? Over the last decade, the US has been way more unreliable than China. There's been a near constant negative impact from the US doing something. At least with China, we are very good at winning trade wars with them here in Australia.

You might feel differently if you were a Filipino or Vietnamese fisherman whose family relied on the income from the stocks of the South China Sea, or a Uighur person living in Western China, or a Ukrainian soldier who has to deal with drones built with Chinese components, or a democracy advocate in Hong Kong, or arguably, a person who had plans for 2020-2021. Or, on a more local note, an Australian automotive worker…

You might feel different if you're a palestinian who's getting american bombs dropped on him, or an afghani collateral damage or...

There is no good guys in general, and whataboutism and making the scope bigger doesn't help.

The thing is that if the models you are building on are open source whether hosted on chinese / american / whatever service at least give you an option to switch provider easier vs a fable / chatgpt 5.6 that gets banned for none americans etc...

2 years ago america would have had the branding/perception advantage but right now that is well and truly gone...

Re: GLM 5.2 beats Claude in our benchmarks

#224

Post by scottcha »

computerex wrote: post #163
sroerick wrote: post #157

Earlier quoted context omitted.

I'm using synthetic.new and Neuralwatt with pi and its good and also cheap

I have had bad experience with neuralwatt GLM 5.2. Seems like they may be using quantized version of the model.

Hi I'm the CTO of neuralwatt, would love to hear your feedback on what your experience was. Feel free to email me scott@neuralwatt.com. Also for GLM5.2 we run the FP8 quantization at 1M context which is a common deployment target.

Re: GLM 5.2 beats Claude in our benchmarks

#225

Post by scosman »

bArray wrote: post #37

Apparently GLM 5.2 is 753B parameters [1], what kind of hardware are people using to run this locally? [1] https://huggingface.co/zai-org/GLM-5.2

short answer: they mostly aren't

A few people are running highly quantized models with limited context windows. It's still impressive, but not the benchmark level intelligence. Very few people could afford a rig for reasonable local performance at a reasonable quant, at full context size.

The antirez example is 2.6bit quant, 32k context, and few tokens per second... on a ~$7000 MacBook M5 (new RAM pricing).

Re: GLM 5.2 beats Claude in our benchmarks

#226

Post by InvertedRhodium »

cheschire wrote: post #192
Aurornis wrote: post #74

Earlier quoted context omitted.

> 8 X RTX6000. It will run you around 80-100k to get started 8 x RTX6000 GPUs cost $100,000 alone. You then need to build a system that can support those GPUs with enough PCIe lanes through a PCIe switch. It's going to be $120K to $150K to build or buy a system to run this.

Not to mention the three separate dedicated 15A circuits you would need to have installed in order to run the 3x 2000W power supplies running ideally at no more than 1400W sustained load each. And definitely would need 200A service to the house if you have a family living there with you. But hey you could save on heating?

That’s a uniquely US issue - in NZ you can get a 100A single phase at 230V nominal without any issue. 23kw, straight to your door.

A single circuit using 10mm TPS would technically be enough to run what you’re describing. Might be pricey though, I’d probably take the excuse to get 3 phase installed so I could get access to the stock of used 3 phase machinery.

Re: GLM 5.2 beats Claude in our benchmarks

#227

Post by ac29 »

Ldorigo wrote: post #180
dakolli wrote: post #55

Earlier quoted context omitted.

8 X RTX6000. It will run you around 80-100k to get started with a model at this size with decent tps.. Don't worry though, open source evangelists will tell you that these will be running on your phone in the next 3 years. For $100k you could run this model 24/7 through open router with 10 concurrent sessions at 50tps for a decade and have money left over for a vacation. There's no point in investing this type of mon…

How do the economics of your statement work out? Clearly inference providers don't have a time to ROI of 10 years on their hardware costs; and that's without even taking ongoing energy costs into account. What's missing here?

The inference providers are running batch sizes much larger than 10

Re: GLM 5.2 beats Claude in our benchmarks

#228

Post by TurdF3rguson »

maxk42 wrote: post #196
TurdF3rguson wrote: post #105

Earlier quoted context omitted.

It also means giving up vision which I don't know how I would deal with. I think I would prefer a weaker model with vision than a stronger without.

Openrouter definitely supports vision models. Why would you have to give up vision?

For example if I want to paste a screenshot of what I mean, I can't.

Re: GLM 5.2 beats Claude in our benchmarks

#229

Post by bjourne »

gertlabs wrote: post #172
pimeys wrote: post #59

I have taken another look on these open models after the fiasco of Fable and GPT 5.6 this weekend and... GLM-5.2 truly is a good workhorse model for daily programming. I consider myself a heavy user of LLMs and a seasoned developer. A typical session for me with GPT is usually over a hundred dollars... This weekend I programmed a matrix bot with encryption and a Rust agent with some tools. Because I need one and Open…

GLM 5.2 is a great model, but if you only want to use the best model available, it isn't there yet. Every lab releases models that memorize benchmark answers, both intentionally and unintentionally. But we consistently find that models from Chinese labs have a wider gap between public benchmarks and our evaluations, which we designed to be less vulnerable to benchmaxxing. In multi-agent coding environments, GLM 5.2 i…

Man, there is exactly zero information on your site about how your benchmarks work. Why should one trust your numbers when there is no way to verify them?

Re: GLM 5.2 beats Claude in our benchmarks

#230

Post by freakynit »

rockwotj wrote: post #218
skeptic_ai wrote: post #205

Earlier quoted context omitted.

Why Deepseek v4 flash is better than pro in your benchmarks?

I have also found deepseek flash beat pro in some of my own internal evals for tasklet.ai it’s really surprising and I don’t understand it

Same.. although rare, but have observed twice till date.

Some blog post I read few weeks back said that DSV4Flash in xHigh effort beats even the pro model in xHigh effort.

Post reply on HN