Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GLM 5.2 beats Claude in our benchmarks

semgrep.dev

Post reply on HN
Print view View original

171–180 of 559 posts

Re: GLM 5.2 beats Claude in our benchmarks

#171

Post by aetch »

krackers wrote: post #68
dakolli wrote: post #55

Earlier quoted context omitted.

8 X RTX6000. It will run you around 80-100k to get started with a model at this size with decent tps.. Don't worry though, open source evangelists will tell you that these will be running on your phone in the next 3 years. For $100k you could run this model 24/7 through open router with 10 concurrent sessions at 50tps for a decade and have money left over for a vacation. There's no point in investing this type of mon…

Would you be better off pooling that money with some hackerspace group and then setting up shared inference infra, so that way you at least get better utilization?

You can then rent spare capacity out to people on a subscription or token basis ….wait

Re: GLM 5.2 beats Claude in our benchmarks

#172

Post by gertlabs »

pimeys wrote: post #59

I have taken another look on these open models after the fiasco of Fable and GPT 5.6 this weekend and... GLM-5.2 truly is a good workhorse model for daily programming. I consider myself a heavy user of LLMs and a seasoned developer. A typical session for me with GPT is usually over a hundred dollars... This weekend I programmed a matrix bot with encryption and a Rust agent with some tools. Because I need one and Open…

GLM 5.2 is a great model, but if you only want to use the best model available, it isn't there yet. Every lab releases models that memorize benchmark answers, both intentionally and unintentionally. But we consistently find that models from Chinese labs have a wider gap between public benchmarks and our evaluations, which we designed to be less vulnerable to benchmaxxing.

In multi-agent coding environments, GLM 5.2 is just shy of Opus 4.6 on average. Data at https://gertlabs.com/rankings

But when factoring in performance/cost, GLM 5.2 is the frontier model.

Re: GLM 5.2 beats Claude in our benchmarks

#173

Post by tiahura »

kibwen wrote: post #127
JumpCrisscross wrote: post #106

Earlier quoted context omitted.

> I do not need the whole model. I don't need the model to have knowledge of the rain history of Algeria nor how many colors are in the Russian flag Isn’t the performance gap between quantized and full models indicative that even if you aren’t using it directly, the model knowing the colors in the Russian flag does have something to do with the intelligence you demand?

Quantizing is one thing. But in general it's self-evident that training the model on information that is irrelevant to your use case does not necessarily improve ability, otherwise you'd have AGI just from reinforcing your model on memorizing the first 10^50 digits of pi. Likewise, LLMs do not violate the laws of information theory, and therefore the only way to encode X amount of information in Y amount of bits wher…

Apparently irrelevant data can help because model weights are entangled.

Re: GLM 5.2 beats Claude in our benchmarks

#174

Post by richardlblair »

rgbrenner wrote: post #19
solenoid0937 wrote: post #3

GLM export controls incoming? I predict Commerce will force OpenRouter, HuggingFace to take some open models down within the next few months. Not that it would make any sense.

If that happens it'll be an absolute disaster. Imagine a scenario where Anthropic and OpenAI prohibit most US companies from using their latest models because of safety.. And meanwhile attackers use equivalent open source models to attack US companies. Any prohibition on open source models will do nothing to fix the problem.. since attackers will never feel bound to the law. All advanced models must be available for…

And someone will start a competing company in a sane environment.

Re: GLM 5.2 beats Claude in our benchmarks

#175

Post by throw10920 »

_s_a_m_ wrote: post #129

I tried GLM many times and it is bad, i have on clue what these people are talking about

Bad for security research or for general coding?

Having used GLM 5.2 for non-security software work, I can say it's better than Sonnet (but not Opus), and cheaper than both (because when you steal someone else's IP, you don't have to amortize the cost of their R&D).

Re: GLM 5.2 beats Claude in our benchmarks

#177

Post by randomNumber7 »

theteapot wrote: post #166
jliptzin wrote: post #150

Earlier quoted context omitted.

Yea as far has hobbies go, I feel like this is on the low end. I know people who collect watches and corvettes, that's way more expensive and functionally you can't really do anything special with them.

The difference is watches and corvettes typically appreciate in value, where as computer hardware typically drops like a rock.

Also LLMs are mainly used for work and if you can spend 6 digits on watches your likely financially independent.

Re: GLM 5.2 beats Claude in our benchmarks

#178

Post by Retro_Dev »

bArray wrote: post #37

Apparently GLM 5.2 is 753B parameters [1], what kind of hardware are people using to run this locally? [1] https://huggingface.co/zai-org/GLM-5.2

I ran it on my laptop, which is a Lenovo Legion 5i (think 32 GB RAM, 4060 w/ 8 GB VRAM, you get the picture). It was a quantized model (otherwise it would not fit on my NVMe 1TB drive) at 4 bits per weight - UD_Q4_K_XL. It ran at about 12 seconds per token (not tokens per second). A fun project, but not worth it. I used 4096 tokens of context cache, and I ran it with llama.cpp - as it supports memory mapping. Because the whole thing could obviously not fit in RAM, I was curious how much it would need to stream from SSD. The answer? For a simple 4 sentence description of who it was, about 1.5 TiB was streamed from disk.

Re: GLM 5.2 beats Claude in our benchmarks

#179

Post by skissane »

bardak wrote: post #124
skissane wrote: post #110

Earlier quoted context omitted.

> They could ban payment processors from processing payments to any hosts of GML 5.2 Can they actually though? Do they have legal authority to tell a payment processor that it has to block transactions of a legal US company, just because the company is hosting a Chinese-developed open source model? I’m sceptical And what about companies (e.g. AWS) that let you “bring your own model”?

It would be extremely heavy handed but the administration has sanctioned the International Criminal Court judges such that they basically have no access to the Wests modern financial system. I think domestic US providers would have to deal with different ways but someone like Herzner could easily be cut off from the financial system if the administration doesn't feel that they are adequately blocking the model

> It would be extremely heavy handed but the administration has sanctioned the International Criminal Court judges

That's sanctioning specific individuals for specific acts they performed which the US claims contravene its interests and those of its allies.

I don't agree with the ICC sanctions, but it really can't be compared with the proposal "sanction any company, even US domestic entities, which use a Chinese-developed open source model".

In fact, I think part of what enables the US to sanction them (under US law) is the fact they are neither US citizens nor residents; if they were US citizens living in the United States, I don't think the President would have the legal authority to impose those kinds of sanctions.

They could sanction Hetzner–because it is a German firm based in Germany. I don't see how they could sanction a US firm based in the US whose owners and staff were US citizens.

Also, the 5th Circuit Court of Appeal decision Van Loon v Treasury (Nov 2024) is relevant–it held that IEEPA (the law used to sanction ICC officials) couldn't be used to sanction the Tornado Cash smart contract system, since open source code wasn't "foreign property" under IEEPA.

Re: GLM 5.2 beats Claude in our benchmarks

#180

Post by Ldorigo »

dakolli wrote: post #55
bArray wrote: post #37

Apparently GLM 5.2 is 753B parameters [1], what kind of hardware are people using to run this locally? [1] https://huggingface.co/zai-org/GLM-5.2

8 X RTX6000. It will run you around 80-100k to get started with a model at this size with decent tps.. Don't worry though, open source evangelists will tell you that these will be running on your phone in the next 3 years. For $100k you could run this model 24/7 through open router with 10 concurrent sessions at 50tps for a decade and have money left over for a vacation. There's no point in investing this type of mon…

How do the economics of your statement work out? Clearly inference providers don't have a time to ROI of 10 years on their hardware costs; and that's without even taking ongoing energy costs into account. What's missing here?
Post reply on HN