Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GLM 5.2 beats Claude in our benchmarks

semgrep.dev

Post reply on HN
Print view View original

431–440 of 559 posts

Re: GLM 5.2 beats Claude in our benchmarks

#431

Post by internet_points »

AussieWog93 wrote: post #219
dakolli wrote: post #55

Earlier quoted context omitted.

8 X RTX6000. It will run you around 80-100k to get started with a model at this size with decent tps.. Don't worry though, open source evangelists will tell you that these will be running on your phone in the next 3 years. For $100k you could run this model 24/7 through open router with 10 concurrent sessions at 50tps for a decade and have money left over for a vacation. There's no point in investing this type of mon…

>Don't worry though, open source evangelists will tell you that these will be running on your phone in the next 3 years. Not sure if you're being sarcastic, but I can run a quantised version of Gemma or Qwen on my 16GB M1 Macbook Pro that beats GPT-4 from 2023 hands-down. I wouldn't be surprised if, in another 3 years, you'd be able to run something as powerful as Opus 4.5 or GLM-5.2 on standard consumer hardware - s…

> memory prices coming down

Are they?

I suspect AI labs are buying stuff not just for their own use, but to make local use too expensive to be an option :-( And they can always make the "best" frontier model even bigger (though only fractionally better) so it's always out of reach of local use, while consumer laptops have nearly the same amount of memory they had a decade ago.

    m                  o
    o
    d
    e
    l             o
    s
    i        o
    z    o
    e  2020 2022 2024 2026
    
    
    c                  
    h
    e
    a
    p             o      
    R        o     
    A    o                
    M                   o
       2020 2022 2024 2026

Re: GLM 5.2 beats Claude in our benchmarks

#432

Post by bandrami »

frangonf wrote: post #401
afavour wrote: post #91

Earlier quoted context omitted.

Because car loans can’t be used to buy computers

Surprising that the banking industry has not come up yet with the AI native consumer product loan for GPUs.

Paging Mr. Son. Mr. Son, please pick up line 3.

Re: GLM 5.2 beats Claude in our benchmarks

#433

Post by bandrami »

Ldorigo wrote: post #180
dakolli wrote: post #55

Earlier quoted context omitted.

8 X RTX6000. It will run you around 80-100k to get started with a model at this size with decent tps.. Don't worry though, open source evangelists will tell you that these will be running on your phone in the next 3 years. For $100k you could run this model 24/7 through open router with 10 concurrent sessions at 50tps for a decade and have money left over for a vacation. There's no point in investing this type of mon…

How do the economics of your statement work out? Clearly inference providers don't have a time to ROI of 10 years on their hardware costs; and that's without even taking ongoing energy costs into account. What's missing here?

Inference providers have been getting a firehose of investor cash to keep the chips running (and are looking around very nervously as that firehose starts to sputter).

Re: GLM 5.2 beats Claude in our benchmarks

#434

Post by spaceman_2020 »

Opus 4.8 is genuinely one of the most frustrating models in casual use. It has a tendency to completely lose context in the middle of a conversation. It’s also too pedantic and nitpicky, and relies on language that’s way too specific to get any work done. I always end up being frustrated with it and revert to opus 4.6

Re: GLM 5.2 beats Claude in our benchmarks

#435

Post by dofm »

jfaat wrote: post #262
gertlabs wrote: post #172

Earlier quoted context omitted.

GLM 5.2 is a great model, but if you only want to use the best model available, it isn't there yet. Every lab releases models that memorize benchmark answers, both intentionally and unintentionally. But we consistently find that models from Chinese labs have a wider gap between public benchmarks and our evaluations, which we designed to be less vulnerable to benchmaxxing. In multi-agent coding environments, GLM 5.2 i…

> but if you only want to use the best model available, it isn't there yet I'm trying to wrap my head around exactly why so may people seem to want the best model available when it has recently become clear that most halfway decent models can write damn good code for a fraction of the price. And the frontier models get nerfed constantly so you with open weight you can get something slightly less performant but way mo…

> I'm trying to wrap my head around exactly why so may people seem to want the best model available

This is the logical end point of the fear-based way LLMs are marketed. You must want the best, because everyone who has the best can work faster than you, generate more — if you don't have the best, you are behind! Why would you want to use anything other than the best?

The thing is, once everyone has the best, the question is: how much can you spend? If you can't spend more, you are behind! If spending the most will get you ahead, why would you not want to spend the most, if you can afford it?

There is only one way through this, in the long run: work out a way forward that doesn't make you dependent on this cycle. If you can compete at all, without the spend, what happens is: they burn money and you don't.

FWIW so far I don't think the benchmarks prove very much about the actual experience, and you can discover this just as easily without spending any money. And we know this about benchmarks! Once a benchmark seems useful as a measurement, it becomes a target and it stops being as useful.

I think your strategy is right. It requires bravery, and as you say, it requires ego balance. But I believe it is obvious that the world will either come around to a more sensible, stable pattern or it doesn't matter either way because we're fucked. So opting out of this mad early cycle and choosing to be calmer and happier is a choice you can just make.

Re: GLM 5.2 beats Claude in our benchmarks

#436

Post by grosswait »

jfaat wrote: post #262
gertlabs wrote: post #172

Earlier quoted context omitted.

GLM 5.2 is a great model, but if you only want to use the best model available, it isn't there yet. Every lab releases models that memorize benchmark answers, both intentionally and unintentionally. But we consistently find that models from Chinese labs have a wider gap between public benchmarks and our evaluations, which we designed to be less vulnerable to benchmaxxing. In multi-agent coding environments, GLM 5.2 i…

> but if you only want to use the best model available, it isn't there yet I'm trying to wrap my head around exactly why so may people seem to want the best model available when it has recently become clear that most halfway decent models can write damn good code for a fraction of the price. And the frontier models get nerfed constantly so you with open weight you can get something slightly less performant but way mo…

Because not every problem is a coding problem or not entirely solvable through code. Other tasks include legal, philosophical, financial, investigative, and combinations of these and others.

Re: GLM 5.2 beats Claude in our benchmarks

#437

Post by echelon »

pizzly wrote: post #270
echelon wrote: post #258

Earlier quoted context omitted.

I can't wait for open models to take over in all categories. Sounds like this is the year for coding.

It looks possible open models will. I never expected the reason would be political/legal rather than technical.

The CEOs spent so much time talking about putting everyone out of work and how "unsafe" their models were that the government stepped in with export controls.

They did this to themselves.

Re: GLM 5.2 beats Claude in our benchmarks

#438

Post by cicko »

grosswait wrote: post #436
jfaat wrote: post #262

Earlier quoted context omitted.

> but if you only want to use the best model available, it isn't there yet I'm trying to wrap my head around exactly why so may people seem to want the best model available when it has recently become clear that most halfway decent models can write damn good code for a fraction of the price. And the frontier models get nerfed constantly so you with open weight you can get something slightly less performant but way mo…

Because not every problem is a coding problem or not entirely solvable through code. Other tasks include legal, philosophical, financial, investigative, and combinations of these and others.

It doesn't look like that's where the conversation was going, though.

Re: GLM 5.2 beats Claude in our benchmarks

#439

Post by dofm »

dsrtslnd23 wrote: post #386
jfaat wrote: post #262

Earlier quoted context omitted.

> but if you only want to use the best model available, it isn't there yet I'm trying to wrap my head around exactly why so may people seem to want the best model available when it has recently become clear that most halfway decent models can write damn good code for a fraction of the price. And the frontier models get nerfed constantly so you with open weight you can get something slightly less performant but way mo…

I agree, but there are use cases for the 'best model' other than converting your 1975 stuff to rust: for use cases where LLMs are just getting started to be useful I really want to use the current 'best' model: e.g. CAD, PCB design etc. In particular anything which requires spatial reasoning. The short time I had access to Fable 5 - it was just way better than any other model.

Except that there is no application for AI in CAD that is better, more appropriate, more robust or more sensible than learning how to use a CAD package and doing it yourself.

It's not fast-changing, it's not abstract, it's just not that difficult, and where it is difficult, the AI cannot help you, because it is not capable of things you are capable of.

Learn CAD yourself. Honestly; I was sure I would never manage to learn CAD but it turns out to be interesting, rewarding, valuable and actually quite quick to learn.

An LLM certainly is not going to be able to do it better than you once you have a tiny bit of experience. (PCB design, perhaps, has a language to it that an LLM can make a bit more headway into, but as a non-PCB-designer I would still bet that it's more like CAD than code)

Re: GLM 5.2 beats Claude in our benchmarks

#440

Post by cicko »

kennywinker wrote: post #382
rolisz wrote: post #342

Earlier quoted context omitted.

With Claude Code Ultrathink, I used 3 million tokens in 20 minutes. At API prices, that would be around 30$. So 90$/h. Model cost is not that much lower.

x40hrs/week * 50 weeks = $180k Congrats, now you’re paying an engineer’s salary to make your engineer at best 20% more productive. Better to hire another engineer, or two jrs, and build up your in house talent.

except this is way more than an engineering salary. At least in Europe.
Post reply on HN