Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GLM-5.2 – How to Run Locally

unsloth.ai

Post reply on HN
Print view View original

321–328 of 328 posts

Re: GLM-5.2 – How to Run Locally

#321

Post by culopatin »

throwawayffffas wrote: post #204
discordance wrote: post #151

Earlier quoted context omitted.

Running that full load is at least 600 W, so in a day ~14 kWh. At $0.2 a kWH, that would be $2.80/day or $1k a year of op-ex in electricity. Unless you really want privacy or the fuzzy feeling of owning your own, it’s cheaper, more convenient and has much faster tok/s if you pay a hyper scaler. That said, I do like the direction we are heading and look forward to seeing what host your own hardware we get in 2 years.

So in my experience with 2 7900XTs with models that sit fully in VRAM it's more like 400W the gpus spend a lot of time waiting for each other.

How’s running models with AMD these days?

Re: GLM-5.2 – How to Run Locally

#322

Post by 3s »

itemize123 wrote: post #308
traceroute66 wrote: post #202

Earlier quoted context omitted.

> We do want privacy, and we also want to own the hardware so the US can't just turn it off whenever it feels like it. I agree and I prefer on-prem where possible. The Apple Mac Studios have been great for that although I don't have enough of them to run GLM-5.2 without heavy quantization. I'm also waiting for the Apple next product refresh which I hope will enable me to do more with less. Meanwhile there are hosted…

tinfoil doesn't seem to distinguish between cached and input tokens. I think that would make it untenably expensive

Hi, cofounder of Tinfoil here. We're working on implementing prompt caching. It just requires more work than with traditional, non-private inference and requires some careful considerations to actually implement it.

Re: GLM-5.2 – How to Run Locally

#323

Post by Astropolak »

dxuh wrote: post #139
segmondy wrote: post #84

I run Q4_K_XL. All it takes to run to get about 6tk/sec is 512gb of ram and 2 3090 GPUs with llama.cpp -cmoe. I also have crappy DDR4, 2400mhz, 3200mhz will bring that speed up to about 9tk/sec. I also have ok 32core epyc CPU, a better 64core would bring it up to about 11tk/sec. I did a budget build before the crazy hardware cost and I regret it everyday. Nevertheless, it's fantastic being able to run this model at h…

"All it takes to run" might be fair if you paid $2400, but right now the total price is way closer to $10k (almost 5k for the RAM and 2k each for the GPUs). Today that is a lot of expensive hardware.

Agree'd here. I am seeing a pattern in this thread - Everyone here seems to justify paying substantially more money to self-host hardware thinking they are saving in the medium-long term. Frankly from the perspective of the company I'm with - lost opportunity cost is more important than trying to future proof on a hunch that AI vendors are going to pull the board out under you. There's alot of competition in this space and we're seeing these frontier models continue to get better for the foreseeable future. As such, you're dumping alot of money upfront on hardware which is not guaranteed to be relevant in a few years time. The benefits to me seem pretty speculative, minus the point of security.

Re: GLM-5.2 – How to Run Locally

#324

Post by tatersolid »

traceroute66 wrote: post #223
patates wrote: post #216

Earlier quoted context omitted.

> Apple next product refresh I have this feeling that it'll be very expensive and still scarce. Normally I wouldn't say this about Apple, because their pricing is part of their brand, but this time the demand (both by data-centers and prosumers) is the force majeure.

> because their pricing is part of their brand I know people usually say that about Apple, but to be fair to them on this occasion they have not hiked up their prices yet because they are clearly at present still under some old deals that they did a good job negotiating. However, of course, at some point Apple will run out of both inventory and old-pricing manufacturing capacity. Yes, I am fully expecting some sort o…

> they have not hiked up their prices yet

That one aged poorly. Apple massively hiked prices 2 days later, by thousands of USD on some RAM-heavy configurations.

https://news.ycombinator.com/item?id=48672732

Re: GLM-5.2 – How to Run Locally

#325

Post by niutech »

SalariedSlave wrote: post #265
leansensei wrote: post #263

Earlier quoted context omitted.

Europe was never at the frontier of any tech? Huh what now?

A hyperbole born of frustration, I admit. Qualify it to software, rather than all tech, if you will.

How about SAP, Dassault Systemes, Amadeus, Hexagon, Visma, Wolters Kluwer, Sage, ...?

Re: GLM-5.2 – How to Run Locally

#326

Post by adastra22 »

niutech wrote: post #325
SalariedSlave wrote: post #265

Earlier quoted context omitted.

A hyperbole born of frustration, I admit. Qualify it to software, rather than all tech, if you will.

How about SAP, Dassault Systemes, Amadeus, Hexagon, Visma, Wolters Kluwer, Sage, ...?

Are any of those at the frontier?

Re: GLM-5.2 – How to Run Locally

#327

Post by adastra22 »

horsawlarway wrote: post #102
gruez wrote: post #81

Earlier quoted context omitted.

>I feel like part of the reason for the relative stagnation in hardware over the last twenty years was simply the lack of use cases to justify hardware refreshes by businesses. No, we're running into limits of moore's law, and it's showing in prices for new nodes, where they're getting denser but not cheaper.

It's true we hit limits, but I feel like a lot of it was "limits" in the sense that the tradeoff stopped being worth the cost, so we optimized in other areas. So we hit limits on clock speed in the early 2000s (ex - the 4ghz wall) but it also turned out that mobile as the driver for sales meant no one really cared much about clock speed compared to performance/watt. Clock speed mattered, but only relative to how many…

Physical limits means the limits of physics. You can’t go much faster than 4GHz because light only moves 7.5 cm per clock cycle at that speed.

Re: GLM-5.2 – How to Run Locally

#328

Post by adastra22 »

gerdesj wrote: post #300
mapontosevenths wrote: post #52

Earlier quoted context omitted.

I have one, and I love it. That said my buddies Mac smokes it for inference workloads in terms of tokens per second AND its more usable for other things. If you are training and doing research it's great, if you want to cluster them it cant be beat, but if you just want local inference on a single box buy a mac or even a strix halo device.

Get your buddy with his smoking Mac to allow multiple concurrent connections and see how it gets on compared to your Spark. Don't ever use a single "chat" test to derive performance - try running say 10 or more. You might also notice that your Spark has a pair of QSFP28 or DD (not sure yet) type interfaces as well as the 10Gb/s ethernet - that network card is a right old beast and adds quite a lot to the cost. It is…

The Mac has TB5 clustering with RDMA.
Post reply on HN