> Runing GLM-5.2 on local hardware Do the runes make it smarter or just run faster (or both)?
depends on the color
GLM-5.2 – How to Run Locally
141–150 of 328 posts
Re: GLM-5.2 – How to Run Locally
#142I feel like the gap is closing to be able to run good enough models locally even for coding and I would assume it could make some companies a bit nervous. Am I wrong about that?
locally on what hardware? something like the new dgx spark, ryzen halo, or mac studio will cost you ~ $4k plus whatever you pay for power. at the rate AI is currently progressing, i think you'd be optimistic to consider that as having a 2 year depreciation. for $4k, you can get 20 months of claude max 200. i'd take claude over the hardware. anthropic will have something to worry about when you can run a local model o…
> at the rate AI is currently progressing, i think you'd be optimistic to consider that as having a 2 year depreciation.
How so? Model capability at a fixed hardware level has been consistently (and rapidly) increasing. You might or might not be able to run state of the art 2 (or 4 or whatever) years from now but you can reasonably expect the hardware to last upwards of a decade with model performance consistently improving over that time frame.
You can get a tolerable (at least by some metrics) experience using 10 year old hardware today.
Re: GLM-5.2 – How to Run Locally
#143I run Q4_K_XL. All it takes to run to get about 6tk/sec is 512gb of ram and 2 3090 GPUs with llama.cpp -cmoe. I also have crappy DDR4, 2400mhz, 3200mhz will bring that speed up to about 9tk/sec. I also have ok 32core epyc CPU, a better 64core would bring it up to about 11tk/sec. I did a budget build before the crazy hardware cost and I regret it everyday. Nevertheless, it's fantastic being able to run this model at h…
How can you combine CPU cores and multiple GPU? Are you running some layers in cpu, others in gpu #1, and others in gpu #2? What about the bandwidth and latency between them? Or maybe the model itself only runs at gpus, and the cpu memory only store the weights for experts not corrently activated? If so, then what's the 32 or 64 cpu cores for? I'm a big fan of fully utilizing one's hardware and it's kinda sad that it…
This is a good place to start reading about dual gpus.
https://github.com/noonghunna/club-3090/blob/master/docs/DUA...
Re: GLM-5.2 – How to Run Locally
#144I feel like the gap is closing to be able to run good enough models locally even for coding and I would assume it could make some companies a bit nervous. Am I wrong about that?
locally on what hardware? something like the new dgx spark, ryzen halo, or mac studio will cost you ~ $4k plus whatever you pay for power. at the rate AI is currently progressing, i think you'd be optimistic to consider that as having a 2 year depreciation. for $4k, you can get 20 months of claude max 200. i'd take claude over the hardware. anthropic will have something to worry about when you can run a local model o…
You can get a 128GB Strix Halo for under $3k. Used to be under $2k. Even if you believe it'll be completely obsolete for AI in two years, it'll still be good for many other things. Games for at least several more years, a great home server and/or desktop almost indefinitely. Plus, we might actually reach good enough levels for some AI use cases, if we're not already there.
And never underestimate the potential for enshittification. Your local rig will only deliver better performance over time as more and more tweaks come out. With cloud services expect the opposite to happen as subsidies run out. It's entirely possible that they will intersect on a bang per buck basis within two years.
Re: GLM-5.2 – How to Run Locally
#145I run Q4_K_XL. All it takes to run to get about 6tk/sec is 512gb of ram and 2 3090 GPUs with llama.cpp -cmoe. I also have crappy DDR4, 2400mhz, 3200mhz will bring that speed up to about 9tk/sec. I also have ok 32core epyc CPU, a better 64core would bring it up to about 11tk/sec. I did a budget build before the crazy hardware cost and I regret it everyday. Nevertheless, it's fantastic being able to run this model at h…
"All it takes to run" might be fair if you paid $2400, but right now the total price is way closer to $10k (almost 5k for the RAM and 2k each for the GPUs). Today that is a lot of expensive hardware.
512gb 2400mhz ddr4 ram = $1600 not $5000. https://www.ebay.com/itm/188284985172
You can get creative and source 2-3 2080ti 22gb from China for about $250 a piece. You can either be resourceful and find a way or find a whole bunch of excuses.
Re: GLM-5.2 – How to Run Locally
#146Can someone explain the math to me? Why is 1-bit only ten percent less memory than 2-bit?
Re: GLM-5.2 – How to Run Locally
#147Earlier quoted context omitted.
Crossing my fingers that this boom jumpstarts 90's like improvements in computing hardware. I feel like part of the reason for the relative stagnation in hardware over the last twenty years was simply the lack of use cases to justify hardware refreshes by businesses. Most of the money and energy went to mobile for the last fifteen years. Affordable local inference might be the gravy train the server, desktop, and lap…
Definitely the stagnation was due to a lack of use cases, but this isn't a bad thing. We don't need most of the hardware advancement we got. Business hardware got beefier because businesses demanded more data (or more specifically: the industry told businesses they needed more data), with no idea of what to actually do with it once they got it. To get all that data, bandwidth needed to be increased, with more iops to…
The natural progression when performance is enough would be price. We were starting to see that but not anymore. I wonder if somebody is afraid the future where generally useful computation is cheap.
Re: GLM-5.2 – How to Run Locally
#148Earlier quoted context omitted.
How can you combine CPU cores and multiple GPU? Are you running some layers in cpu, others in gpu #1, and others in gpu #2? What about the bandwidth and latency between them? Or maybe the model itself only runs at gpus, and the cpu memory only store the weights for experts not corrently activated? If so, then what's the 32 or 64 cpu cores for? I'm a big fan of fully utilizing one's hardware and it's kinda sad that it…
This is a good place to start reading about dual gpus. https://github.com/noonghunna/club-3090/blob/master/docs/DUA...
But in this case he used a cpu too
Re: GLM-5.2 – How to Run Locally
#149Earlier quoted context omitted.
I have been putting up with it forever. We are spoiled by MixtureOfExperts. Folks were delighted to run llama3-70B at such speed. We were happy with 15-20tk/sec with 8b models, and if you could run llama3-405B at 1tk/sec you were a god. To each their own. I can live with 6 high quality tokens. If I could get a Fable equivalent model, I'll gladly take 2tk/sec if that's what it took to run it locally.
But what is it doing for you that you couldn’t do yourself at that speed? I‘m really curious and on the fence of partly going local.
Run one task, while you do another? Or while you sleep / eat / rave?
Re: GLM-5.2 – How to Run Locally
#150Can someone explain the math to me? Why is 1-bit only ten percent less memory than 2-bit?
Keyword dynamic, the parameters are quantized on a case by case basis