Earlier quoted context omitted.
For 10k you can buy a used dual socket Intel or amd based rackmount server with a terabyte of ram, and run models on cpu only at a reasonable speed. Same server would have been 4-5k a couple years ago before ram price rise. Or buy one on eBay with 512GB that has half its slots populated and then buy the matching 512GB kit to add.
Which CPU gen are you suggesting, is there any writeup on such setup where In my experience with rig half that cost, entire exercise of running coding models locally has been a huge disappointment. Cost/Value when compared to cloud services is just not there, but I see the merit for those who value privacy over quality of output and want a backup of huge condensed corpus of data within their control. Kudos to OP thou…
Re: Show HN: Getting GLM 5.2 running on my slow computer
#171I would suspect that one would buy based on mem-bus & PCIe bus speeds more than CPU for this, and just dial down the CPU parameters to save power. Most of the time and power will be consumed by memory and bus transfers because the CPU will mostly be waiting to the right set of weights and factors to multiply.