I'm now staring at a 10 year old 4U with 256 GB of DDR4 and thinking hmmmmm
A 10 year old Xeon is all you need
21–30 of 301 posts
Re: A 10 year old Xeon is all you need
#22Might consider going for even older CPUs which don't have the Intel ME ring -3 thing which is full of backdoors
Re: A 10 year old Xeon is all you need
#23Earlier quoted context omitted.
(purple on black is really hard to read) You say it runs "at reading speed". Have you benchmarked it?
> (purple on black is really hard to read) Noted, and agree (it looks like it has also already been clicked, which I dislike). I honestly I need to redo the themes. > You say it runs "at reading speed". Have you benchmarked it? At some point a few weeks ago, yes I think so, but I didn't write it down for some reason... so I'll have to find a time when it's not busy and do it again without a noisy system. Right now th…
I am pretty sure llamacpp have their own benchmarking binary that you can use.
Re: A 10 year old Xeon is all you need
#24How about the iMac Pro? Would that work? I was able to put 128gb in it (not as easy as the regular iMac but possible).
I've been running various models on a Mac Pro 2013 (8 cores, 32 GB RAM) at about 8 to 10 t/s for months. It's not fast, but it's more than enough for many actual tasks, in particular background tasks. An iMac pro will do just as well I suppose.
Re: A 10 year old Xeon is all you need
#25Hi HN. I wrote this post after getting frustrated by the lack of ways to run the new Gemma 4 Drafter models, and mainstream tools not prioritizing this, and hiding all the performance levers. I ended up getting a modern 26B MoE model (Gemma 4) running at reading speed on an old recycled server with a single Xeon E5-2620 v4 and 128GB of DDR3 RAM (and no GPU). It took a lot of work, but it actually worked out somehow.…
How many watts is that setup? Cool you got it to work, but maybe only useful for vintage / retro computing rather than practical if the energy consumption makes it economically wasteful.
Re: A 10 year old Xeon is all you need
#26> The argument for speculative decoding is stronger on CPU than on GPU.
Uh. Uuuh.
No?
___
Also
> While a GPU has a massive pool of ultra-fast High-Bandwidth Memory (HBM), a CPU relies on small, lightning-fast “caches” (L1, L2, L3) built directly onto the processor chip.
What purpose does the quoting of "caches" serve there? Is this AI writing written by that model running on that host?
Re: A 10 year old Xeon is all you need
#27What kind of tokens per second did the op get I saw nothing of this written.
Re: A 10 year old Xeon is all you need
#28Makes you wonder if its possible to squeeze more tps out of a strix halo system using the 16 zen5 cores as well as the gpu.
In general you’re mem bandwidth constrained so cpu vs gpu often ends up similar on APUs
There are ways to trade off compute power for memory bandwidth (like MTP and other speculative decoding approaches). The CPU and GPU would need to be able to share the same cache for this to work. In the Strix Halo case the GPU has a private cache on the GPU die I think, which is the snag.
Re: A 10 year old Xeon is all you need
#29I have an ancient DDR3 Xeon that doesn't support any AVX (dual x5690 and 96GB 1333 MHz RAM). You reckon it would even build / run at all?
I run Win 11 Enterprise on an el cheapo spare parts Xeon E3-1275 V2 + 32 GiB DDR3-2133 + Gigabyte GA-B75M-D3H rev. 1.2 (TPM support)
Re: A 10 year old Xeon is all you need
#30What kind of tokens per second did the op get I saw nothing of this written.
11.94 tokens/sec (from another answer above)