Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Benchmarking 15 “E-Waste” GPUs with Modern Workloads

esologic.com

Post reply on HN
Print view View original

11–20 of 68 posts

Re: Benchmarking 15 “E-Waste” GPUs with Modern Workloads

#11

Post by russianGuy83829 »

eso_logic wrote: post #10
russianGuy83829 wrote: post #9

Have you tried 27B class models like qwen3.6?

This initial round of benchmarking was to understand if there was any usecase here at all and I think there is. In a follow up, I'll be trying to answer questions like this. How big of a model can you fit on 4x M60, 4x P100, 4x V100? What are the tok/second when varying context length? Do you have a set of models you'd like me to look at?

That's great. Personally, I'd interested in Qwen3.6-27B and deepseek V4 flash (or pro), with contexts above 60k. They seem to be popular and have good coding performance. I'd appreciate numbers on a single or two GPUs where a quantized version fits reasonably into the VRAM (Qwen in 16 or 24GB). 4 older GPUs approach a used 3090 in price, and the 3090 has better support for speedups like MTP. So cheaper but slower looks like a reasonable target to me.

Re: Benchmarking 15 “E-Waste” GPUs with Modern Workloads

#12

Post by eso_logic »

russianGuy83829 wrote: post #11
eso_logic wrote: post #10

Earlier quoted context omitted.

This initial round of benchmarking was to understand if there was any usecase here at all and I think there is. In a follow up, I'll be trying to answer questions like this. How big of a model can you fit on 4x M60, 4x P100, 4x V100? What are the tok/second when varying context length? Do you have a set of models you'd like me to look at?

That's great. Personally, I'd interested in Qwen3.6-27B and deepseek V4 flash (or pro), with contexts above 60k. They seem to be popular and have good coding performance. I'd appreciate numbers on a single or two GPUs where a quantized version fits reasonably into the VRAM (Qwen in 16 or 24GB). 4 older GPUs approach a used 3090 in price, and the 3090 has better support for speedups like MTP. So cheaper but slower loo…

No problem. Varying context size is a common request I've been getting as well. Personally I'm looking forward to seeing how much we can cram into the ancient K80's 24GB of VRAM :0

Re: Benchmarking 15 “E-Waste” GPUs with Modern Workloads

#15

Post by eso_logic »

rrhjm53270 wrote: post #14

Would it possible to stack up to 16x32GB VRAM, and test the performance of a MOE model such as Deepseek-v4-flash?

16 GPUs would require one or more 220V breaker panels, more akin to an EV charger than a computer. You would also quickly run out of PCIe lanes. My goal with this benchmarking is to think about what is the most cost effective way to fill 4U.

Re: Benchmarking 15 “E-Waste” GPUs with Modern Workloads

#16

Post by NortySpock »

russianGuy83829 wrote: post #11
eso_logic wrote: post #10

Earlier quoted context omitted.

This initial round of benchmarking was to understand if there was any usecase here at all and I think there is. In a follow up, I'll be trying to answer questions like this. How big of a model can you fit on 4x M60, 4x P100, 4x V100? What are the tok/second when varying context length? Do you have a set of models you'd like me to look at?

That's great. Personally, I'd interested in Qwen3.6-27B and deepseek V4 flash (or pro), with contexts above 60k. They seem to be popular and have good coding performance. I'd appreciate numbers on a single or two GPUs where a quantized version fits reasonably into the VRAM (Qwen in 16 or 24GB). 4 older GPUs approach a used 3090 in price, and the 3090 has better support for speedups like MTP. So cheaper but slower loo…

Similar interest here, possibly including if qwen 3.6, Gemma4 or DiffusionGemma (with the largest quants that will fit in a single card) will offer, say, 50 tokens-per-second (fast enough for interactive human-in-the-loop code research, print-f iterations on code to debug things, etc; or let the LLM churn on a problem for a minute while I step out to handle something else), context of up to 200k preferred.

Also if nothing else the below project lets you use an NVidia graphics card as low-latency swap, which has been nice as a buffer as RAM prices remain high and leaves me eyeing that 24GB card you mentioned as an alternative...

https://github.com/c0deJedi/nbd-vram

Re: Benchmarking 15 “E-Waste” GPUs with Modern Workloads

#17

Post by SillyUsername »

No mention of the venerable Tesla P4. 75W peak, 8GB VRAM, about $80 (£60).

I have 6x P4s, a Xeon E5 2696v3 (36 threads, 3.8ghz peak but all core turbo unlocked, so 6 cores at 3.8Ghz - about 8 cores at 3.5ghz, or all cores at 3.1ghz), 48GB DDR4, all fit into a micro atx case running on a 650W MSI psu. This gives me a virtual 48GB GPU (llama.cpp ftw) to backup that 48GB of RAM.

I typically see scores of at least 7-12t/s on 20-30B Q4KM size dense models, on a 32K/48K/64K context, adequate for modern inference.

The pain point is the prompt loading, it is far far slower, minutes not seconds, than modern tensor core 8GB 5060s (my other machine's 2x GPUs) but is quite similar in regular inference speed once it has loaded.

Re: Benchmarking 15 “E-Waste” GPUs with Modern Workloads

#19

Post by eso_logic »

SillyUsername wrote: post #17

No mention of the venerable Tesla P4. 75W peak, 8GB VRAM, about $80 (£60). I have 6x P4s, a Xeon E5 2696v3 (36 threads, 3.8ghz peak but all core turbo unlocked, so 6 cores at 3.8Ghz - about 8 cores at 3.5ghz, or all cores at 3.1ghz), 48GB DDR4, all fit into a micro atx case running on a 650W MSI psu. This gives me a virtual 48GB GPU (llama.cpp ftw) to backup that 48GB of RAM. I typically see scores of at least 7-12t/…

Yes! P4 and the other small cards are fantastic. I've been more focused on developing a cooling system around the 2 slot sized cards so I don't have any of these lying around. Are you using P4 for anything outside of LLM work? I'm interested in seeing their image processing capabilities.

Re: Benchmarking 15 “E-Waste” GPUs with Modern Workloads

#20

Post by riedel »

latchkey wrote: post #3

Darn, I was hoping to see bc-250's (aka PS5 chips) in there. They've recently become popular for inference and they are only about $200 on ebay. They hold a special place in my heart because I deployed 20k of them and I'm glad to see they are finding a purpose now and not just e-waste.

Cool stuff. Just read: https://github.com/akandr/bc250
Post reply on HN