231–236 of 236 posts
Earlier quoted context omitted.
What I'd like to know is how well those dual-Epyc machines run the 1.58 bit dynamic quant model. It really does seem to be almost as good as the full Q8.
I tried that that: ~1.5 to 3 tokens/sec.
Earlier quoted context omitted.
But Deepseek R1 doesn't use CUDA, so maybe for this specific case, it isn't a big deal?
The PTX hack is for backend runner and training infra, the public weights are often executed using existing backends. Especially R1-distill-* models are.
Earlier quoted context omitted.
I tried that that: ~1.5 to 3 tokens/sec.
Ouch, thanks. About what I get now on a single-CPU box with 128 GB+a 4090. Was hoping for a major speedup.
Let me know if you find some config that really leverages more cores!
A lot of people build personal deep learning machines. The economics and convenience can definitely work out... I am confused however by "dummy GPU" - I searched for "dummy" for an explanation but didn't find one. Modern motherboards all include an integrated video card, so I'm not sure what this would be for? My personal DL machine has a 24 core CPU, 128GB RAM and 2 x 3060 GPUs and 2 x 2TB NVMe drives in a RAID 1 ar…
Look under "Available Graphics" at https://www.hp.com/us-en/shop/mdp/business-solutions/z440-wo... No integrated graphics. Author's explanation of the problem: The Teslas are intended to crunch numbers, not to play video games with. Consequently, they don't have any ports to connect a monitor to. The BIOS of the HP Z440 does not like this. It refuses to boot if there is no way to output a video signal.
Earlier quoted context omitted.
It is poor form to condemn someone from asking a question. Thank you for providing the information to clear up ignorance though.
this is a question: > is deepseak's use of PTX instead of CUDA relevant here? this is a conclusion/assumption thinly veiled as a question > Deepseek R1 doesn't use CUDA, so ... it isn't a big deal? note, genuine questions don't already presuppose an answer.
As others have said, a high powered Mac could be used for the same purpose at a comparable price and lower power usage. Which makes me wonder: why doesn't Apple get into the enterprise AI chip game and compete with Nvidia? They could design their own ASIC for it with all their hardware & manufacturing knowledge. Maybe they already are.