91–100 of 124 posts
Earlier quoted context omitted.
I was about to correct you because this doesn't use PCIe for anything, and then I realized Arc was a GPU (and they support up to 8 per machine). Any idea how many Arc's it takes to match an H100?
I am reading from time to time about multi GPU solution and last time I found some real life information about this (it was two 7900 xtx) the result was that performance is the same at best often it is slower. So even if you manage to slap like 8 cheap cards onto motherboard, even if you would somehow make it work (people have problems with such setups), even if this would work continuously without much problems (cra…
The only thing which doesn't work well is running on iGPUs. It might work but it's very unstable.
I'm sure this question has been asked before, but why not launch a GPU with more but slower ram? That would fit bigger models while still affordable...
That's Nvidia's current MO. There's more demand for GPUs for AI than there are GPUs available, and most of that demand still has stupid amounts of money behind it (being able to get grants, loans or investment based on potential/hype) - money that can be captured by GPU vendors. Unfortunately, VRAM is the perfect discriminator between "casual" and "monied" use.
(This is not unlike the "SSO tax" - single sign-on is pretty much the perfect discriminator between "enterprise use" and "not enterprise use".)
Earlier quoted context omitted.
I am reading from time to time about multi GPU solution and last time I found some real life information about this (it was two 7900 xtx) the result was that performance is the same at best often it is slower. So even if you manage to slap like 8 cheap cards onto motherboard, even if you would somehow make it work (people have problems with such setups), even if this would work continuously without much problems (cra…
Ollama works fine with multi-gpu setups. Since rocm 6.3 everything is stable and you can mix different GPU generations. The performance is good enough for the models to be useful. The only thing which doesn't work well is running on iGPUs. It might work but it's very unstable.
As other commenters have mentioned, the performance of this set up is probably not really great since there's not enough VRAM and lots of bits have to be moved between CPU and GPU RAM. That said, there are sub-256GB quants of DeepSeek-R1 out there (not the distilled versions). See https://unsloth.ai/blog/deepseekr1-dynamic I can't quantify the difference between these and the full FP8 versions of DSR1, but I've been…
Earlier quoted context omitted.
It better be! AMD @ $2k: https://digitalspaceport.com/how-to-run-deepseek-r1-671b-ful...
What a teaser article! All this info for setting up the system, but no performance numbers.
https://github.com/intel/ipex-llm/tree/main?tab=readme-ov-fi...
Earlier quoted context omitted.
CPU inference is both bandwidth and compute constrained. If your prompt has 10 tokens, it’ll do ok, like in the LinkedIn demo. If you need to increase the context, compute bottleneck will kick in quickly.
Prompt length mainly impacts prefill latency (FTFF), not the decoding speed (TPOT)