Are there any Intel GPUs with a lot of vRAM that someone could recommend that would work with this?
For consumer stuff there's the Intel Arc A770 with 16GB VRAM. More than that and you start moving into enterprise stuff.
PyTorch Library for Running LLM on Intel CPU and GPU
11–20 of 98 posts
Re: PyTorch Library for Running LLM on Intel CPU and GPU
#12Earlier quoted context omitted.
No, but for consumers they're a great offering. 16GB RAM and performance around a 4060ti or so, but for 65% of the price
and 65% of the software support, less I'm inclined to believe? Although having more players in the fold is definitely a good thing.
Intel is historically really good at the software side, though.
For all their hardware research hiccups in the last 10 years, they've been delivering on open source machine learning libraries.
It's apparently the same on driver improvements and gaming GPU features in the last year.
Re: PyTorch Library for Running LLM on Intel CPU and GPU
#13Are there any Intel GPUs with a lot of vRAM that someone could recommend that would work with this?
There's the Max GPU (Ponte Vecchio), their datacentre offering, with 128GB of HBM2e memory, 408 MB of L2 cache, and 64 MB of L1 cache. Then there's Gaudi, which has similar numbers but with cores specific for AI workloads (as far as I know from the marketing).
You can pick them up in prebuilds from Dell and Supermicro: https://www.supermicro.com/en/accelerators/intel
Read more about them here: https://www.servethehome.com/intel-shows-gpu-max-1550-perfor...
Re: PyTorch Library for Running LLM on Intel CPU and GPU
#14Any performance benchmark against 'llamafile'[0] or others? [0] - https://github.com/mozilla-Ocho/llamafile
You can already use intel GPUs (both ARC and iGPUS) with llama.cpp on a bunch of backends:
- SYCL [1]
- Vulkan
- OpenCL
I don't own the hardware, but I imagine SYCL is more performant for ARC , because it's the one intel is pushing for their datacenter stuff
[1]: https://www.intel.com/content/www/us/en/developer/articles/t...
Re: PyTorch Library for Running LLM on Intel CPU and GPU
#15The company that did 4-cores-forever, has the opportunity to redeem itself, in its next consumer GPU release, by disrupting the "8-16GB VRAM forever" that AMD and Nvidia have been imposing on us for a decade. It would be poetic to see 32-48GB at a non-eye-watering price point.
Intel definitely seems to be doing all the right things on software support.
Re: PyTorch Library for Running LLM on Intel CPU and GPU
#16[dead]
Re: PyTorch Library for Running LLM on Intel CPU and GPU
#17The company that did 4-cores-forever, has the opportunity to redeem itself, in its next consumer GPU release, by disrupting the "8-16GB VRAM forever" that AMD and Nvidia have been imposing on us for a decade. It would be poetic to see 32-48GB at a non-eye-watering price point. Intel definitely seems to be doing all the right things on software support.
What is obvious to us, is an industry standard to Product Managers. When is the last time you have seen an industry player upset the status quo? Intel has not changed that much.
Re: PyTorch Library for Running LLM on Intel CPU and GPU
#18The company that did 4-cores-forever, has the opportunity to redeem itself, in its next consumer GPU release, by disrupting the "8-16GB VRAM forever" that AMD and Nvidia have been imposing on us for a decade. It would be poetic to see 32-48GB at a non-eye-watering price point. Intel definitely seems to be doing all the right things on software support.
"It would be poetic to see 32-48GB at a non-eye-watering price point."
I heard some Asrock motherboard BIOSes could set the VRAM up to 64GB on Ryzen5.
Doing some investigations with different AMD hardware atm.
Re: PyTorch Library for Running LLM on Intel CPU and GPU
#19The company that did 4-cores-forever, has the opportunity to redeem itself, in its next consumer GPU release, by disrupting the "8-16GB VRAM forever" that AMD and Nvidia have been imposing on us for a decade. It would be poetic to see 32-48GB at a non-eye-watering price point. Intel definitely seems to be doing all the right things on software support.
still wondering why we can't have gpu's with sodimm slots so you can crank the vram
Re: PyTorch Library for Running LLM on Intel CPU and GPU
#20The company that did 4-cores-forever, has the opportunity to redeem itself, in its next consumer GPU release, by disrupting the "8-16GB VRAM forever" that AMD and Nvidia have been imposing on us for a decade. It would be poetic to see 32-48GB at a non-eye-watering price point. Intel definitely seems to be doing all the right things on software support.
Going above 24GB is probably not going to be cheap until gddr7 is out, and even that will only push it to 36gb. The fancier stacked gddr6 stuff is probably pretty expensive and you can’t just add more dies because of signal integrity issues.