Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

PyTorch Library for Running LLM on Intel CPU and GPU

github.com

Post reply on HN
Print view View original

31–40 of 98 posts

Re: PyTorch Library for Running LLM on Intel CPU and GPU

#31

Post by riskable »

frognumber wrote: post #26
amir_karbasi wrote: post #21

Earlier quoted context omitted.

I believe that the issue is that graphic cards require really fast memory. This requires close memory placement (that's why the memory is so close to the core on the board). expandable memory will not be able to provide the required bandwidth here.

The universe used to have hierarchies. Fast memory close, slow memory far. Registers. L1. L2. L3. RAM. Swap. The same thing would make a lot of sense here. Super-fast memory close, with overflow into classic DDR slots. As a footnote, going parallel also helps. 8 sticks of RAM at 1/8 the bandwidth each is the same as one stick of RAM at 8x the bandwidth, if you don't multiplex onto the same traces.

It's not so simple... The way GPU architecture works is that it needs as-fast-as-possible access to its VRAM. The concept of "overflow memory" for a GPU is your PC's RAM. Adding a secondary memory controller and equivalent DRAM to the card itself would only provide a trivial improvement over, "just using the PC RAM".

Point of fact: GPUs don't even use all the PCI Express lanes they have available to them! Most GPUs (even top of the line ones like Nvidia's 4090) only use about 8 lanes of bandwidth. This is why some newer GPUs are being offered with M.2 slots so you can add an SSD (https://press.asus.com/news/asus-dual-geforce-rtx-4060-ti-ss... ).

Re: PyTorch Library for Running LLM on Intel CPU and GPU

#32

Post by HarHarVeryFunny »

VHRanger wrote: post #12
_joel wrote: post #9

Earlier quoted context omitted.

and 65% of the software support, less I'm inclined to believe? Although having more players in the fold is definitely a good thing.

Intel is historically really good at the software side, though. For all their hardware research hiccups in the last 10 years, they've been delivering on open source machine learning libraries. It's apparently the same on driver improvements and gaming GPU features in the last year.

But even if Intel have stable optimized drivers and ML support, it'd still need to be supported by PyTorch/etc for most developers to want to use it. People want to write at high level, not at CUDA-type level.

Re: PyTorch Library for Running LLM on Intel CPU and GPU

#33

Post by [deleted] »

vegabook wrote: post #15

The company that did 4-cores-forever, has the opportunity to redeem itself, in its next consumer GPU release, by disrupting the "8-16GB VRAM forever" that AMD and Nvidia have been imposing on us for a decade. It would be poetic to see 32-48GB at a non-eye-watering price point. Intel definitely seems to be doing all the right things on software support.

[deleted]

Re: PyTorch Library for Running LLM on Intel CPU and GPU

#34

Post by haunter »

vegabook wrote: post #15

The company that did 4-cores-forever, has the opportunity to redeem itself, in its next consumer GPU release, by disrupting the "8-16GB VRAM forever" that AMD and Nvidia have been imposing on us for a decade. It would be poetic to see 32-48GB at a non-eye-watering price point. Intel definitely seems to be doing all the right things on software support.

First crypto then AI, I wish GPUs were left alone for gaming.

Re: PyTorch Library for Running LLM on Intel CPU and GPU

#35

Post by 0cf8612b2e1e »

frognumber wrote: post #23
chessgecko wrote: post #20

Earlier quoted context omitted.

Going above 24GB is probably not going to be cheap until gddr7 is out, and even that will only push it to 36gb. The fancier stacked gddr6 stuff is probably pretty expensive and you can’t just add more dies because of signal integrity issues.

Assuming you want to maintain full bandwidth. Which I don't care too much about. However, even 16->24GB is a big step, since a lot of the model are developed for 3090/4090-class hardware. 36GB would place it lose to the class of the fancy 40GB data center cards. If Intel decided to push VRAM, it will definitely have a market. Critically, a lot of folks will also be incentivized to make software compatible, since it w…

At this point, I cannot run an entire class of models without OOM. I will take a performance hit if it lets me run it at all.

I want a consumer card that can do some number of tokens per second. I do not need a monster that can serve as the basis for a startup.

Re: PyTorch Library for Running LLM on Intel CPU and GPU

#36

Post by azinman2 »

haunter wrote: post #34
vegabook wrote: post #15

The company that did 4-cores-forever, has the opportunity to redeem itself, in its next consumer GPU release, by disrupting the "8-16GB VRAM forever" that AMD and Nvidia have been imposing on us for a decade. It would be poetic to see 32-48GB at a non-eye-watering price point. Intel definitely seems to be doing all the right things on software support.

First crypto then AI, I wish GPUs were left alone for gaming.

Didn’t nvidia try to block this in software by slowing down mining?

Seems like we just need consumer matrix math cards with literally no video out, and then a different set of requirements for those with a video out.

Re: PyTorch Library for Running LLM on Intel CPU and GPU

#37

Post by glitchc »

riskable wrote: post #24
vegabook wrote: post #15

The company that did 4-cores-forever, has the opportunity to redeem itself, in its next consumer GPU release, by disrupting the "8-16GB VRAM forever" that AMD and Nvidia have been imposing on us for a decade. It would be poetic to see 32-48GB at a non-eye-watering price point. Intel definitely seems to be doing all the right things on software support.

No kidding... Intel is playing catch-up with Nvidia in the AI space and a big reason for that is their offerings aren't competitive. You can get an Intel Arc A770 with 16GB of VRAM (which was released in October, 2022) for about $300 or an Nvidia 4060 Ti with 16GB of VRAM for ~$500 which is twice as fast for AI workloads in reality (see: https://cdn.mos.cms.futurecdn.net/FtXkrY6AD8YypMiHrZuy4K-120... ) This is a huge…

I think the answer to that is fairly straightforward. Intel isn't in the business of producing RAM. They would have to buy and integrate a third-party product which is likely not something their business side has ever contemplated as a viable strategy.

Re: PyTorch Library for Running LLM on Intel CPU and GPU

#38

Post by stefanka »

zoobab wrote: post #18
vegabook wrote: post #15

The company that did 4-cores-forever, has the opportunity to redeem itself, in its next consumer GPU release, by disrupting the "8-16GB VRAM forever" that AMD and Nvidia have been imposing on us for a decade. It would be poetic to see 32-48GB at a non-eye-watering price point. Intel definitely seems to be doing all the right things on software support.

"It would be poetic to see 32-48GB at a non-eye-watering price point." I heard some Asrock motherboard BIOSes could set the VRAM up to 64GB on Ryzen5. Doing some investigations with different AMD hardware atm.

That would be an interesting information. Which MB works with with which APU with 32 or more GB of VRAM. Can you post your findings please?

Re: PyTorch Library for Running LLM on Intel CPU and GPU

#40

Post by ponector »

riskable wrote: post #24
vegabook wrote: post #15

The company that did 4-cores-forever, has the opportunity to redeem itself, in its next consumer GPU release, by disrupting the "8-16GB VRAM forever" that AMD and Nvidia have been imposing on us for a decade. It would be poetic to see 32-48GB at a non-eye-watering price point. Intel definitely seems to be doing all the right things on software support.

No kidding... Intel is playing catch-up with Nvidia in the AI space and a big reason for that is their offerings aren't competitive. You can get an Intel Arc A770 with 16GB of VRAM (which was released in October, 2022) for about $300 or an Nvidia 4060 Ti with 16GB of VRAM for ~$500 which is twice as fast for AI workloads in reality (see: https://cdn.mos.cms.futurecdn.net/FtXkrY6AD8YypMiHrZuy4K-120... ) This is a huge…

I don't agree. Who will buy it? A few enthusiasts who wants to run LLM locally but cannot afford M3 or 4090?

It will be a niche product with poor sales.

Post reply on HN