Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

PyTorch Library for Running LLM on Intel CPU and GPU

github.com

Post reply on HN
Print view View original

21–30 of 98 posts

Re: PyTorch Library for Running LLM on Intel CPU and GPU

#21

Post by amir_karbasi »

whalesalad wrote: post #19
vegabook wrote: post #15

The company that did 4-cores-forever, has the opportunity to redeem itself, in its next consumer GPU release, by disrupting the "8-16GB VRAM forever" that AMD and Nvidia have been imposing on us for a decade. It would be poetic to see 32-48GB at a non-eye-watering price point. Intel definitely seems to be doing all the right things on software support.

still wondering why we can't have gpu's with sodimm slots so you can crank the vram

I believe that the issue is that graphic cards require really fast memory. This requires close memory placement (that's why the memory is so close to the core on the board). expandable memory will not be able to provide the required bandwidth here.

Re: PyTorch Library for Running LLM on Intel CPU and GPU

#22

Post by chessgecko »

whalesalad wrote: post #19
vegabook wrote: post #15

The company that did 4-cores-forever, has the opportunity to redeem itself, in its next consumer GPU release, by disrupting the "8-16GB VRAM forever" that AMD and Nvidia have been imposing on us for a decade. It would be poetic to see 32-48GB at a non-eye-watering price point. Intel definitely seems to be doing all the right things on software support.

still wondering why we can't have gpu's with sodimm slots so you can crank the vram

You could, but the memory bandwidth wouldn’t be amazing unless you had a lot of sticks and it would end up getting pretty expensive

Re: PyTorch Library for Running LLM on Intel CPU and GPU

#23

Post by frognumber »

chessgecko wrote: post #20
vegabook wrote: post #15

The company that did 4-cores-forever, has the opportunity to redeem itself, in its next consumer GPU release, by disrupting the "8-16GB VRAM forever" that AMD and Nvidia have been imposing on us for a decade. It would be poetic to see 32-48GB at a non-eye-watering price point. Intel definitely seems to be doing all the right things on software support.

Going above 24GB is probably not going to be cheap until gddr7 is out, and even that will only push it to 36gb. The fancier stacked gddr6 stuff is probably pretty expensive and you can’t just add more dies because of signal integrity issues.

Assuming you want to maintain full bandwidth.

Which I don't care too much about.

However, even 16->24GB is a big step, since a lot of the model are developed for 3090/4090-class hardware. 36GB would place it lose to the class of the fancy 40GB data center cards.

If Intel decided to push VRAM, it will definitely have a market. Critically, a lot of folks will also be incentivized to make software compatible, since it will be the cheapest way to run models.

Re: PyTorch Library for Running LLM on Intel CPU and GPU

#24

Post by riskable »

vegabook wrote: post #15

The company that did 4-cores-forever, has the opportunity to redeem itself, in its next consumer GPU release, by disrupting the "8-16GB VRAM forever" that AMD and Nvidia have been imposing on us for a decade. It would be poetic to see 32-48GB at a non-eye-watering price point. Intel definitely seems to be doing all the right things on software support.

No kidding... Intel is playing catch-up with Nvidia in the AI space and a big reason for that is their offerings aren't competitive. You can get an Intel Arc A770 with 16GB of VRAM (which was released in October, 2022) for about $300 or an Nvidia 4060 Ti with 16GB of VRAM for ~$500 which is twice as fast for AI workloads in reality (see: https://cdn.mos.cms.futurecdn.net/FtXkrY6AD8YypMiHrZuy4K-120... )

This is a huge problem because in theory the Arc A770 is faster! It's theoretical performance (TFLOPS) is more than twice as fast as an Nvidia 4060 (see: https://cdn.mos.cms.futurecdn.net/Q7WgNxqfgyjCJ5kk8apUQE-120... ). So why does it perform so poorly? Because everything AI-related has been developed and optimized to run on Nvidia's CUDA.

Mostly, this is a mindshare issue. If Intel offered a workstation GPU (i.e. not a ridiculously expensive "enterprise" monster) that developers could use that had something like 32GB or 64GB of VRAM it would sell! They'd sell zillions of them! In fact, I'd wager that they'd be so popular it'd be hard for consumers to even get their hands on one because it would sell out everywhere.

It doesn't even need to be the fastest card. It just needs to offer more VRAM than the competition. Right now, if you want to do things like training or video generation the lack of VRAM is a bigger bottleneck than the speed of the GPU. How does Intel not see this‽ They have the power to step up and take over a huge section of the market but instead they're just copying (poorly) what everyone else is doing.

Re: PyTorch Library for Running LLM on Intel CPU and GPU

#25

Post by belter »

vegabook wrote: post #15

The company that did 4-cores-forever, has the opportunity to redeem itself, in its next consumer GPU release, by disrupting the "8-16GB VRAM forever" that AMD and Nvidia have been imposing on us for a decade. It would be poetic to see 32-48GB at a non-eye-watering price point. Intel definitely seems to be doing all the right things on software support.

AMD making drivers of high quality? I would pay to see that :-)

Re: PyTorch Library for Running LLM on Intel CPU and GPU

#26

Post by frognumber »

amir_karbasi wrote: post #21
whalesalad wrote: post #19

Earlier quoted context omitted.

still wondering why we can't have gpu's with sodimm slots so you can crank the vram

I believe that the issue is that graphic cards require really fast memory. This requires close memory placement (that's why the memory is so close to the core on the board). expandable memory will not be able to provide the required bandwidth here.

The universe used to have hierarchies. Fast memory close, slow memory far. Registers. L1. L2. L3. RAM. Swap.

The same thing would make a lot of sense here. Super-fast memory close, with overflow into classic DDR slots.

As a footnote, going parallel also helps. 8 sticks of RAM at 1/8 the bandwidth each is the same as one stick of RAM at 8x the bandwidth, if you don't multiplex onto the same traces.

Re: PyTorch Library for Running LLM on Intel CPU and GPU

#27

Post by frognumber »

VHRanger wrote: post #12
_joel wrote: post #9

Earlier quoted context omitted.

and 65% of the software support, less I'm inclined to believe? Although having more players in the fold is definitely a good thing.

Intel is historically really good at the software side, though. For all their hardware research hiccups in the last 10 years, they've been delivering on open source machine learning libraries. It's apparently the same on driver improvements and gaming GPU features in the last year.

I'm optimistic Intel will get the software right in due course. Last I looked, it wasn't all there yet, but it was on the right track.

Right now, I have a nice NVidia card, but if things stay on track, I think it's very likely my next GPU might be Intel. Open-source, not to mention better value.

Re: PyTorch Library for Running LLM on Intel CPU and GPU

#28

Post by justsomehnguy »

whalesalad wrote: post #19
vegabook wrote: post #15

The company that did 4-cores-forever, has the opportunity to redeem itself, in its next consumer GPU release, by disrupting the "8-16GB VRAM forever" that AMD and Nvidia have been imposing on us for a decade. It would be poetic to see 32-48GB at a non-eye-watering price point. Intel definitely seems to be doing all the right things on software support.

still wondering why we can't have gpu's with sodimm slots so you can crank the vram

Look at the motherboards with >2 Memory channels. That would require a lot of physical space, which is quite restricted on a 50 y/o standard for the expansion cards.

Re: PyTorch Library for Running LLM on Intel CPU and GPU

#29

Post by riskable »

whalesalad wrote: post #19
vegabook wrote: post #15

The company that did 4-cores-forever, has the opportunity to redeem itself, in its next consumer GPU release, by disrupting the "8-16GB VRAM forever" that AMD and Nvidia have been imposing on us for a decade. It would be poetic to see 32-48GB at a non-eye-watering price point. Intel definitely seems to be doing all the right things on software support.

still wondering why we can't have gpu's with sodimm slots so you can crank the vram

You can do this sort of thing but you can't use SODIMM slots because that places the actual memory chips too far away from the GPU. Instead what you need is something like BGA sockets (https://www.nxp.com/design/design-center/development-boards/... ) which are stupidly expensive (e.g. $600 per socket).

Re: PyTorch Library for Running LLM on Intel CPU and GPU

#30

Post by Workaccount2 »

riskable wrote: post #24
vegabook wrote: post #15

The company that did 4-cores-forever, has the opportunity to redeem itself, in its next consumer GPU release, by disrupting the "8-16GB VRAM forever" that AMD and Nvidia have been imposing on us for a decade. It would be poetic to see 32-48GB at a non-eye-watering price point. Intel definitely seems to be doing all the right things on software support.

No kidding... Intel is playing catch-up with Nvidia in the AI space and a big reason for that is their offerings aren't competitive. You can get an Intel Arc A770 with 16GB of VRAM (which was released in October, 2022) for about $300 or an Nvidia 4060 Ti with 16GB of VRAM for ~$500 which is twice as fast for AI workloads in reality (see: https://cdn.mos.cms.futurecdn.net/FtXkrY6AD8YypMiHrZuy4K-120... ) This is a huge…

Based on leaks, it looks like intel somehow missed an easy opportunity here. There is an insane demand for high VRAM cards now, and it seems the next intel cards will be 12GB.

Intel, screw everything else, just pack as much VRAM in those as you can. Build it and they will come.

Post reply on HN