Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

PyTorch Library for Running LLM on Intel CPU and GPU

github.com

Post reply on HN
Print view View original

41–50 of 98 posts

Re: PyTorch Library for Running LLM on Intel CPU and GPU

#41

Post by talldayo »

ponector wrote: post #40
riskable wrote: post #24

Earlier quoted context omitted.

No kidding... Intel is playing catch-up with Nvidia in the AI space and a big reason for that is their offerings aren't competitive. You can get an Intel Arc A770 with 16GB of VRAM (which was released in October, 2022) for about $300 or an Nvidia 4060 Ti with 16GB of VRAM for ~$500 which is twice as fast for AI workloads in reality (see: https://cdn.mos.cms.futurecdn.net/FtXkrY6AD8YypMiHrZuy4K-120... ) This is a huge…

I don't agree. Who will buy it? A few enthusiasts who wants to run LLM locally but cannot afford M3 or 4090? It will be a niche product with poor sales.

> Who will buy it?

Frustrated AMD customers willing to put their money where their mouth is?

Re: PyTorch Library for Running LLM on Intel CPU and GPU

#42

Post by VHRanger »

HarHarVeryFunny wrote: post #32
VHRanger wrote: post #12

Earlier quoted context omitted.

Intel is historically really good at the software side, though. For all their hardware research hiccups in the last 10 years, they've been delivering on open source machine learning libraries. It's apparently the same on driver improvements and gaming GPU features in the last year.

But even if Intel have stable optimized drivers and ML support, it'd still need to be supported by PyTorch/etc for most developers to want to use it. People want to write at high level, not at CUDA-type level.

Intel is supported in Pytorch, though. It's supported from their own branch, which is presumably a big annoyance to install, but they do work

Re: PyTorch Library for Running LLM on Intel CPU and GPU

#43

Post by ZeroCool2u »

goosedragons wrote: post #11
captaindiego wrote: post #10

Are there any Intel GPUs with a lot of vRAM that someone could recommend that would work with this?

For consumer stuff there's the Intel Arc A770 with 16GB VRAM. More than that and you start moving into enterprise stuff.

Which seems like their biggest mistake. If they would just release a card with more than 24GB VRAM, people would be clamoring for their cards, even if they were marginally slower. It's the same reason that 3090's are still in high demand compared to the 4090's.

Re: PyTorch Library for Running LLM on Intel CPU and GPU

#44

Post by talldayo »

haunter wrote: post #34
vegabook wrote: post #15

The company that did 4-cores-forever, has the opportunity to redeem itself, in its next consumer GPU release, by disrupting the "8-16GB VRAM forever" that AMD and Nvidia have been imposing on us for a decade. It would be poetic to see 32-48GB at a non-eye-watering price point. Intel definitely seems to be doing all the right things on software support.

First crypto then AI, I wish GPUs were left alone for gaming.

Are there actually gamers out there that are still struggling to source GPUs? Even at the height of the mining craze, it was still possible to backorder cards at MSRP if you're patient.

The serious crypto and AI nuts are all using custom hardware. Crypto moved onto ASICs for anything power-efficient, and Nvidia's DGX systems aren't being cannibalized from the gaming market.

Re: PyTorch Library for Running LLM on Intel CPU and GPU

#45

Post by bt1a »

ponector wrote: post #40
riskable wrote: post #24

Earlier quoted context omitted.

No kidding... Intel is playing catch-up with Nvidia in the AI space and a big reason for that is their offerings aren't competitive. You can get an Intel Arc A770 with 16GB of VRAM (which was released in October, 2022) for about $300 or an Nvidia 4060 Ti with 16GB of VRAM for ~$500 which is twice as fast for AI workloads in reality (see: https://cdn.mos.cms.futurecdn.net/FtXkrY6AD8YypMiHrZuy4K-120... ) This is a huge…

I don't agree. Who will buy it? A few enthusiasts who wants to run LLM locally but cannot afford M3 or 4090? It will be a niche product with poor sales.

I think there's more than a few enthusiasts who would be very interesting in buying 1 or more of these cards (if they had 32+ GB of memory), but I don't have any data to back that opinion up. It is not only those who can't afford a 4090 though.

While the 4090 can run models that use less than 24GB of memory at blistering speeds, models are going to continue to scale up and 24GB is fairly limiting. Because LLM inference can take advantage of splitting the layers among multiple GPUs, high memory GPUs that aren't super expensive are desirable.

To share a personal perspective, I have a desktop with a 3090 and an M1 Max Studio with 64GB of memory. I use the M1 for local LLMs because I can use up to 57~GB of memory, even though the output (in terms of tok/s) is much slower than ones I can fit on a 3090.

Re: PyTorch Library for Running LLM on Intel CPU and GPU

#46

Post by monocasa »

glitchc wrote: post #37
riskable wrote: post #24

Earlier quoted context omitted.

No kidding... Intel is playing catch-up with Nvidia in the AI space and a big reason for that is their offerings aren't competitive. You can get an Intel Arc A770 with 16GB of VRAM (which was released in October, 2022) for about $300 or an Nvidia 4060 Ti with 16GB of VRAM for ~$500 which is twice as fast for AI workloads in reality (see: https://cdn.mos.cms.futurecdn.net/FtXkrY6AD8YypMiHrZuy4K-120... ) This is a huge…

I think the answer to that is fairly straightforward. Intel isn't in the business of producing RAM. They would have to buy and integrate a third-party product which is likely not something their business side has ever contemplated as a viable strategy.

Their GPUs as sold already include RAM.

Re: PyTorch Library for Running LLM on Intel CPU and GPU

#47

Post by monocasa »

riskable wrote: post #29
whalesalad wrote: post #19

Earlier quoted context omitted.

still wondering why we can't have gpu's with sodimm slots so you can crank the vram

You can do this sort of thing but you can't use SODIMM slots because that places the actual memory chips too far away from the GPU. Instead what you need is something like BGA sockets ( https://www.nxp.com/design/design-center/development-boards/... ) which are stupidly expensive (e.g. $600 per socket).

You could probably use something like CAMM which solved a similar problem for lpddr.

https://en.wikipedia.org/wiki/CAMM_(memory_module)

Re: PyTorch Library for Running LLM on Intel CPU and GPU

#48

Post by loudmax »

ponector wrote: post #40
riskable wrote: post #24

Earlier quoted context omitted.

No kidding... Intel is playing catch-up with Nvidia in the AI space and a big reason for that is their offerings aren't competitive. You can get an Intel Arc A770 with 16GB of VRAM (which was released in October, 2022) for about $300 or an Nvidia 4060 Ti with 16GB of VRAM for ~$500 which is twice as fast for AI workloads in reality (see: https://cdn.mos.cms.futurecdn.net/FtXkrY6AD8YypMiHrZuy4K-120... ) This is a huge…

I don't agree. Who will buy it? A few enthusiasts who wants to run LLM locally but cannot afford M3 or 4090? It will be a niche product with poor sales.

I tend to agree that it would be niche. The machine learning enthusiast market is far smaller than the gamer market.

But selling to machine learning enthusiasts is not a bad place to be. A lot of these enthusiasts are going to go on to work at places that are deploying enterprise AI at scale. Right now, almost all of their experience is CUDA and they're likely to recommend hardware they're familiar with. By making consumer Intel GPUs attractive to ML enthusiasts, Intel would make their enterprise GPUs much more interesting for enterprise.

Re: PyTorch Library for Running LLM on Intel CPU and GPU

#49

Post by rnewme »

frognumber wrote: post #23
chessgecko wrote: post #20

Earlier quoted context omitted.

Going above 24GB is probably not going to be cheap until gddr7 is out, and even that will only push it to 36gb. The fancier stacked gddr6 stuff is probably pretty expensive and you can’t just add more dies because of signal integrity issues.

Assuming you want to maintain full bandwidth. Which I don't care too much about. However, even 16->24GB is a big step, since a lot of the model are developed for 3090/4090-class hardware. 36GB would place it lose to the class of the fancy 40GB data center cards. If Intel decided to push VRAM, it will definitely have a market. Critically, a lot of folks will also be incentivized to make software compatible, since it w…

How comes you don't care about full bandwidth?

Re: PyTorch Library for Running LLM on Intel CPU and GPU

#50

Post by mysteria »

loudmax wrote: post #48
ponector wrote: post #40

Earlier quoted context omitted.

I don't agree. Who will buy it? A few enthusiasts who wants to run LLM locally but cannot afford M3 or 4090? It will be a niche product with poor sales.

I tend to agree that it would be niche. The machine learning enthusiast market is far smaller than the gamer market. But selling to machine learning enthusiasts is not a bad place to be. A lot of these enthusiasts are going to go on to work at places that are deploying enterprise AI at scale. Right now, almost all of their experience is CUDA and they're likely to recommend hardware they're familiar with. By making co…

The problem is that this now becomes a long term investment, which doesn't work out when we have CEOs chasing quarterly profits and all that. Meanwhile Nvidia stuck with CUDA all those years back (while ensuring that it worked well on both the consumer and enterprise line) and now they reap the rewards.
Post reply on HN