Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

PyTorch Library for Running LLM on Intel CPU and GPU

github.com

Post reply on HN
Print view View original

91–98 of 98 posts

Re: PyTorch Library for Running LLM on Intel CPU and GPU

#91

Post by hnfong »

Dalewyn wrote: post #64
hnfong wrote: post #59

Earlier quoted context omitted.

It doesn't matter whether anyone is "spoiled" or not. The fact is large language models require a lot of VRAM, and the more interesting ones need more than 24GB to run. The people who are able to afford systems with more than 24GB VRAM will go buy hardware that gives them that, and when GPU vendors release products with insufficient VRAM they limit their market. I mean inequality is definitely increasing at a worryin…

I'm just fascinated that the response/demand to running out of RAM is "Just sell us more RAM, god damn!" instead of engineering a solution to make due with what is practically (and realistically) available.

What makes you think people aren't trying to engineer a solution that uses less RAM?

There are millions (billions?) of dollars at stake here, and obviously the best minds are already tackling the problem. Only plebs like us who don't have the skills to do so bicker on an internet forum... It's not like we could realistically spend the time inventing ways to run inference with fewer resources and make significant headway.

Re: PyTorch Library for Running LLM on Intel CPU and GPU

#92

Post by zoobab »

jmward01 wrote: post #73
ponector wrote: post #40

Earlier quoted context omitted.

I don't agree. Who will buy it? A few enthusiasts who wants to run LLM locally but cannot afford M3 or 4090? It will be a niche product with poor sales.

Microsoft got where they are because the developed tools that everyone used. The got the developers and the consumers followed. Intel (or AMD) could do the same thing. Get a big card with lost of ram so that the developers get used to your ecosystem and then sell the enterprise GPUs to make the $$$. It is a clear path with a lot of history and it blows my mind Intel and AMD aren't doing it.

"Microsoft got where they are because the developed tools that everyone used."

It's not like they don't have a monopoly on pre-installed OSes.

Re: PyTorch Library for Running LLM on Intel CPU and GPU

#93

Post by alecco »

ponector wrote: post #79
alecco wrote: post #60

Earlier quoted context omitted.

AFAIK, unless you are a huge American corp with orders above $100m Nvidia will only sell you old and expensive server cards like the crappy A40 PCIe 4.0 48GB GDDR6 at $5,000. Good luck getting SXM H100s or GH200. If Intel sells a stackable kit with a lot of RAM and a reasonable interconnect a lot of corporate customers will buy. It doesn't even have to be that good, just half way between PCIe 5.0 and NVLink. But it s…

Parent comment requested non-enterprise, consumer grade GPU with tons of memory. I'm sure there is no market for this. However, server solutions could have some traction.

Hobbyists are stacking 3090s with NVLink.

Re: PyTorch Library for Running LLM on Intel CPU and GPU

#94

Post by zoobab »

LoganDark wrote: post #70
zoobab wrote: post #18

Earlier quoted context omitted.

"It would be poetic to see 32-48GB at a non-eye-watering price point." I heard some Asrock motherboard BIOSes could set the VRAM up to 64GB on Ryzen5. Doing some investigations with different AMD hardware atm.

When has an APU ever been as fast as a GPU? How much cache does it have, a few hundred megabytes? That can't possibly be enough for matmul, no matter how much slow DDR4/5 is technically addressable.

"APU ever been as fast as a GPU"

Ryzen5 has both CPU+GPU on one chip, the BIOS allows you set the amount of VRAM. They share the same RAM bank, you can set 16GB of VRAM and 16GB for the OS if you use a 32GB RAM bank.

Re: PyTorch Library for Running LLM on Intel CPU and GPU

#95

Post by LoganDark »

zoobab wrote: post #94
LoganDark wrote: post #70

Earlier quoted context omitted.

When has an APU ever been as fast as a GPU? How much cache does it have, a few hundred megabytes? That can't possibly be enough for matmul, no matter how much slow DDR4/5 is technically addressable.

"APU ever been as fast as a GPU" Ryzen5 has both CPU+GPU on one chip, the BIOS allows you set the amount of VRAM. They share the same RAM bank, you can set 16GB of VRAM and 16GB for the OS if you use a 32GB RAM bank.

What I'm saying is that GPUs rely on having the memory close to the die so that it actually has enough bandwidth to saturate the cores. System memory is not very close to the CPU (compared to GPUs), so I have doubts about whether an APU would be able to reach GPU levels of performance over gigabytes of model weights.

Re: PyTorch Library for Running LLM on Intel CPU and GPU

#96

Post by dragonwriter »

Dalewyn wrote: post #64
hnfong wrote: post #59

Earlier quoted context omitted.

It doesn't matter whether anyone is "spoiled" or not. The fact is large language models require a lot of VRAM, and the more interesting ones need more than 24GB to run. The people who are able to afford systems with more than 24GB VRAM will go buy hardware that gives them that, and when GPU vendors release products with insufficient VRAM they limit their market. I mean inequality is definitely increasing at a worryin…

I'm just fascinated that the response/demand to running out of RAM is "Just sell us more RAM, god damn!" instead of engineering a solution to make due with what is practically (and realistically) available.

Quantization and CPU mode and hybrid mode where the model is split between CPU and GPU exist and work well for LLMs, but in the end more VRAM is a massive quality of life improvement for running (and probably more for training, which has higher RAM needs and forbwhich quantization isn't useful, AFAIK) them, even ifbyou technically can do them on CPU alone or hybrid with no/lower VRAM requirements.

Re: PyTorch Library for Running LLM on Intel CPU and GPU

#97

Post by dragonwriter »

wongarsu wrote: post #51
azinman2 wrote: post #36

Earlier quoted context omitted.

Didn’t nvidia try to block this in software by slowing down mining? Seems like we just need consumer matrix math cards with literally no video out, and then a different set of requirements for those with a video out.

But Nvidia doesn't want to make consumer compute cards because those might steal market share from the datacenter compute cards they are selling at 5x markup.

What we need is more real competition for NVidia cards at all levels, so rather than avoiding competing with themselves, they are worried about actually competing with the competition.

But capital doesn't want to invest in competition when it can instead invest in a chance at a moat that allows charging monopoly rents.

Re: PyTorch Library for Running LLM on Intel CPU and GPU

#98

Post by actionfromafar »

glitchc wrote: post #37
riskable wrote: post #24

Earlier quoted context omitted.

No kidding... Intel is playing catch-up with Nvidia in the AI space and a big reason for that is their offerings aren't competitive. You can get an Intel Arc A770 with 16GB of VRAM (which was released in October, 2022) for about $300 or an Nvidia 4060 Ti with 16GB of VRAM for ~$500 which is twice as fast for AI workloads in reality (see: https://cdn.mos.cms.futurecdn.net/FtXkrY6AD8YypMiHrZuy4K-120... ) This is a huge…

I think the answer to that is fairly straightforward. Intel isn't in the business of producing RAM. They would have to buy and integrate a third-party product which is likely not something their business side has ever contemplated as a viable strategy.

I wonder what the market dynamics for NVidia would be if Intel bought as much VRAM as it could.
Post reply on HN