Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

PyTorch Library for Running LLM on Intel CPU and GPU

github.com

Post reply on HN
Print view View original

61–70 of 98 posts

Re: PyTorch Library for Running LLM on Intel CPU and GPU

#61

Post by hnfong »

0cf8612b2e1e wrote: post #35
frognumber wrote: post #23

Earlier quoted context omitted.

Assuming you want to maintain full bandwidth. Which I don't care too much about. However, even 16->24GB is a big step, since a lot of the model are developed for 3090/4090-class hardware. 36GB would place it lose to the class of the fancy 40GB data center cards. If Intel decided to push VRAM, it will definitely have a market. Critically, a lot of folks will also be incentivized to make software compatible, since it w…

At this point, I cannot run an entire class of models without OOM. I will take a performance hit if it lets me run it at all. I want a consumer card that can do some number of tokens per second. I do not need a monster that can serve as the basis for a startup.

A maxed out Mac Studio probably fits your requirements as stated.

Re: PyTorch Library for Running LLM on Intel CPU and GPU

#62

Post by resource_waste »

ponector wrote: post #40
riskable wrote: post #24

Earlier quoted context omitted.

No kidding... Intel is playing catch-up with Nvidia in the AI space and a big reason for that is their offerings aren't competitive. You can get an Intel Arc A770 with 16GB of VRAM (which was released in October, 2022) for about $300 or an Nvidia 4060 Ti with 16GB of VRAM for ~$500 which is twice as fast for AI workloads in reality (see: https://cdn.mos.cms.futurecdn.net/FtXkrY6AD8YypMiHrZuy4K-120... ) This is a huge…

I don't agree. Who will buy it? A few enthusiasts who wants to run LLM locally but cannot afford M3 or 4090? It will be a niche product with poor sales.

>M3

>4090

These are noob hardware. A6000 is my choice.

Which really only further emphesizes your point.

>CPU based is a waste of everyone's time/effort

>GPU based is 100% limited by VRAM, and is what you are realistically going to use.

Re: PyTorch Library for Running LLM on Intel CPU and GPU

#63

Post by resource_waste »

loudmax wrote: post #48
ponector wrote: post #40

Earlier quoted context omitted.

I don't agree. Who will buy it? A few enthusiasts who wants to run LLM locally but cannot afford M3 or 4090? It will be a niche product with poor sales.

I tend to agree that it would be niche. The machine learning enthusiast market is far smaller than the gamer market. But selling to machine learning enthusiasts is not a bad place to be. A lot of these enthusiasts are going to go on to work at places that are deploying enterprise AI at scale. Right now, almost all of their experience is CUDA and they're likely to recommend hardware they're familiar with. By making co…

I need offline LLMs for work.

It doesnt need to be consumer grade, it doesnt need to be ultra high either.

It needs to be cheap enough for my department to expensive it via petty cash.

Re: PyTorch Library for Running LLM on Intel CPU and GPU

#64

Post by Dalewyn »

hnfong wrote: post #59
Dalewyn wrote: post #54

Earlier quoted context omitted.

>models are going to continue to scale up and 24GB is fairly limiting >24GB is fairly limiting Can I take a moment to suggest that maybe we're very spoiled? 24GB of VRAM is more than most peoples' system RAM, and that is "fairly limiting"? To think Bill once said 640KB would be enough.

It doesn't matter whether anyone is "spoiled" or not. The fact is large language models require a lot of VRAM, and the more interesting ones need more than 24GB to run. The people who are able to afford systems with more than 24GB VRAM will go buy hardware that gives them that, and when GPU vendors release products with insufficient VRAM they limit their market. I mean inequality is definitely increasing at a worryin…

I'm just fascinated that the response/demand to running out of RAM is "Just sell us more RAM, god damn!" instead of engineering a solution to make due with what is practically (and realistically) available.

Re: PyTorch Library for Running LLM on Intel CPU and GPU

#65

Post by baq »

haunter wrote: post #34
vegabook wrote: post #15

The company that did 4-cores-forever, has the opportunity to redeem itself, in its next consumer GPU release, by disrupting the "8-16GB VRAM forever" that AMD and Nvidia have been imposing on us for a decade. It would be poetic to see 32-48GB at a non-eye-watering price point. Intel definitely seems to be doing all the right things on software support.

First crypto then AI, I wish GPUs were left alone for gaming.

They were.

But then those pesky researchers and hackers figured out how to use the matmul hardware for non-gaming.

Re: PyTorch Library for Running LLM on Intel CPU and GPU

#66

Post by Wytwwww »

mysteria wrote: post #50
loudmax wrote: post #48

Earlier quoted context omitted.

I tend to agree that it would be niche. The machine learning enthusiast market is far smaller than the gamer market. But selling to machine learning enthusiasts is not a bad place to be. A lot of these enthusiasts are going to go on to work at places that are deploying enterprise AI at scale. Right now, almost all of their experience is CUDA and they're likely to recommend hardware they're familiar with. By making co…

The problem is that this now becomes a long term investment, which doesn't work out when we have CEOs chasing quarterly profits and all that. Meanwhile Nvidia stuck with CUDA all those years back (while ensuring that it worked well on both the consumer and enterprise line) and now they reap the rewards.

Current Intel and its leadership seems to be much more focused on long term goals/growth than before, or so they claim.

Re: PyTorch Library for Running LLM on Intel CPU and GPU

#67

Post by 0cf8612b2e1e »

hnfong wrote: post #61
0cf8612b2e1e wrote: post #35

Earlier quoted context omitted.

At this point, I cannot run an entire class of models without OOM. I will take a performance hit if it lets me run it at all. I want a consumer card that can do some number of tokens per second. I do not need a monster that can serve as the basis for a startup.

A maxed out Mac Studio probably fits your requirements as stated.

If I were willing to drop $4k on that setup, I might as well get the real NVidia offering.

The hobbyist market needs something priced well under $1k to make it accessible.

Re: PyTorch Library for Running LLM on Intel CPU and GPU

#68

Post by xoranth »

Dalewyn wrote: post #64
hnfong wrote: post #59

Earlier quoted context omitted.

It doesn't matter whether anyone is "spoiled" or not. The fact is large language models require a lot of VRAM, and the more interesting ones need more than 24GB to run. The people who are able to afford systems with more than 24GB VRAM will go buy hardware that gives them that, and when GPU vendors release products with insufficient VRAM they limit their market. I mean inequality is definitely increasing at a worryin…

I'm just fascinated that the response/demand to running out of RAM is "Just sell us more RAM, god damn!" instead of engineering a solution to make due with what is practically (and realistically) available.

People have engineered solutions to make what is available practical (see all the various quantization schemes that have come out).

It is just that there's a limit to how much you can compress the models.

Re: PyTorch Library for Running LLM on Intel CPU and GPU

#69

Post by OkayPhysicist »

vegabook wrote: post #15

The company that did 4-cores-forever, has the opportunity to redeem itself, in its next consumer GPU release, by disrupting the "8-16GB VRAM forever" that AMD and Nvidia have been imposing on us for a decade. It would be poetic to see 32-48GB at a non-eye-watering price point. Intel definitely seems to be doing all the right things on software support.

The issue from the manufacturer's perspective is that they've got two different customer bases with wildly different willingness to pay, but not substantially different needs from their product. If Nvidia and AMD didn't split the two markets somehow, then there would be no cards available to the PC market, since the AI companies with much deeper pockets would buy up the lot. This is undesirable from the manufacturer's perspective for a couple reasons, but I suspect a big one is worries that the next AI winter would cause their entire business to crater out, whereas the PC market is pretty reliable for the foreseeable future.

Right now, the best discriminator they have is that PC users are willing to put up with much smaller amounts of VRAM.

Re: PyTorch Library for Running LLM on Intel CPU and GPU

#70

Post by LoganDark »

zoobab wrote: post #18
vegabook wrote: post #15

The company that did 4-cores-forever, has the opportunity to redeem itself, in its next consumer GPU release, by disrupting the "8-16GB VRAM forever" that AMD and Nvidia have been imposing on us for a decade. It would be poetic to see 32-48GB at a non-eye-watering price point. Intel definitely seems to be doing all the right things on software support.

"It would be poetic to see 32-48GB at a non-eye-watering price point." I heard some Asrock motherboard BIOSes could set the VRAM up to 64GB on Ryzen5. Doing some investigations with different AMD hardware atm.

When has an APU ever been as fast as a GPU? How much cache does it have, a few hundred megabytes? That can't possibly be enough for matmul, no matter how much slow DDR4/5 is technically addressable.
Post reply on HN