Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Intel announces Arc B-series "Battlemage" discrete graphics with Linux support

phoronix.com

Post reply on HN
Print view View original

681–690 of 722 posts

Re: Intel announces Arc B-series "Battlemage" discrete graphics with Linux support

#681

Post by ryao »

rubatuga wrote: post #511
bongodongobob wrote: post #152

Earlier quoted context omitted.

I don't know a single person in real life that has any desire to run local LLMs. Even amongst my colleagues and tech friends, not very many use LLMs period. It's still very niche outside AI enthusiasts. GPT is better than anything I can run locally anyway. It's not as popular as you think it is.

I run a 12GB model on my 3060 and use it to help answer healthcare questions. I'm currently doing a medical residency. (No I don't use it to diagnose). It helps comply with any HIPAA style regulations. I sometimes use it to fix up my emails. Not sure why people are longing for a 128GB card, just download a quantized model and run with LM Studio ( https://lmstudio.ai/ ). At least two of my colleagues are using ChatGPT…

A 128GB card could run llama 3.1B 70B with FP8. It is a huge increase in quality over what the current 24GB cards can do.

Re: Intel announces Arc B-series "Battlemage" discrete graphics with Linux support

#682

Post by ryao »

tremon wrote: post #29
jeffbee wrote: post #15

Earlier quoted context omitted.

What is the newest platform that lacks resizable BAR? It was standardized in 2006. Is 4060-level graphics performance useful in whatever old computer has that problem?

The newest platform is probably POWER10. ReBar is not supported on any POWER platform, most likely including the upcoming POWER11. Also, I don't think you'll find many mainboards from 2006 supporting it. It may have been standardized in 2006, but a quick online search leads me to think that even on x86 mainboards it didn't become commonly available until at least 2020.

If the firmware is open source, perhaps it could be retrofitted. For amd64 motherboards that do not have native UEFI support, retrofits are possible through this:

https://github.com/xCuri0/ReBarUEFI

Re: Intel announces Arc B-series "Battlemage" discrete graphics with Linux support

#683

Post by stephenr »

wing-_-nuts wrote: post #185
philistine wrote: post #158

Earlier quoted context omitted.

I'm baffled that PC gamers have decided that 1440p is the endgame for graphics. When I look at a 27-inch 1440p display, I see pixel edges everywhere. It's right at the edge of losing the visibility of individual pixels, since I can't perceive them at 27-inch 2160p, but not quite there yet for desktop distances. Time marches on, and I become ever more separated from gaming PC enthusiasts.

I used to be in the '4k or bust' camp, but then I realized that I needed 1.5x scaling on a 27" display to have my UI at a comfy size. That put me right back at 1440p screen real estate and you had to deal with fractional scaling issues. Instead, I bought a good 27" 1440p monitor, and you know what? I am not the discerning connoisseur of pixels that I thought I was. Honestly, it's fine . I will hold out with this setu…

> That put me right back at 1440p screen real estate and you had to deal with fractional scaling issues.

Get a 5K 27".

No fractional scaling, same real estate, much better picture.

Re: Intel announces Arc B-series "Battlemage" discrete graphics with Linux support

#684

Post by Zandikar »

ryao wrote: post #584
Zandikar wrote: post #493

Earlier quoted context omitted.

> They can't just slap more memory on the board, they would need to dedicate significantly more silicon area to memory IO and drive up the cost of the part, In the pedantic sense of just literally slapping more on existing boards? No, they might have one empty spot for an extra BGA VRAM chip, but not enough for the gain's we're talking about. But this is absolutely possible, trivially so for someone like Intel/AMD/NV…

A 128GB VRAM Intel Arc card at a low price would be an OSS developer’s dream come true. It would be the WRT54G of inference.

Of course. A cheap card with oodles of VRAM would benefit some people, I'm not denying that. I'm tackling the question of would it benefit intel (as the original question was "why doesn't intel do this"), and the answer is: Profit/Loss.

There's a huge number of people in that community that would love to have such a card. How many are actually willing and able to pony up >=$3k per unit? How many units would they buy? Given all of the other considerations that go into making such cards useful and easy to use (as described), the answer is - in intel's mind - nowhere near enough, especially when the financial side of the company's jimmies are so rustled that they sacked Pat G without a proper replacement and nominated some finance bros in as interim CEO's. Intel is ALREADY taking a big risk and financial burden trying to get into this space in the first place, and they're already struggling, so the prospect of betting the house like that just isn't going to fly for the finance bros that can't see passed the next 2 quarters.

To be clear, I personally think there is huge potential value in trying to support the OSS community to, in essence, "crowd source" and speedrun some of that ecosystem by supplying (Compared to the competition) "cheap" cards that aschew the artificial segmentation everyone else is doing and investing in that community. But I'm not running Intel, so while that'd be nice, it's not really relevant.

Re: Intel announces Arc B-series "Battlemage" discrete graphics with Linux support

#685

Post by ryao »

treprinum wrote: post #641
ryao wrote: post #629

Earlier quoted context omitted.

Isn’t that setting just a historical thing annd ann integrated GPU is able to access any system memory that is mapped by the IOMMU? I assume this is how it works for people using the NVIDIA Jetson AGX Orin 64GB Developer Kit to do inference. I do not know why it would be different for AMD APUs.

I remember somebody complaining about it on reddit, unable to overcome some BIOS limitation on an AMD G processor. Even on M3 Max one had to issue a special command to enable GPU to access more memory.

The command on the M3 Max is a sysctl command to adjust an operating system enforced limit. That is different than the aperture setting in the bios. The limitation on AMD is more interesting and more relevant.

Re: Intel announces Arc B-series "Battlemage" discrete graphics with Linux support

#686

Post by ryao »

Numerlor wrote: post #660
ryao wrote: post #626

Earlier quoted context omitted.

Look at the 3090, which uses 24 chips (12 on one side and 12 on another). Pushing it to 32 is doable. 32 is all you need to reach 128GB VRAM with the 32Gbit GDDR7 chips that should be on the market in the near future.

Where would you route the connection to the additional 4 groups of chips around the die? The PCIe connection needs to be there too, and they also may not like power delivery going through them

Nvidia has done a 512-bit bus in the past. The 3090 has 4 groups of 3 on each side of the card. Switching to 4 groups of 4 should be doable visually. That said, I would not want to be the one responsible for doing the trace routing.

Re: Intel announces Arc B-series "Battlemage" discrete graphics with Linux support

#687

Post by ryao »

FeepingCreature wrote: post #630
ryao wrote: post #625

Earlier quoted context omitted.

What did he say?

Geohot (temporarily) giving up. https://github.com/ROCm/ROCm/issues/2198#issuecomment-157438... Sadly most of the real spicy Twitter messages are gone since he deleted all his content, but there was a really fun one where he went off on a beautifully cryptic commit message in the driver. He also begged AMD to opensource the firmware so he could debug it. Sadly, AMD promised to do it and then nothing happened, as is t…

Who is tinygrad?

Re: Intel announces Arc B-series "Battlemage" discrete graphics with Linux support

#688

Post by MrDrMcCoy »

ryao wrote: post #639
ddtaylor wrote: post #297

Earlier quoted context omitted.

Intel has earned a lot of credit in the Linux space. Nvidia is trash tier in terms of support and only recently making serious steps to actually support the platform. AMD went all in nearly a decade ago and it's working pretty well for them. They are mostly caught up to being Intel grade support in the kernel. Meanwhile, Intel has been doing this since I was in college. I was running the i915 driver in Ubuntu 20 year…

This is repeated often, but I have had very good support from Nvidia on Linux over the years. AMD on the other hand gives lousy support. File a bug report about a problem and expect to be ignored, especially if it has anything to do with emulation. Intel’s Linux support on the other hand has been very good for me too.

Nvidia desktop GPUs may be fine on Linux now, but are very much still trash on Laptops. I can't even begin to express how troublesome they have been on the work laptops I've been forced to use. I would've killed for any non-Nvidia laptop, even if just integrated graphics, but IT won't approve any other models, and the UEFI lacks an option to fully disable the Nvidia GPU.

Re: Intel announces Arc B-series "Battlemage" discrete graphics with Linux support

#689

Post by theshrike79 »

JohnBooty wrote: post #667
theshrike79 wrote: post #603

Earlier quoted context omitted.

The future of local LLMs is not people running it on their PCs. It's going to be a HomePod/AppleTV/Echo/Google Home -style box you set up in a corner and forget about it. Then your devices in the ecosystem can offload some LLM tasks to that local system for inference, without having to do everything on-device.

This makes sense in some ways technologically, but just having a "centralized compute box" seems like a lot more complexity than many/most would want in their homes. I mean, everything could have been already working that way for a lot of years right? One big shared compute box in your house and everything else is a dumb screen? But few people roll that way, even nerds, so I don't see that becoming a thing for offloa…

It most likely won't be a separate device. It'll get integrated into something like Apple TV or a HomePod that has an actual function and will be plugged in and networked all the time anyway. The LLM stuff would be just a bonus.

Both are currently used as the hub for HomeKit devices. Making the ATV into a "magic AI Box" won't need anything else except "just" upgrading the CPU from A-series to M-series. Actually the A18 Pro would be enough, it's already used for local inference on the iPhone 16 Pro.

Re: Intel announces Arc B-series "Battlemage" discrete graphics with Linux support

#690

Post by ryao »

FeepingCreature wrote: post #632
ryao wrote: post #624

Earlier quoted context omitted.

Usually, you can do 2 AVX-512 operations per cycle and using FMADD (fused multiply-add) instructions, you can do two floating point operations for the price of one. That would be 128 operations per cycle per core. The result would be 16TFlops on a 2GHz 64 core CPU, not 4 TFlops. This would give a 1 order of magnitude difference, rather than 4 orders of magnitude. For inference, prompt processing is compute intensive,…

That's fair. On the other hand, there's like exactly one CPU with FP16 AVX512 anyways, and 64core aren't exactly commonplace either. And even with all those advantages, using a datacenter CPU, you're still a factor of 10 off from a GPU that isn't even consumer top-end. With a normal processor, say 16 cores, 16 float ops, even with fused ops and dispatching two ops per cycle you're still only at 2T and ~50x. In consum…

Zen 6 is supposed to add FP16 AVX512 support if AMD’s leaked slides mean what I think they mean. Here is a link to a screenshot of the leaked slides MLID published:

https://overclockers.ru/st/legacy/blog/428111/424644_O.jpg

I have been working on doing inference on a Ryzen 7 5800X lately and I have had good results:

https://github.com/ryao/llama3.c/blob/master/run.c

Running on a GPU like my 3090 Ti will likely outperform it by two orders of magnitude, but I have managed to push the needle slightly on the state of the art performance for prompt processing on my CPU. I suspect an additional 15% improvement is possible, but I do not expect to be able to realize it. In any case, it is an active R&D project that I am doing to learn how these things work.

Finally to answer your question, I have no good answers for you (or more specifically, answers that I like). I have been trying to think of ways to do fast local inference on high end models cost effectively for months. So far, I have nothing to show for it aside from my R&D into CPU llama 3 inference since none of my ideas are able to bring hardware costs needed for llama 3.1 405B below $10,000 with performance at an acceptable level. My idea of an acceptable performance level is 10 tokens per second for token generation and 4000 tokens per second for prompt processing, although perhaps lower prompt processing performance is acceptable with prompt caching.

Post reply on HN