Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Intel announces Arc B-series "Battlemage" discrete graphics with Linux support

phoronix.com

Post reply on HN
Print view View original

351–360 of 722 posts

Re: Intel announces Arc B-series "Battlemage" discrete graphics with Linux support

#351

Post by CoastalCoder »

throwaway48476 wrote: post #338
CoastalCoder wrote: post #150

Earlier quoted context omitted.

> Buying a GPU is also not "investing" in anything. It is in the (minor) sense that I'd rely on Intel for warranty support, driver updates (if closed source), and firmware fixes. But I agree with your main point that the worst-case downside isn't that big of a deal.

There's no way you're going to maintain and develop the intel linux driver as a solo dev.

> There's no way you're going to maintain and develop the intel linux driver as a solo dev.

I agree entirely.

My point was that even if Intel disappeared tomorrow, there's a good chance that Linux developer community would take over maintenance of those drivers.

In contrast to, e.g., 10-years-ago nvidia, where IIUC it was very difficult for outsiders to obtain the documentation needed to write proper drivers for their GPUs.

Re: Intel announces Arc B-series "Battlemage" discrete graphics with Linux support

#352

Post by m00x »

treprinum wrote: post #137
Muskyinhere wrote: post #114

Earlier quoted context omitted.

No one is running LLMs on consumer NVidia GPUs or apple MacBooks. A dev, if they want to run local models, probably run something which just fits on a proper GPU. For everything else, everyone uses an API key from whatever because its fundamentaly faster. IF a affordable intel GPU would be relevant faster for inferencing, is not clear at all. A 4090 is at least double the speed of Apples GPU.

4090 is 5x faster than M3 Max 128GB according to my tests but it can't even inference LLaMA-30B. The moment you hit that memory limit the inference is suddenly 30x slower than M3 Max. So a basic GPU with 128GB RAM would trash 4090 on those larger LLMs.

Do you have the code for that test?

Re: Intel announces Arc B-series "Battlemage" discrete graphics with Linux support

#353

Post by refulgentis »

marcyb5st wrote: post #263
m00x wrote: post #136

Earlier quoted context omitted.

You can just use a CPU in that case, no? You can run most ML inference on vectorized operations on modern CPUs at a fraction of the price.

My 7800x says not really. Compared to my 3070 it feels so incredibly slow that gets in the way of productivity. Specifically, waiting ~2 seconds vs ~20 for a code snippet is much more detrimental to my productivity than the time difference would suggest. In ~2 seconds I don't get distracted, in ~20 seconds my mind starts wandering and then I have to spend time refocusing. Make a GPU that is 50% slower than a 2 genera…

I don't understand, make it slower so it's faster?

Re: Intel announces Arc B-series "Battlemage" discrete graphics with Linux support

#354

Post by phkahler »

bayindirh wrote: post #307
throwaway48476 wrote: post #288

Earlier quoted context omitted.

All of that is highly relevant for training but what the poster was asking for is a desktop inference card.

You use at least half of this stack for desktop setups. You need copying daemons, the ecosystem support (docker-nvidia, etc.), some of the libraries, etc. even when you're on a single system. If you're doing inference on a server; MIG comes into play. If you're doing inference on a larger cloud, GPU-direct storage comes into play. It's all modular.

No. I've been reading up. I'm planning to run Flux 12b on my AMD 5700G with 64GB RAM. CPU will take 5-10minutes per image which will be fine for me tinkering while writing code. Maybe I'll be able to get the GPU going on it too.

Point of the OP is this is entirely possible with even an iGPU if only we have the RAM. nVidia should be irrelevant for local inference.

Re: Intel announces Arc B-series "Battlemage" discrete graphics with Linux support

#355

Post by m00x »

marcyb5st wrote: post #263
m00x wrote: post #136

Earlier quoted context omitted.

You can just use a CPU in that case, no? You can run most ML inference on vectorized operations on modern CPUs at a fraction of the price.

My 7800x says not really. Compared to my 3070 it feels so incredibly slow that gets in the way of productivity. Specifically, waiting ~2 seconds vs ~20 for a code snippet is much more detrimental to my productivity than the time difference would suggest. In ~2 seconds I don't get distracted, in ~20 seconds my mind starts wandering and then I have to spend time refocusing. Make a GPU that is 50% slower than a 2 genera…

I'm running codeseeker 13B model on my macbook with no perf issues and I get a response within a few seconds.

Running a specialist model makes more sense on small devices.

Re: Intel announces Arc B-series "Battlemage" discrete graphics with Linux support

#356

Post by magicalhippo »

treprinum wrote: post #190
dimensi0nal wrote: post #172

Earlier quoted context omitted.

The only consumer demand for local AI models is for generating pornography

How about running your intelligent home with a voice assistant on your own computer? In privacy-oriented countries (Germany) that would be massive.

This is what I'm fiddling with. My 2080Ti is not quite enough to make it viable. I find the small models fail too often, so need larger Whisper and LLM models.

Like the 4060 Ti would have been a nice fit if it hadn't been for the narrow memory bus, which makes it slower than my 2080 Ti for LLM inference.

A more expensive card has the downside of not being cheap enough to justify idling in my server, and my gaming card is at times busy gaming.

Re: Intel announces Arc B-series "Battlemage" discrete graphics with Linux support

#357

Post by treprinum »

amelius wrote: post #209
wtallis wrote: post #53

Earlier quoted context omitted.

Just how "basic" do you think a GPU can be while having the capability to interface with that much DRAM? Getting there with GDDR6 would require a really wide memory bus even if you could get it to operate with multiple ranks. Getting to 128GB with LPDDR5x would be possible with the 256-bit bus width they used on the top parts of the last generation, but would result in having half the bandwidth of an already mediocre…

What if they put 8 identical GPUs in the package, each with 1/8 the memory? Would that be a useful configuration for a modern LLM?

K80 used to be two glued K40 but their interconnect was barely faster than PCIe so it didn't have much benefit as one had to move stuff between two internal GPUs anyway.

Re: Intel announces Arc B-series "Battlemage" discrete graphics with Linux support

#358

Post by bayindirh »

landryraccoon wrote: post #342
bayindirh wrote: post #307

Earlier quoted context omitted.

You use at least half of this stack for desktop setups. You need copying daemons, the ecosystem support (docker-nvidia, etc.), some of the libraries, etc. even when you're on a single system. If you're doing inference on a server; MIG comes into play. If you're doing inference on a larger cloud, GPU-direct storage comes into play. It's all modular.

It's possible you're underestimating the open source community. If there's a competing platform that hobbyists can tinker with, the ecosystem can improve quite rapidly, especially when the competing platform is completely closed and hobbyists basically are locked out and have no alternative.

> It's possible you're underestimating the open source community.

On the contrary. You really don't know how I love and prefer open source and love a more leveling playing field.

> If there's a competing platform that hobbyists can tinker with...

AMD's cards are better from hardware and software architecture standpoint, but the performance is not there yet. Plus, ROCm libraries are not that mature, but they're getting there. Developing high performance, high quality code is deceivingly expensive, because it's very heavy in theory, and you fly very close to the metal. I did that in my Ph.D., so I know what it entails. So it requires more than a couple (hundred) hobbyists to pull off (see the development of Eigen linear algebra library, or any high end math library).

Some big guns are pouring money into AMD to implement good ROCm libraries, and it started paying off (Debian has a ton of ROCm packages now, too). However, you need to be able to pull it off in the datacenter to be able to pull it off on the desktop.

AMD also needs to be able to enable ROCm on desktop properly, so people can start hacking it at home.

> especially when the competing platform is completely closed...

NVIDIA gives a lot of support to universities, researchers and institutions who play with their cards. Big cards may not be free, but know-how, support and first steps are always within reach. Plus, their researchers dogfood their own cards, and write papers with them.

So, as long as papers got published, researchers do their research, and something got invented, many people don't care about how open source the ecosystem is. This upsets me a ton, but when closed source AI companies and researchers who forget to add crucial details to their papers so what they did can't be reproduced don't care about open source, because they think like NVIDIA. "My research, my secrets, my fame, my money".

It's not about sharing. It's about winning, and it's ugly in some aspects.

Re: Intel announces Arc B-series "Battlemage" discrete graphics with Linux support

#359

Post by kimixa »

dragontamer wrote: post #326
jsheard wrote: post #289

Earlier quoted context omitted.

> GPUs with soldered on RAM has 500GB/sec bandwidths, far in excess of Apples chips. Apple is doing 800GB/sec on the M2 Ultra and should reach about 1TB/sec with the M4 Ultra, but that's still lagging behind GPUs. The 4090 was already at the 1TB/sec mark two years ago, the 5090 is supposedly aiming for 1.5TB/sec, and the H200 is doing 5TB/sec.

HBM is kind of not fair lol. But 4096-line bus is gonna have more bandwidth than any competitor. It's pretty expensive though. The 500GB/sec number is for a more ordinary GPU like the B580 Battlemage in the $250ish price range. Obviously the $2000ish 4090 will be better, but I don't expect the typical consumer to be using those.

But an on-package memory bus has some of the advantages of HBM, just to a lesser extent, so it's arguably comparable as an "intermediate stage" between RAM chips and HBM. Distances are shorter (so voltage drop and capacitance are lower, so can be driven at lower power), routing is more complex but can be worked around by more layers, which increases cost but on a significantly smaller area than required for dimms, and the dimms connections themselves can hurt performance (reflection from poor contacts, optional termination makes things more complex, and the expectations of mix-and-match for dimm vendors and products likely reduce fine tuning possibilities).

There's pretty much a direct opposite scaling between flexibility and performance - dimms > soldered ram > on-package ram > die-interconnects.

Re: Intel announces Arc B-series "Battlemage" discrete graphics with Linux support

#360

Post by arcticbull »

dgfitz wrote: post #292
arcticbull wrote: post #291

Earlier quoted context omitted.

Having the complete ecosystem affords them significant margins.

Against what?

As of today they have SaaS company margins as a hardware company which is practically unheard of.
Post reply on HN