Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Intel announces Arc B-series "Battlemage" discrete graphics with Linux support

phoronix.com

Post reply on HN
Print view View original

541–550 of 722 posts

Re: Intel announces Arc B-series "Battlemage" discrete graphics with Linux support

#541

Post by ryao »

treprinum wrote: post #26

Why don't they just release a basic GPU with 128GB RAM and eat NVidia's local generative AI lunch? The networking effect of all devs porting their LLMs etc. to that card would instantly put them as a major CUDA threat. But beancounters running the company would never get such an idea...

Qualcomm built a card designed to do inferencing with 128GB of RAM:

https://www.qualcomm.com/news/onq/2023/11/introducing-qualco...

I have no idea how much it costs. They do not sell it via PC parts channels.

Re: Intel announces Arc B-series "Battlemage" discrete graphics with Linux support

#543

Post by ryao »

ksec wrote: post #260
treprinum wrote: post #162

Earlier quoted context omitted.

Intel has Xeon Phi which was a spin-off of their first attempt at GPU so they have a lot of tech in place they can reuse already. They don't need to go with GDDRx/HBMx designs that require large dies.

I don't want to further this discussions but may be you dont realise some of the people who replied to you either design hardware for a living or has been in the hardware industry for longer than 20 years.

While it is not a GPU, Qualcomm already made an inferencing card with 128GB RAM:

https://www.qualcomm.com/news/onq/2023/11/introducing-qualco...

It would be interesting if those saying that a regular GPU with 128GB of VRAM cannot be made would explain how Qualcomm was able to make this card. It is not a big stretch to imagine a GPU with the same memory configuration. Note that Qualcomm did not use HBM for this.

Re: Intel announces Arc B-series "Battlemage" discrete graphics with Linux support

#544

Post by ryao »

derefr wrote: post #535
ryao wrote: post #480

Earlier quoted context omitted.

I have been hacking on local llama 3 inference software (for the CPU, but I have been thinking about how I would port it to a GPU) and would like to do a rebuttal: https://github.com/ryao/llama3.c Inference workloads are easy to parallelize to N cards with minimal connectivity between them. The Nvlink crossbars and switches just are not needed. In particular, inference can be divided into two distinct phases, which a…

I presume the counterargument is that inference hosting is commoditized (sort of like how stateless CPU-based containerized workload hosts are commoditized); there’s no margin in that business, because it is parallelizable, and arbitrarily schedulable, and able to be spread across heterogenous hardware pretty easily (just route individual requests to sub-cluster A or B), preventing any kind of lock-in and thus any ki…

Intel does make cards aimed at this space too:

https://www.intel.com/content/www/us/en/products/details/pro...

Coincidentally, it has 128GB of RAM. However, it is not a GPU, is designed to do training too and uses expensive HBM.

Modern GPUs can do more than inference/training and the original poster asked about a GPU with 128GB of RAM, not a card that can only do inferencing as you described. Interestingly, Qualcomm made its own card targeted at only inferencing with 128GB of RAM without using HBM:

https://www.qualcomm.com/news/onq/2023/11/introducing-qualco...

They do not sell it through PC parts channels so I do not know the price, but it is exactly what you described and it has been built. Presumably, a GPU with the same memory configuration would be of interest to the original poster.

Re: Intel announces Arc B-series "Battlemage" discrete graphics with Linux support

#545

Post by ryao »

ksec wrote: post #306
wtallis wrote: post #53

Earlier quoted context omitted.

Just how "basic" do you think a GPU can be while having the capability to interface with that much DRAM? Getting there with GDDR6 would require a really wide memory bus even if you could get it to operate with multiple ranks. Getting to 128GB with LPDDR5x would be possible with the 256-bit bus width they used on the top parts of the last generation, but would result in having half the bandwidth of an already mediocre…

Thank You Wtallis. Somewhere along the line, this basic "knowledge" of hardware is completely lost. I dont expect this to be explained in any comment section on old Anandtech. It seems hardware enthusiast has mostly disappeared, I guess that is also why Anandtech closed. We now live in a world where most site are just BS rumours.

Qualcomm made an AI inferencing card with 128GB RAM without using HBM:

https://www.qualcomm.com/news/onq/2023/11/introducing-qualco...

Would someone with “basic ‘knowledge’ of hardware” explain why a GPU cannot be made with the same memory configuration?

Re: Intel announces Arc B-series "Battlemage" discrete graphics with Linux support

#546

Post by ryao »

amelius wrote: post #209
wtallis wrote: post #53

Earlier quoted context omitted.

Just how "basic" do you think a GPU can be while having the capability to interface with that much DRAM? Getting there with GDDR6 would require a really wide memory bus even if you could get it to operate with multiple ranks. Getting to 128GB with LPDDR5x would be possible with the 256-bit bus width they used on the top parts of the last generation, but would result in having half the bandwidth of an already mediocre…

What if they put 8 identical GPUs in the package, each with 1/8 the memory? Would that be a useful configuration for a modern LLM?

Yes:

https://news.ycombinator.com/item?id=42313615

Re: Intel announces Arc B-series "Battlemage" discrete graphics with Linux support

#547

Post by ryao »

SoothingSorbet wrote: post #537
ryao wrote: post #488

Earlier quoted context omitted.

Inferencing is much more simple than you think: See the precompute_input_logits() and forward() functions here: https://github.com/ryao/llama3.c/blob/master/run.c#L520 As a preface, precompute_input_logits() is really just a generalized version of the forward() function that can operate on multiple input tokens at a time to do faster input processing, although it can be used in place of the forward() function for out…

> but sadly, the secrets of how I am doing it are in Intel’s proprietary cblas_sgemm_batch() function. Perhaps you can reverse engineer it?

My plan is to make more attempts at rolling my own. Reverse engineering things is not something that I do, since that would prevent me from publishing the result as OSS.

Re: Intel announces Arc B-series "Battlemage" discrete graphics with Linux support

#548

Post by MindSpunk »

kimixa wrote: post #523
whalesalad wrote: post #393

Earlier quoted context omitted.

I still don't understand why graphics cards haven't evolved to include sodimm slots so that the vram can be upgraded by the end user. At this point memory requirements vary so much from gamer to scientist so it would make more sense to offer compute packages with user-supplied memory. tl;dr GPU's need to transition from being add-in cards to being a sibling motherboard. A sisterboard? Not a daughter board.

One of the reasons GPUs can have multiples of CPU bandwidth is they avoid the difficulties of pluggable dimms - direct soldered can have much higher frequencies at lower power. It's one of the reasons why ARM Macbooks get great performance/watt, memory being even "closer" than mainboard soldered RAM so getting more of those benefits, though naturally less flexibility.

Even DDR5 has this problem. Go look at what soldered DDR5 can do frequency wise compared to DIMMs. It's one of the problems the new CAMM form factor aims to help solve, making it tractable to push the memory frequency up beyond what DIMMs can get yout currently.

Re: Intel announces Arc B-series "Battlemage" discrete graphics with Linux support

#549

Post by ryao »

pixelpoet wrote: post #336
timschmidt wrote: post #327

Earlier quoted context omitted.

$1,961.19: H13SSL-N Motherboard And EPYC 9334 QS CPU + DDR5 4*128GB 2666MHZ REG ECC RAM Server motherboard kit https://www.aliexpress.us/item/3256807766813460.html Doesn't seem like 20x to me. I'm sure spending more than 30 seconds searching could find even better deals.

Isn't 2666 MHz ECC RAM obscenely slow? 32 cores without the fast AVX-512 of Zen5 isn't what anyone is looking for in terms of floating point throughput (ask me about electricity prices in Germany), and for that money I'd rather just take a 4090 with 24GB memory and do my own software fixed point or floating point (which is exactly what I do personally and professionally). This is exactly what I meant about Intel's re…

Yes and no. I have been developing some local llama 3 inference software on a machine with 3200MT/s ECC RAM and a Ryzen 7 5800X:

https://github.com/ryao/llama3.c

My experience is that input processing (prompt processing) is compute bottlenecked in GEMM. AVX-512 would help there, although my CPU’s Zen 3 cores do not support it and the memory bandwidth does not matter very much. For output generation (token generation), memory bandwidth is a bottleneck and AVX-512 would not help at all.

Re: Intel announces Arc B-series "Battlemage" discrete graphics with Linux support

#550

Post by [deleted] »

jsheard wrote: post #289
dragontamer wrote: post #285

Earlier quoted context omitted.

Because LPDDR5x is soldered on RAM. Everyone else wants configurable RAM that scales both down (to 16GB) and up (to 2TB), to cover smaller laptops and bigger servers. GPUs with soldered on RAM has 500GB/sec bandwidths, far in excess of Apples chips. So the 8GB or 16GB offered by NVidia or AMD is just far superior at vid o game graphics (where textures are the priority)

> GPUs with soldered on RAM has 500GB/sec bandwidths, far in excess of Apples chips. Apple is doing 800GB/sec on the M2 Ultra and should reach about 1TB/sec with the M4 Ultra, but that's still lagging behind GPUs. The 4090 was already at the 1TB/sec mark two years ago, the 5090 is supposedly aiming for 1.5TB/sec, and the H200 is doing 5TB/sec.

[deleted]
Post reply on HN