Intel announces Arc B-series "Battlemage" discrete graphics with Linux support

There isn't a cheaper card that's worth buying over using the iGPU you already have so yeah, that's the low end.

> If they would only actually open source their driver the community would have made their cards better than nvidia ones long ago.

Software wise, maybe. But you can't change AMD's hardware with a magic wand, and that's where a lot of CUDA's optimizations come from. AMD's GPU architecture is optimized for raster compute, and it's been that way for decades.

I can assure you that AMD does not have a magic button to press that would make their systems competitive for AI. If that was possible it would have been done years ago, with or without their consent. The problem is deeper and extends to design decisions and disagreement over the complexity of GPU designs. If you compare AMD's cards to Nvidia on "fair ground" (eg. no CUDA, only OpenCL) the GPGPU performance still leans in Nvidia's favor.

The question is why Intel GPUs, which already have soldered memory, aren't sold with more of it. The market here isn't something that can beat enterprise GPUs at training, it's something that can beat desktop CPUs at inference with enough VRAM to fit large models at an affordable price.

Inferencing is much more simple than you think:

See the precompute_input_logits() and forward() functions here:

https://github.com/ryao/llama3.c/blob/master/run.c#L520

As a preface, precompute_input_logits() is really just a generalized version of the forward() function that can operate on multiple input tokens at a time to do faster input processing, although it can be used in place of the forward() function for output generation just by passing only a single token at a time.

Also, my apologies for the code being a bit messy. matrix_multiply() and batched_matrix_multiply() are wrappers for GEMM, which I ended up having to use directly anyway when I needed to do strided access. Then matmul() is a wrapper for GEMV, which is really just a special case of GEMM. This is a work in progress personal R&D project that is based on prior work others did (as it spared me from having to do the legwork to implement the less interesting parts of inferencing), so it is not meant to be pretty.

Anyway, my purpose in providing that link is to show what is needed to do inferencing (on llama 3). You have a bunch of matrix weights, plus a lookup table for vectors that represent tokens, in memory. Then your operations are:

  * memcpy()
  * memset()
  * GEMM (GEMV is a special case of GEMM)
  * sinf()
  * cosf()
  * expf()
  * sqrtf()
  * rmsnorm (see the C function for the definition)
  * softmax (see the C function for the definition)
  * Addition, subtraction, multiplication and division.

I specify rmsnorm and softmax for completeness, but they can be implemented in terms of the other operations.

If you can do those, you can do inferencing. You don’t really need very specialized things. Over 95% of time will be spent in GEMM too.

My next steps likely will be to figure out how to implement fast GEMM kernels on my CPU. While my own SGEMV code outperforms the Intel MKL SGEMV code on my CPU (Ryzen 7 5800X where 1 core can use all memory bandwidth), my initial attempts at implementing SGEMM have not fared quite so well, but I will likely figure it out eventually. After I do, I can try adapting this to FP16 and then memory usage will finally be low enough that I can port it to a GPU with 24GB of VRAM. That would enable me to do what I say is possible rather than just saying it as I do here.

By the way, the llama.cpp project has already figured all of this out and has things running on both GPUs and CPUs using just about every major quantization. I am rolling my own to teach myself how things work. By happy coincidence, I am somehow outperforming llama.cpp in prompt processing on my CPU but sadly, the secrets of how I am doing it are in Intel’s proprietary cblas_sgemm_batch() function. However, since I know it is possible for the hardware to perform like that, I can keep trying ideas for my own implementation until I get something that performs at the same level or better.

Intel management is just brain dead. They could have sold the cards for mining when there was a massive GPU shortage and called it the developer edition but no. It's hard to develop a driver for games when you have no silicon.