Intel announces Arc B-series "Battlemage" discrete graphics with Linux support

If Intel or AMD sold a niche product with 48GB RAM even at a loss, but hit high-end consumer pricing, there would be a flood of people doing various AI work to buy it. The end result would be that parts of NVidia's moat would start draining rather quickly, and AMD / Intel would be in a stronger position for AI products.

I use NVidia because when I bought AMD during the GPU shortage, ROCm simply didn't work for AI. This was a few years back, but I was burned badly enough that I'm unlikely to risk AMD again for a long, long time. Unused code sits broken, and no ecosystem gets built up. A few years later, things are gradually improving for AMD for the kinds of things I wanted to do years ago, but all my code is already built around NVidia, and all my computers have NVidia cards. It's a project with users, and all those users are buying NVidia as well (even if just for surface dependencies, like dev-ops scripts which install CUDA). That, times thousands of projects, is part of NVidia's moat.

If I could build a cheap system with around 200GB, that would be incentive for me to move the relatively surface dependencies to work on a different platform. I can buy a motherboard with four PCI slots, and plug in four 48GB cards to get there. I'd build things around Intel or AMD instead.

The alternative is NVidia would start shipping competitive cards. If they did that, their high-end profit margins would dissolve.

The breakpoints for inference functionality are really at around 16GB, 48GB, and 200GB, for various historical reasons.

If I need 300GB/s memory bandwidth for my workload, that can be accomplished with:

* One RAM chip with 300GB/s

* Two RAM chips with 150GB/s each

* Four RAM chips with 75GB/s each

Etc.

Stepping up from 16GB to 196GB, the bandwidth requirements for each chip go down 10-fold, and you can use much cheaper RAM as a result. And all the signalling requirements relax too.

Much of this discussion presumes a 200GB card would individually need the same capacity to each RAM chip as a 12GB card. This is just false. An A770 or 4060-grade card couldn't keep up with that much data. And if I'm using a small model, I can get the same bandwidth by properly distributing it among RAM chips (which most hardware does automatically).

An A770 or 4060-grade card, with the same total memory capacity as we have today, but 200GB RAM, would allow us to run high-quality LLMs locally or do high-resolution renders. That wouldn't have the same performance as a $200k card, obviously, but for many inferences uses, that's just not very important.

If I were buying for my own uses, I'd want 12x 32GB PC3200 DIMMs for a total of 384GB RAM at $600 for the RAM (say $2k total), with an individual throughput of 25GB/sec and a total throughput of 300GB/sec. I'd be okay with 4060-grade performance. My own uses are a bit niche, and I think for most other people's uses, something with a little more throughput and a little less capacity (48-196GB) might make more sense. But you definitely don't need the same throughput as existing GPU RAM.

Beyond that, you'd have to move to GDDR7 (which has 24Gb/3GB chips incoming) or to HBM stacks, but at that point you're well beyond a "basic GPU". I think the only way you could get to 128GB would be either using regular (LP)DDR or HBM.

Note, Apple M chips have weak GPUs with decent MBW and large memory capacities (up to 192GB @ 800GB/s for an M2 Ultra, launched mid 2023) and have not been a major CUDA threat so I don't think your hypothesis actually stands up.