Why don't they just release a basic GPU with 128GB RAM and eat NVidia's local generative AI lunch? The networking effect of all devs porting their LLMs etc. to that card would instantly put them as a major CUDA threat. But beancounters running the company would never get such an idea...
48 GB is at the tail end of what's reasonale for normal GPUs. The IO requires a lot of die space. And intel's architecture is not very space efficient right now compared to nvidia's
And even if you spend a lot of die space on memory controllers, you can only fit so many GDDR chips around the GPU core while maintaining signal integrity. HBM sidesteps that issue but it's still too expensive for anything but the highest end accelerators, and the ordinary LPDDR that Apple uses is lacking in bandwidth compared to GDDR, so they have to compensate with ginormous amounts of IO silicon. The M4 Ultra is expected to have similar bandwidth to a 4090 but the former will need a 1024bit bus to get there while the latter is only 384bit.