61–70 of 722 posts
Why don't they just release a basic GPU with 128GB RAM and eat NVidia's local generative AI lunch? The networking effect of all devs porting their LLMs etc. to that card would instantly put them as a major CUDA threat. But beancounters running the company would never get such an idea...
Why don't they just release a basic GPU with 128GB RAM and eat NVidia's local generative AI lunch? The networking effect of all devs porting their LLMs etc. to that card would instantly put them as a major CUDA threat. But beancounters running the company would never get such an idea...
Just how "basic" do you think a GPU can be while having the capability to interface with that much DRAM? Getting there with GDDR6 would require a really wide memory bus even if you could get it to operate with multiple ranks. Getting to 128GB with LPDDR5x would be possible with the 256-bit bus width they used on the top parts of the last generation, but would result in having half the bandwidth of an already mediocre…
Why don't they just release a basic GPU with 128GB RAM and eat NVidia's local generative AI lunch? The networking effect of all devs porting their LLMs etc. to that card would instantly put them as a major CUDA threat. But beancounters running the company would never get such an idea...
In this case, the die I/O limits precludes more than a reasonable number of DDR channels.
Why don't they just release a basic GPU with 128GB RAM and eat NVidia's local generative AI lunch? The networking effect of all devs porting their LLMs etc. to that card would instantly put them as a major CUDA threat. But beancounters running the company would never get such an idea...
GDDR isnt like the ram that connects to cpu, it's much more difficult and expensive to add more. You can get up to 48GB with some expensive stacked gddr, but if you wanted to add more stacks you'd need to solve some serious signal timing related headaches that most users wouldn't benefit from. I think the high memory local inference stuff is going to come from "AI enabled" cpus that share the memory in your computer.…
Earlier quoted context omitted.
48 GB is at the tail end of what's reasonale for normal GPUs. The IO requires a lot of die space. And intel's architecture is not very space efficient right now compared to nvidia's
> The IO requires a lot of die space. And even if you spend a lot of die space on memory controllers, you can only fit so many GDDR chips around the GPU core while maintaining signal integrity. HBM sidesteps that issue but it's still too expensive for anything but the highest end accelerators, and the ordinary LPDDR that Apple uses is lacking in bandwidth compared to GDDR, so they have to compensate with ginormous am…
Earlier quoted context omitted.
Because if they could just do that and it would rival what NVidia has, they would just do it. But obvoiusly they don't. And for reasons: NVidia has worked on CUDA for ages, do you believe they just replace this whole thing in no time?
llama.cpp and its derivatives say yes.
I wanted to have alternative choices than Nvidia for high power GPUs. Then the more I thought about it, the more it made sense to rent cloud services for AI/ML workloads and lesser powered ones for gaming. The only use cases I could come up with for wanting high-end cards are 4k gaming (a luxury I can't justify for infrequent use) or for PC VR which may still be valid if/when a decent OLED (or mini-OLED) headset is a…
IMO you want those frames getting rendered as close to the monitor as possible, and you'd probably have a better time with lower fidelity graphics rendered locally. You'd also get to keep gaming during a network outage.
Why don't they just release a basic GPU with 128GB RAM and eat NVidia's local generative AI lunch? The networking effect of all devs porting their LLMs etc. to that card would instantly put them as a major CUDA threat. But beancounters running the company would never get such an idea...
They are probably held back by same reason thats preventing AMD and nVidia from doing it either.