91–100 of 776 posts
32GB of GDDR7 at 1.8TB/sec for $2000, best of luck to the gamers trying to buy one of those while AI people are buying them by the truckload. Presumably the pro hardware based on the same silicon will have 64GB, they usually double whatever the gaming cards have.
At what point do we stop calling them graphics cards?
It's a shame to see they max out at just 32GB, for that price in 2025 you'd be hoping for a lot more, especially with Apple Silicon - while not nearly as fast - being very usable with 128GB+ for LLMs for $6-7k USD (comes with a free laptop too ;))
Presumably the workstation version will have 64GB of VRAM. By the way, this is even better as far as memory size is concerned: https://www.asrockrack.com/minisite/AmpereAltraFamily/ However, memory bandwidth is what matters for token generation. The memory bandwidth of this is only 204.8GB/sec if I understand correctly. Apple's top level hardware reportedly does 800GB/sec.
Earlier quoted context omitted.
Presumably the workstation version will have 64GB of VRAM. By the way, this is even better as far as memory size is concerned: https://www.asrockrack.com/minisite/AmpereAltraFamily/ However, memory bandwidth is what matters for token generation. The memory bandwidth of this is only 204.8GB/sec if I understand correctly. Apple's top level hardware reportedly does 800GB/sec.
All of this is true only while no software is utilizing parallel inference of multiple LLM queries. The Macs will hit the wall.
Earlier quoted context omitted.
Presumably the workstation version will have 64GB of VRAM. By the way, this is even better as far as memory size is concerned: https://www.asrockrack.com/minisite/AmpereAltraFamily/ However, memory bandwidth is what matters for token generation. The memory bandwidth of this is only 204.8GB/sec if I understand correctly. Apple's top level hardware reportedly does 800GB/sec.
AMD Strix Halo is 256GB/sec or so. Similarly AMD's Epyc Sienna family is similar. The EPYC turin family (zen 5) has 576GB/sec or so per socket. Not sure how well any of them do on LLMs. Bandwidth helps, but so does hardware support for FP8 or FP4.
As for how well those CPUs do with LLMs. The token generation will be close to model size / memory bandwidth. At least, that is what I have learned from local experiments:
https://github.com/ryao/llama3.c
Note that prompt processing is the phase where the LLM is reading the conversation history and token generation is the phase where the LLM is writing a response.
By the way, you can get an ampere altra motherboard + CPU for $1,434.99:
https://www.newegg.com/asrock-rack-altrad8ud-1l2t-q64-22-amp...
I would be shocked if you can get any EYPC CPU with similar/better memory bandwidth for anything close to that price. As for Strix Halo, anyone doing local inference would love it if it is priced like a gaming part. 4 of them could run llama 3.1 405B on paper. I look forward to seeing its pricing.
Earlier quoted context omitted.
Presumably the workstation version will have 64GB of VRAM. By the way, this is even better as far as memory size is concerned: https://www.asrockrack.com/minisite/AmpereAltraFamily/ However, memory bandwidth is what matters for token generation. The memory bandwidth of this is only 204.8GB/sec if I understand correctly. Apple's top level hardware reportedly does 800GB/sec.
All of this is true only while no software is utilizing parallel inference of multiple LLM queries. The Macs will hit the wall.
Earlier quoted context omitted.
The 4090 already seemed positioned as a card for consumer AI enthusiast workloads. But this $1000 price gap between the 5080 and 5090 seems to finally cement that. Though we're probably still going to see tons of tech YouTubers making videos specifically about how the 5090 isn't a good value for gaming as if it even matters. The people who want to spend $2000 on a GPU for gaming don't care about the value and everyon…
From all the communication I’ve had with Nvidia, the prevailing sentiment was that the 4090 was an 8K card, that happened to be good for AI due to vram requirements from 8K gaming. However, I’m a AAA gamedev CTO and they might have been telling me what the card means to me .
Even though they are all marketed as gaming cards, Nvidia is now very clearly differentiating between 5070/5070 Ti/5080 for mid-high end gaming and 5090 for consumer/entry-level AI. The gap between xx80 and xx90 is going to be too wide for regular gamers to cross this generation.
The 4090 already seemed positioned as a card for consumer AI enthusiast workloads. But this $1000 price gap between the 5080 and 5090 seems to finally cement that. Though we're probably still going to see tons of tech YouTubers making videos specifically about how the 5090 isn't a good value for gaming as if it even matters. The people who want to spend $2000 on a GPU for gaming don't care about the value and everyon…
Did they discontinue Titan series for good?
Earlier quoted context omitted.
The 4090 already seemed positioned as a card for consumer AI enthusiast workloads. But this $1000 price gap between the 5080 and 5090 seems to finally cement that. Though we're probably still going to see tons of tech YouTubers making videos specifically about how the 5090 isn't a good value for gaming as if it even matters. The people who want to spend $2000 on a GPU for gaming don't care about the value and everyon…
From all the communication I’ve had with Nvidia, the prevailing sentiment was that the 4090 was an 8K card, that happened to be good for AI due to vram requirements from 8K gaming. However, I’m a AAA gamedev CTO and they might have been telling me what the card means to me .
https://www.nvidia.com/en-us/geforce/news/geforce-rtx-3090-8...
Earlier quoted context omitted.
From all the communication I’ve had with Nvidia, the prevailing sentiment was that the 4090 was an 8K card, that happened to be good for AI due to vram requirements from 8K gaming. However, I’m a AAA gamedev CTO and they might have been telling me what the card means to me .
I do recall an 8K push but I thought that was on the 3090 (and was conditional on DLSS doing the heavy lifting). I don't remember any general marketing about the 4090 being an 8K card but I could very well have missed it or be mixing things up! I mean it does make sense to market it for 8K since anyone who is trying to drive that many pixels when gaming probably has deep pockets.
https://www.nvidia.com/en-us/geforce/technologies/8k/
That said, I recall that the media was more enthusiastic about christening the 4090 as an 8K card than Nvidia was:
https://wccftech.com/rtx-4090-is-the-first-true-8k-gaming-gp...