471–480 of 722 posts
Earlier quoted context omitted.
Sufficient for LLMs and image/video gen.
FLUX.1 D generation is about a minute at 20 steps on a 4080, but takes 35 minutes on the CPU.
Earlier quoted context omitted.
Gaming at 2160p is just too expensive still, imo. You gotta pay more for your monitor, GPU and PSU. Then if you want side monitors that match in resolution, you're paying more for those as well. You say PC gamers at the start of your comment and gaming PC enthusiasts at the end. These groups are not the same and I'd say the latter is largely doing ultrawide, 4k monitor or even 4k TV. According to steam, 56% are on 10…
I recently recently upgraded my main monitor from 1440p x 144hz to 4K x 144hz (with lots of caveats) and I agree with your assessment. If I had not made significant compromises, it would have cost at least $500 to get a decent monitor, which most people are not willing to spend. Even with this monitor, I'm barely able to run it with my (expensive, though older) graphics card, and the screen alarmingly flashes wheneve…
Who is the target audience for this? Well informed gamers know Intel's discrete GPU is hanging by a thread, so they're not hoping on that bandwagon. Too small for ML. The only people really happy seem to be the ones buying it for transcoding and I can't imagine there is a huge market of people going "I need to go buy a card for AV1 encoding".
> Too small for ML. What do you mean by this - I assume you mean too small for SoTA LLMs? There are many ML applications where 12GB is more than enough. Even w.r.t. LLMs, not everyone requires the latest & biggest LLM models. Some "small", distilled and/or quantized LLMs are perfectly usable with <24GB
Still...tangibly cheaper than even a 2nd hand 3090 so there is perhaps a market for it
Earlier quoted context omitted.
if you can see the pixels on a 27 inch 1440p display, you're just sitting too close to the screen lol
I don't directly see the pixels per se like on 1080p at 27-inch at desktop distances. But I see harsh edges in corners and text is not flawless like on 2160p. Like I said, it's on the cusp of invisible pixels.
I've not tried but I've heard that a butter-smooth 90, 120, or 300 FPS frame rate (that is also synchronized with the display) is really wonderful in many such games, and once you experience that you can't go back. On less powerful systems it then requires making a tradeoff with rendering quality and resolution.
Why don't they just release a basic GPU with 128GB RAM and eat NVidia's local generative AI lunch? The networking effect of all devs porting their LLMs etc. to that card would instantly put them as a major CUDA threat. But beancounters running the company would never get such an idea...
Disclosure: HPC admin who works with NIVIDA cards here. Because, no. It's not as simple as that. NVIDIA has a complete ecosystem now. They have cards. They have cards of cards (platforms), which they produce, validate and sell. They have NVLink crossbars and switches which connects these cards on their card of cards with very high speeds and low latency. For inter-server communication they have libraries which coordi…
Why don't they just release a basic GPU with 128GB RAM and eat NVidia's local generative AI lunch? The networking effect of all devs porting their LLMs etc. to that card would instantly put them as a major CUDA threat. But beancounters running the company would never get such an idea...
Disclosure: HPC admin who works with NIVIDA cards here. Because, no. It's not as simple as that. NVIDIA has a complete ecosystem now. They have cards. They have cards of cards (platforms), which they produce, validate and sell. They have NVLink crossbars and switches which connects these cards on their card of cards with very high speeds and low latency. For inter-server communication they have libraries which coordi…
Wow what he said is way above your head! Please reread what he wrote.
Who is the target audience for this? Well informed gamers know Intel's discrete GPU is hanging by a thread, so they're not hoping on that bandwagon. Too small for ML. The only people really happy seem to be the ones buying it for transcoding and I can't imagine there is a huge market of people going "I need to go buy a card for AV1 encoding".
> Intel's discrete GPU is hanging by a thread, so they're not hoping on that bandwagon Why would that matter? You buy one GPU, in a few years you buy another GPU. It's not a life decision.
The game devs are going to spend all their time & effort targetting amd/nvidia. Custom code paths etc.
It's not a one size fits all world. OpenCL etc abstraction are good at covering up differences, but not that good. So if you're the player with <10% market share you're going to have an uphill battle to just be on par.
Why don't they just release a basic GPU with 128GB RAM and eat NVidia's local generative AI lunch? The networking effect of all devs porting their LLMs etc. to that card would instantly put them as a major CUDA threat. But beancounters running the company would never get such an idea...
Disclosure: HPC admin who works with NIVIDA cards here. Because, no. It's not as simple as that. NVIDIA has a complete ecosystem now. They have cards. They have cards of cards (platforms), which they produce, validate and sell. They have NVLink crossbars and switches which connects these cards on their card of cards with very high speeds and low latency. For inter-server communication they have libraries which coordi…
https://github.com/ryao/llama3.c
Inference workloads are easy to parallelize to N cards with minimal connectivity between them. The Nvlink crossbars and switches just are not needed.
In particular, inference can be divided into two distinct phases, which are input processing (prompt processing) and output generation (token generation). They are remarkably different in their requirements. Input processing is compute bound via GEMM operations while output generation is memory bandwidth bound via GEMV operations. Technically, you can do the input processing via GEMV too by processing 1 token at a time, but that is slow, so you do not want to do that. Anyway, these phases can be further subdivided into the model’s layers. You can have 1 GPU per layer with the logits passing from GPU to GPU in a pipeline. The GPUs just need the layer’s weights and the key-value cache for all of the tokens in that layer in memory to be able to work effectively. For llama 3.1 405B, there are 126 layers, so that is up to 126 GPUs.
That is of course slightly slower than if you just had 1 GPU with an incredible amount of VRAM, but you can always have more than one query in flight to get better than 1 GPU’s worth of performance from this pipeline approach. There are other ways of doing parallelization too, such as having output processing use GEMM to do multiple queries in parallel. This would be what others call batching, although I am only interested in doing 1 query at a time right now, so I have not touched it.
In essence, you can connect n cards on PCIe and have them solve inferencing problems magically, with the right software. Training is a different matter and I cannot comment on it as I have not studied it yet.