81–90 of 127 posts
Earlier quoted context omitted.
> It's not clear from the above what a "CUDA core" (singular) _is_ A CUDA core is basically a SIMD lane on an actual core on an NVIDIA GPUs. For a longer version of this answer: https://stackoverflow.com/a/48130362/1593077
So it's a "SIMD lane" that can itself perform actual SIMD instructions? I think you want a metaphor that doesn't also depend on its literal meaning.
Earlier quoted context omitted.
Sure, but you can make money in the field and retire faster than it becomes irrelevant. FWIW none of the ideas here are novel or nontransferable–it's just the specific design that is proprietary. Understanding how to do an AllReduce has been of theoretical interest for decades and will probably remain worth doing far into the future.
Tech is always like this. You move from one thing to the next. With your transferable skills, experience and thinking that is beyond one programming language. Even Apple is simply exporting to CUDA now.
This is like when journalists write clickbait article titles by omitting all qualifiers (eg "states banning fluoride" when it's only some states).
One framework added a CUDA backend. You think all of Apple uses only one framework? Further what makes you think this even gets internal use?
Earlier quoted context omitted.
It's been a while since I thought about this but isn't the reason providers advertise only 3.2tbps because that's the limit of a single node's connection to the IB network? DGX is spec'ed to pair each H100 with a Connect-X 7 NIC and those cap out at 400gbps. 8 gpus * 400gbps / gpu = 3.2tbps. Quiz 2 is confusingly worded but is, iiuc, referring to intranode GPU connections rather than internode networking.
Yes, 450GB/s is the per GPU bandwidth in the nvlink domain. 3.2Tbps is the per-host bandwidth in the scale out IB/Ethernet domain.
I find the piece, much like a lot of other documentation, "imprecise". Like most such efforts, it likely caters to a group of people expected to benefit from being explained what a GPU is, but it fumbles it terms, e.g. (the first image with burned-in text): > The "Warp Scheduler" is a SIMD vector unit like the TPU VPU with 32 lanes, called "CUDA Cores" It's not clear from the above what a "CUDA core" (singular) _is_…
From the resource intro: > Expected background: We’re going to assume you have a basic understanding of LLMs and the Transformer architecture but not necessarily how they operate at scale.
I suppose this doesn’t require any knowledge about how computers work, but core CPU functionality seems…reasonable?
I find the piece, much like a lot of other documentation, "imprecise". Like most such efforts, it likely caters to a group of people expected to benefit from being explained what a GPU is, but it fumbles it terms, e.g. (the first image with burned-in text): > The "Warp Scheduler" is a SIMD vector unit like the TPU VPU with 32 lanes, called "CUDA Cores" It's not clear from the above what a "CUDA core" (singular) _is_…
I find the piece, much like a lot of other documentation, "imprecise". Like most such efforts, it likely caters to a group of people expected to benefit from being explained what a GPU is, but it fumbles it terms, e.g. (the first image with burned-in text): > The "Warp Scheduler" is a SIMD vector unit like the TPU VPU with 32 lanes, called "CUDA Cores" It's not clear from the above what a "CUDA core" (singular) _is_…
Earlier quoted context omitted.
So it's a "SIMD lane" that can itself perform actual SIMD instructions? I think you want a metaphor that doesn't also depend on its literal meaning.
Nvidia’s marketing team uses confusing terminology to make their product sound cooler than it is. An Intel “core” can perform AVX512 SIMD instructions that involve 16 lanes of 32-bit data. Intel cores are packaged in groups of up to 16. And, they use hyperthreading, speculative execution and shadow registers to cover latency. An Nvidia “Streaming Multiprocessor” can perform SIMD instructions on 32 lanes of 32-bits ea…
Earlier quoted context omitted.
Yes, 450GB/s is the per GPU bandwidth in the nvlink domain. 3.2Tbps is the per-host bandwidth in the scale out IB/Ethernet domain.
I believe this is correct. For an H100, the 4 NVLink switches each have 64 ports supporting 25GB/s each, and each GPU uses a total of 18 ports. This gives us 450GB/s bandwidth within the node. But once you start trying to leave the node, you're limited by the per-node InfiniBand cabling, which only gives you 400GB/s out of the entire node (50GB / GPU).
The article is consistent and uses Gigabytes.
I find the piece, much like a lot of other documentation, "imprecise". Like most such efforts, it likely caters to a group of people expected to benefit from being explained what a GPU is, but it fumbles it terms, e.g. (the first image with burned-in text): > The "Warp Scheduler" is a SIMD vector unit like the TPU VPU with 32 lanes, called "CUDA Cores" It's not clear from the above what a "CUDA core" (singular) _is_…
Shamelessly responding as the author. I (mostly) agree with you here. > please be surgically precise with your terms There's always a tension between precision in every explanation and the "moral" truth. I can say "a SIMD (Single Instruction Multiple Data) vector unit like the TPU VPU with 32 ALUs (SIMD lanes) which NVIDIA calls CUDA Cores", which starts to get unwieldy and even then leaves terms like vector units un…
2) What are your thoughts on links to the wiki articles under things such as "SIMD" or "ALUs" for the precise meaning while using the metaphors in your prose?
Most novices tend to Google and end up on Wikipedia for the trees. It's harder to find the forest.
We should remember that these structural diagrams are _not_ necessarily what NVIDIA actually has as hardware. They carefully avoid guaranteeing that any of the entities or blocks you see in the diagrams actually _exist_. It is still just a mental model NVIDIA offers for us to think about their GPUs, and more specifically the SMs, rather than a simplified circuit layout. For example, we don't know how many actual func…
Interesting perspective. Aren't SMs basically blocked while running tensor core operations, which might hint that it's the same FPUs doing the work after all?