111–120 of 183 posts
80 billion transistors boggles my mind. How many molecules are their per transistor?
Earlier quoted context omitted.
You are indeed correct, I was (kinda) fooled by the marketing and I think that TF32 is deceptively named. I think the tensor cores are being used in this architecture for FP64 and 60 TFlops is still pretty decent. I'm on Turing architecture so I've never used TF32. I've only used FP32 and FP16 but FP32 isn't supported by these tensor cores.
Well the addition is done in FP32, and it's a 32-bit storage format in memory, so calling it a 16-bit format isn't right either. It's really a hybrid format where everything is 32-bit except multiplication. Given that it's 32-bit in memory (so all your data structures are 32-bit) and also that in my experience using it is very transparent (I haven't run into any numerical issues compared to full FP32), I think callin…
Addition is done in 10-bit mantissa. So maybe TF19 might be the better name, since its a 19-bit format (slightly more than 16-bit BFloats).
Really, its a BFloat with a 10-bit mantissa instead of a 7-bit mantissa. 10-bit mantissa matches FP16, while the 8-bit exponent matches FP32.
So TF19 probably would have been the best name, but NVidia like marketing so they call it TF32 instead.
Earlier quoted context omitted.
Well the addition is done in FP32, and it's a 32-bit storage format in memory, so calling it a 16-bit format isn't right either. It's really a hybrid format where everything is 32-bit except multiplication. Given that it's 32-bit in memory (so all your data structures are 32-bit) and also that in my experience using it is very transparent (I haven't run into any numerical issues compared to full FP32), I think callin…
> Well the addition is done in FP32 Addition is done in 10-bit mantissa. So maybe TF19 might be the better name, since its a 19-bit format (slightly more than 16-bit BFloats). Really, its a BFloat with a 10-bit mantissa instead of a 7-bit mantissa. 10-bit mantissa matches FP16, while the 8-bit exponent matches FP32. So TF19 probably would have been the best name, but NVidia like marketing so they call it TF32 instead…
Off topic but I can't stand when corporations use actual people's names for their marketing who never gave them the permission to do so. For something like Shakespeare or Cicero I'm OK with it but Grace Hopper was alive in my lifetime, and even Tesla feels a little weird. What gives you the right to use that person's reputation to shill your product?
> What gives you the right to use that person's reputation to shill your product? Practically speaking you have the right to do anything unless someone complains about it. A lot of popular figures, even those long dead, have estates and organizations that manage their likeliness and other related copyright and IP. IDK what the situation is in this case, but Nvidia may very well have paid for the name.
Earlier quoted context omitted.
Yeah it is, but unless you've memorised the history of Nvidia architectures it doesn't tell you which is the newer one Fermi -> Kepler -> Maxwell -> Pascal -> Volta (HPC only) -> Turing -> Ampere -> Hopper (HPC only?) -> Lovelace?
Isn't this the norm? Only AMD started the trend of naming the uArch with Numbers as Zen 4 or RDNA 3 fairly recently. With Intel it is Haswell > Broadwell > ..... Whatever Lake.
We keep using code names in discussions because the actual names are ass backwards and not very descriptive.
Earlier quoted context omitted.
Couple points: 1) NVIDIA will likely release a variant of H100 with 2x memory, so we may not even have to wait a generation. They did this for V100-16GB/32GB and A100-40GB/80GB. 2) In a generation or two, the SOTA model architecture will change, so it will be hard to predict the memory reqs... even today, for a fixed train+inference budget, it is much better to train Mixture-Of-Experts (MoE) models, and even NVIDIA a…
Oh I totally expect the size of models to grow along with whatever hardware can provide. I really do wonder how much more you could squeeze out of a full pod of gen2-H100's, obviously the model size would be ludicrous, but how far are we into the realm of dimishing returns. Your point about MoE architectures certainly sounds like the more _useful_ deployment, but the research seems to be pushing towards ludicrously l…
A pod of gen2-H100s might have 256 GPUs with 40 TB of total memory, and could easily run a 10T param model. So I think we are far from diminishing returns on the hardware side :) The model quality also continues to get better at scale.
Re. reading material, I would take a look at DeepSpeed’s blog posts (not affiliated btw). That team is super super good at hardware+software optimization for ML. See their post on MoE models here: https://www.microsoft.com/en-us/research/blog/deepspeed-adva...
Earlier quoted context omitted.
You can buy A100s in a server today, a number of integrators will happily sell it to you.
As someone who've tried for some weeks, it really seems like it's out-of-stock literally everywhere. The demand seems to be a lot higher than the supply at the moment, so much that I'm considering buying one myself instead of renting servers with it.
Off topic but I can't stand when corporations use actual people's names for their marketing who never gave them the permission to do so. For something like Shakespeare or Cicero I'm OK with it but Grace Hopper was alive in my lifetime, and even Tesla feels a little weird. What gives you the right to use that person's reputation to shill your product?
"Combined with the additional memory on H100 and the faster NVLink 4 I/O, and NVIDIA claims that a large cluster of GPUs can train a transformer up to 9x faster, which would bring down training times on today’s largest models down to a more reasonable period of time, and make even larger models more practical to tackle." Looking good.
We have needed wide use of NVlink or something like it for a long time now......heres to hoping mobo manufacturers actually widely implement it!
How does a DGX Pod w/ the new 3.2Tbps per machine NVLINK switch compare to Tesla Dojo?
I don’t know anything about the state of Dojo, but Tesla was very hand wavy about their software stack during their presentation. And running AI algorithms efficiently on a piece of hardware is one of those things that many HW vendors have a hard time getting right.