1 petaflop on a chip?? What is the catch?
DP Linpack flops is what counts in supercomputer ranking. Stuck at .44 Exoflops in 2021.
Nvidia Hopper GPU Architecture and H100 Accelerator
101–110 of 183 posts
Re: Nvidia Hopper GPU Architecture and H100 Accelerator
#102Anyone find details about the DPX instructions for dynamic programming?
Re: Nvidia Hopper GPU Architecture and H100 Accelerator
#103Earlier quoted context omitted.
So, we're about a 25-50% memory increase off of being able to run GPT3 on a single machine? Sounds doable in a generation or two.
Couple points: 1) NVIDIA will likely release a variant of H100 with 2x memory, so we may not even have to wait a generation. They did this for V100-16GB/32GB and A100-40GB/80GB. 2) In a generation or two, the SOTA model architecture will change, so it will be hard to predict the memory reqs... even today, for a fixed train+inference budget, it is much better to train Mixture-Of-Experts (MoE) models, and even NVIDIA a…
Is it difficult/desirable to squeeze/compress an open-sourced 200B parameter model to fit into 40GB?
Are these techniques for specific architectures or can they be made generic ?
Re: Nvidia Hopper GPU Architecture and H100 Accelerator
#104And may be taking this opportunity to ask, what happen to Nvidia's leak? The hacker hasn't made any more news, and Nvidia hasn't provide an update either.
The simplest explanation is that Nvidia just paid up.
Re: Nvidia Hopper GPU Architecture and H100 Accelerator
#105Earlier quoted context omitted.
Couple points: 1) NVIDIA will likely release a variant of H100 with 2x memory, so we may not even have to wait a generation. They did this for V100-16GB/32GB and A100-40GB/80GB. 2) In a generation or two, the SOTA model architecture will change, so it will be hard to predict the memory reqs... even today, for a fixed train+inference budget, it is much better to train Mixture-Of-Experts (MoE) models, and even NVIDIA a…
Is it difficult/desirable to squeeze/compress an open-sourced 200B parameter model to fit into 40GB? Are these techniques for specific architectures or can they be made generic ?
Re: Nvidia Hopper GPU Architecture and H100 Accelerator
#106Earlier quoted context omitted.
I don’t think my kids have any more right to use my name than a corporation, unless I specifically grant them that right (like Walt Disney did by naming it the Walt Disney company). Another sickening one is the Ed Lee Club in SF, who endorses political candidates under the name of a much-loved dead SF mayor.
Your kids have the right to everything you own ( including your name) by default unless you take steps to change that, say using a will or estate.
Yes, I know, I'm saying that it should not be that way. Rights to your likeness should end at your death unless you specifically write down otherwise.
Re: Nvidia Hopper GPU Architecture and H100 Accelerator
#107NVidia and AMD datacenter GPUs continue to diverge between focusing on deep learning and traditional scientific computing respectively.
Re: Nvidia Hopper GPU Architecture and H100 Accelerator
#108Earlier quoted context omitted.
> but the higher precision matmul ops in the tensor cores are still very fast and very useful for wide variety of tasks. The FP64 matrix-multiplication is only 60 TFlops, no where near the advertized 1000 TFlops. TF32 matrix-multiplication is a poorly named 16-bit operation.
You are indeed correct, I was (kinda) fooled by the marketing and I think that TF32 is deceptively named. I think the tensor cores are being used in this architecture for FP64 and 60 TFlops is still pretty decent. I'm on Turing architecture so I've never used TF32. I've only used FP32 and FP16 but FP32 isn't supported by these tensor cores.
Well the addition is done in FP32, and it's a 32-bit storage format in memory, so calling it a 16-bit format isn't right either. It's really a hybrid format where everything is 32-bit except multiplication.
Given that it's 32-bit in memory (so all your data structures are 32-bit) and also that in my experience using it is very transparent (I haven't run into any numerical issues compared to full FP32), I think calling it a 32-bit format is a reasonable compromise.
Re: Nvidia Hopper GPU Architecture and H100 Accelerator
#109Earlier quoted context omitted.
Because it's 32-bits wide in memory. The effective mantissa is like FP16 but it's padded out to be the same size as FP32. In other words, there's 1 sign bit, 8 exponent bits, 10 mantissa bits that are USED, and 13 mantissa bits that are IGNORED. 1 + 8 + 10 + 13 = 32 The 13 ignored mantissa bits are part of the memory image: they pad the number out to 32-bit alignment.
But the user never sees that memory right? Doesn't it go in FP32 and come out FP32? I still think it's deceptive marketing.
The user does see 32-bits and all bits are used because all the additions (and other operations besides the multiply in matrix ops) are in FP32. So the bottom bits are populated with useful information.
Re: Nvidia Hopper GPU Architecture and H100 Accelerator
#110Earlier quoted context omitted.
Isn't this the norm? Only AMD started the trend of naming the uArch with Numbers as Zen 4 or RDNA 3 fairly recently. With Intel it is Haswell > Broadwell > ..... Whatever Lake.
Usually the architecture name isn't the only distinguishing feature of the product name, you don't need to remember Intel codenames because a Core 12700 is obviously newer than a Core 11700 Nvidia's accelerators are just called 100 every time so if you don't remember the order of the letters it's not obvious They could have just named them P100, V200, A300 and H400 instead
We need a canonical, chronologically monotonic, marketing independent ID scheme. Marketing people always tries to disrupt naming schemes and that’s the real problem.