Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Nvidia Hopper GPU Architecture and H100 Accelerator

anandtech.com

Post reply on HN
Print view View original

91–100 of 183 posts

Re: Nvidia Hopper GPU Architecture and H100 Accelerator

#91

Post by cjbgkagh »

dragontamer wrote: post #74
cjbgkagh wrote: post #30

Earlier quoted context omitted.

The main tensor op is a matmul intrinsic which is useful for way more than just deep learning. Edit; many of these speeds are low precision which is less useful outside of deep learning, but the higher precision matmul ops in the tensor cores are still very fast and very useful for wide variety of tasks.

> but the higher precision matmul ops in the tensor cores are still very fast and very useful for wide variety of tasks. The FP64 matrix-multiplication is only 60 TFlops, no where near the advertized 1000 TFlops. TF32 matrix-multiplication is a poorly named 16-bit operation.

You are indeed correct, I was (kinda) fooled by the marketing and I think that TF32 is deceptively named. I think the tensor cores are being used in this architecture for FP64 and 60 TFlops is still pretty decent.

I'm on Turing architecture so I've never used TF32. I've only used FP32 and FP16 but FP32 isn't supported by these tensor cores.

Re: Nvidia Hopper GPU Architecture and H100 Accelerator

#94

Post by ml_hardware »

rafaelero wrote: post #26

"Combined with the additional memory on H100 and the faster NVLink 4 I/O, and NVIDIA claims that a large cluster of GPUs can train a transformer up to 9x faster, which would bring down training times on today’s largest models down to a more reasonable period of time, and make even larger models more practical to tackle." Looking good.

The 9x speedup is a bit inflated... it's measured at a reference point of ~8k GPUs, on a workload that the A100 cluster is particularly bad at.

When measured at smaller #s of GPUs which are more realistic, the speedup is somewhere between 3.5x - 6x. See the GTC Keynote video at 38:50: https://youtu.be/39ubNuxnrK8?t=2330

Based on hardware specs alone, I think that training transformers with FP8 on H100 systems vs. FP16 on A100 systems should only be 3-4x faster. Definitely looking forward to external benchmarks over the coming months...

Re: Nvidia Hopper GPU Architecture and H100 Accelerator

#95

Post by cjbgkagh »

37ef_ced3 wrote: post #84
cjbgkagh wrote: post #68

Earlier quoted context omitted.

Why not call it TF19 then.

Because it's 32-bits wide in memory. The effective mantissa is like FP16 but it's padded out to be the same size as FP32. In other words, there's 1 sign bit, 8 exponent bits, 10 mantissa bits that are USED, and 13 mantissa bits that are IGNORED. 1 + 8 + 10 + 13 = 32 The 13 ignored mantissa bits are part of the memory image: they pad the number out to 32-bit alignment.

But the user never sees that memory right? Doesn't it go in FP32 and come out FP32? I still think it's deceptive marketing.

Re: Nvidia Hopper GPU Architecture and H100 Accelerator

#96

Post by paxys »

spoonjim wrote: post #86
paxys wrote: post #57

Earlier quoted context omitted.

> What gives you the right to use that person's reputation to shill your product? Practically speaking you have the right to do anything unless someone complains about it. A lot of popular figures, even those long dead, have estates and organizations that manage their likeliness and other related copyright and IP. IDK what the situation is in this case, but Nvidia may very well have paid for the name.

I don’t think my kids have any more right to use my name than a corporation, unless I specifically grant them that right (like Walt Disney did by naming it the Walt Disney company). Another sickening one is the Ed Lee Club in SF, who endorses political candidates under the name of a much-loved dead SF mayor.

Your kids have the right to everything you own (including your name) by default unless you take steps to change that, say using a will or estate.

Re: Nvidia Hopper GPU Architecture and H100 Accelerator

#98

Post by cjbgkagh »

lostmsu wrote: post #72
cjbgkagh wrote: post #23

Earlier quoted context omitted.

I think the 1PFLOPS figure for TF32 is with sparsity, which should be called out in the name. Maybe ‘TFS32’? I mainly use dense FP16 so the 1PFLOPS for that looks pretty good.

Asked elsewhere, but why FP16 as opposed to BF16?

I'm using older Turing GPUs BF16 would require Ampere. The weights in my models tend to be normalized so the fraction would be more important than the exponent so I would probably still use FP16. I would need to test it though.

Re: Nvidia Hopper GPU Architecture and H100 Accelerator

#99

Post by TOMDM »

ml_hardware wrote: post #79
TOMDM wrote: post #49

Earlier quoted context omitted.

So, we're about a 25-50% memory increase off of being able to run GPT3 on a single machine? Sounds doable in a generation or two.

Couple points: 1) NVIDIA will likely release a variant of H100 with 2x memory, so we may not even have to wait a generation. They did this for V100-16GB/32GB and A100-40GB/80GB. 2) In a generation or two, the SOTA model architecture will change, so it will be hard to predict the memory reqs... even today, for a fixed train+inference budget, it is much better to train Mixture-Of-Experts (MoE) models, and even NVIDIA a…

Oh I totally expect the size of models to grow along with whatever hardware can provide.

I really do wonder how much more you could squeeze out of a full pod of gen2-H100's, obviously the model size would be ludicrous, but how far are we into the realm of dimishing returns.

Your point about MoE architectures certainly sounds like the more _useful_ deployment, but the research seems to be pushing towards ludicrously large models.

You seem to know a fair amount about the field, is there anything you'd suggest if I wanted to read more into the subject?

Re: Nvidia Hopper GPU Architecture and H100 Accelerator

#100

Post by baq »

HelloNurse wrote: post #80
jmole wrote: post #58

Earlier quoted context omitted.

Here's what a propane heater of similar output looks like: https://www.amazon.com/Dura-Heat-Propane-Forced-Heater/dp/B0...

Most of the propane heater is a fan in a tube, the flame is probably quite smaller than a CPU package.

I've got an 8kW wood stove and that thing gets rather hot to touch - as in, you will get a blister... 40kW is a small city car worth of power.
Post reply on HN