Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Nvidia Hopper GPU Architecture and H100 Accelerator

anandtech.com

Post reply on HN
Print view View original

71–80 of 183 posts

Re: Nvidia Hopper GPU Architecture and H100 Accelerator

#71

Post by moinnadeem »

nynx wrote: post #3

Sounds like we need some new training methods. If training could take place locally and asynchronously instead of globally through backpropagation, the amount of energy could probably be significantly reduced.

Disclosure: I work at MosaicML

Yeah, I strongly agree. While Nvidia is working on better hardware (and they're doing a great job at it!), we believe that better training methods should be a big source of efficiency. We've released a new PyTorch library for efficient training at http://github.com/mosaicml/composer.

Our combinations of methods can train CV models ~4x faster to the same accuracy on CV tasks, and ~2x faster to the same perplexity/GLUE score on NLP tasks!

Re: Nvidia Hopper GPU Architecture and H100 Accelerator

#72

Post by lostmsu »

cjbgkagh wrote: post #23
waynecochran wrote: post #2

This seems fast... TF32 ....... 1,000 TFLOPS (tensor core) FP64/FP32 ... 60 TFLOPS I am more interested in the 144-core Grace CPU Superchip. nVidia is getting into the CPU business...

I think the 1PFLOPS figure for TF32 is with sparsity, which should be called out in the name. Maybe ‘TFS32’? I mainly use dense FP16 so the 1PFLOPS for that looks pretty good.

Asked elsewhere, but why FP16 as opposed to BF16?

Re: Nvidia Hopper GPU Architecture and H100 Accelerator

#73

Post by p1esk »

cjbgkagh wrote: post #68
p1esk wrote: post #65

Earlier quoted context omitted.

TF32 = FP32 range + FP16 precision

Why not call it TF19 then.

Because your existing FP32 models should run fine when converted to TF32, so TF32 is equivalent to FP32 as far as DL practitioners are concerned.

Re: Nvidia Hopper GPU Architecture and H100 Accelerator

#74

Post by dragontamer »

cjbgkagh wrote: post #30
dragontamer wrote: post #9

Earlier quoted context omitted.

Tensor petaflops are useful in only very few circumstances. One of which is the highly lucrative deep learning community though.

The main tensor op is a matmul intrinsic which is useful for way more than just deep learning. Edit; many of these speeds are low precision which is less useful outside of deep learning, but the higher precision matmul ops in the tensor cores are still very fast and very useful for wide variety of tasks.

> but the higher precision matmul ops in the tensor cores are still very fast and very useful for wide variety of tasks.

The FP64 matrix-multiplication is only 60 TFlops, no where near the advertized 1000 TFlops. TF32 matrix-multiplication is a poorly named 16-bit operation.

Re: Nvidia Hopper GPU Architecture and H100 Accelerator

#75

Post by bee_rider »

jsheard wrote: post #44
ksec wrote: post #39

Earlier quoted context omitted.

Isn't this the norm? Only AMD started the trend of naming the uArch with Numbers as Zen 4 or RDNA 3 fairly recently. With Intel it is Haswell > Broadwell > ..... Whatever Lake.

Usually the architecture name isn't the only distinguishing feature of the product name, you don't need to remember Intel codenames because a Core 12700 is obviously newer than a Core 11700 Nvidia's accelerators are just called 100 every time so if you don't remember the order of the letters it's not obvious They could have just named them P100, V200, A300 and H400 instead

I don't really mind the incomprehensible letters -- looking up the generation is pretty easy, and these are data-center focused products... getting the name right is somebody's job and the easiest possible thing.

However, is the number superfluous at this point?

Re: Nvidia Hopper GPU Architecture and H100 Accelerator

#76

Post by cjbgkagh »

p1esk wrote: post #73
cjbgkagh wrote: post #68

Earlier quoted context omitted.

Why not call it TF19 then.

Because your existing FP32 models should run fine when converted to TF32, so TF32 is equivalent to FP32 as far as DL practitioners are concerned.

There is a lot of redundancy in DL that forgives all manner of sins; think it’s sneaky.

Re: Nvidia Hopper GPU Architecture and H100 Accelerator

#77

Post by lostmsu »

jxy wrote: post #50
throw0101a wrote: post #40

Earlier quoted context omitted.

And not just Linux: FreeBSD. * https://www.nvidia.com/en-us/drivers/unix/freebsd-x64-archiv... * https://www.freshports.org/x11/nvidia-driver Heck, Solaris : * https://www.nvidia.com/en-us/drivers/unix/solaris-display-ar... * https://www.nvidia.com/en-us/drivers/unix/

CUDA and related software/libraries only work on Linux or Windows.

Some are even Linux only like nccl (AFAIK required to fully use NVLink)

Re: Nvidia Hopper GPU Architecture and H100 Accelerator

#78

Post by ipsin »

minimaxir wrote: post #13

So the product naming for Nvidia's server-GPUs by compute power now goes: P100 -> V100 -> A100 -> H100 This is not confusing at all.

I mean, I'd already internalized P100 Schedule me on an H100 and I promise I won't mind the "confusing" naming.

Re: Nvidia Hopper GPU Architecture and H100 Accelerator

#79

Post by ml_hardware »

TOMDM wrote: post #49
ml_hardware wrote: post #43

Earlier quoted context omitted.

At inference time it will be possible to do 4000 TFLOPS using sparse FP8 :) But keep in mind the model won't fit on a single H100 (80GB) because it's 175B params, and ~90GB even with sparse FP8 model weights, and then more needed for live activation memory. So you'll still want atleast 2+ H100s to run inference, and more realistically you would rent a 8xH100 cloud instance. But yeah the latency will be insanely fast…

So, we're about a 25-50% memory increase off of being able to run GPT3 on a single machine? Sounds doable in a generation or two.

Couple points:

1) NVIDIA will likely release a variant of H100 with 2x memory, so we may not even have to wait a generation. They did this for V100-16GB/32GB and A100-40GB/80GB.

2) In a generation or two, the SOTA model architecture will change, so it will be hard to predict the memory reqs... even today, for a fixed train+inference budget, it is much better to train Mixture-Of-Experts (MoE) models, and even NVIDIA advertises MoE models on their H100 page.

MoEs are more efficient in compute, but occupy a lot more memory at runtime. To run an MoE with GPT3-like quality, you probably need to occupy a full 8xH100 box, or even several boxes. So your min-inference-hardware has gone up, but your efficiency will be much better (much higher queries/sec than GPT3 on the same system).

So it's complicated!

Re: Nvidia Hopper GPU Architecture and H100 Accelerator

#80

Post by HelloNurse »

jmole wrote: post #58
wmwmwm wrote: post #53

Earlier quoted context omitted.

I was recently researching how you'd host systems like this in a datacentre and was blown away to find out that you can cool 40kW in a single air cooled rack - this might be old news for many, but it was 2x or 3x what I expected! Glad I'm not paying the electricity bill :)

Here's what a propane heater of similar output looks like: https://www.amazon.com/Dura-Heat-Propane-Forced-Heater/dp/B0...

Most of the propane heater is a fan in a tube, the flame is probably quite smaller than a CPU package.
Post reply on HN