Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Nvidia Hopper GPU Architecture and H100 Accelerator

anandtech.com

Post reply on HN
Print view View original

41–50 of 183 posts

Re: Nvidia Hopper GPU Architecture and H100 Accelerator

#41

Post by [deleted] »

Uehreka wrote: post #38
neogodless wrote: post #35

Earlier quoted context omitted.

You just blew my mind. That did not occur to me, but it is obvious in retrospect.

Sure, but the way Nvidia names generations is far from obvious. It seems to be “names of famous scientists, progressing in alphabetical order, we skip some letters if we can’t find a well known scientist with a matching last name and are excited about a scientist 2 letters from now, we wrap around to the beginning of the alphabet when we get to the end, and we just skipped from A to H, so expect another wraparound in…

[deleted]

Re: Nvidia Hopper GPU Architecture and H100 Accelerator

#42

Post by Uehreka »

algo_trader wrote: post #11

From the nvidia page, > 80 billion transistors > Hopper H100 .. generational leap > 9x at-scale training performance over A100 > 30x LLM inference throughput > Transformer Engine .. speed .. 6x without losing accuracy So another monster chip - same size of the Apple M1-max thingy .. I guess it comes down to pricing. The A100 is already ridiculously expensive at $10K. They can this one at $50K and it would sell out?

Most people who use one of these will be doing so through an EC2 VM (or equivalent). Given that cloud platforms can spread load, keep these GPUs churning close to 24/7 and more easily predict/amortize costs, they’ll probably buy the amount that they know they need, and Nvidia probably has some approximately correct idea of what that number is.

Re: Nvidia Hopper GPU Architecture and H100 Accelerator

#43

Post by ml_hardware »

komuher wrote: post #7

1000 TFLOPS so i can run my GPT3 in under 100 ms locally :D If 1000 TFLOPS is possible to do in inference time then im speechless

At inference time it will be possible to do 4000 TFLOPS using sparse FP8 :)

But keep in mind the model won't fit on a single H100 (80GB) because it's 175B params, and ~90GB even with sparse FP8 model weights, and then more needed for live activation memory. So you'll still want atleast 2+ H100s to run inference, and more realistically you would rent a 8xH100 cloud instance.

But yeah the latency will be insanely fast given how massive these models are!

Re: Nvidia Hopper GPU Architecture and H100 Accelerator

#44

Post by jsheard »

ksec wrote: post #39
jsheard wrote: post #32

Earlier quoted context omitted.

Yeah it is, but unless you've memorised the history of Nvidia architectures it doesn't tell you which is the newer one Fermi -> Kepler -> Maxwell -> Pascal -> Volta (HPC only) -> Turing -> Ampere -> Hopper (HPC only?) -> Lovelace?

Isn't this the norm? Only AMD started the trend of naming the uArch with Numbers as Zen 4 or RDNA 3 fairly recently. With Intel it is Haswell > Broadwell > ..... Whatever Lake.

Usually the architecture name isn't the only distinguishing feature of the product name, you don't need to remember Intel codenames because a Core 12700 is obviously newer than a Core 11700

Nvidia's accelerators are just called 100 every time so if you don't remember the order of the letters it's not obvious

They could have just named them P100, V200, A300 and H400 instead

Re: Nvidia Hopper GPU Architecture and H100 Accelerator

#45

Post by kcb »

savant_penguin wrote: post #33
kcb wrote: post #19

Earlier quoted context omitted.

That's a strange statement. The vast vast majority of these cards will be in systems running Linux.

I for one suffer deeply when I try to install the nvidia drivers on Linux. The website binaries _always_ break my system Only the ppas from graphics-drivers work properly My experience on windows is much more automatic and it never breaks anything. But I'd rather pay the price (installing on Linux) to avoid windows at all costs

Yea it's not ideal but really no option is. Built in to Linux would be a problem too given the rate of GPU driver development. Most Linux installs in the corporate world are stuck on the major version of the kernel and system packages they shipped with.

Re: Nvidia Hopper GPU Architecture and H100 Accelerator

#47

Post by macrolocal »

waynecochran wrote: post #2

This seems fast... TF32 ....... 1,000 TFLOPS (tensor core) FP64/FP32 ... 60 TFLOPS I am more interested in the 144-core Grace CPU Superchip. nVidia is getting into the CPU business...

50% sparsity and rated at 700W. The new DGX is 10kW!

Re: Nvidia Hopper GPU Architecture and H100 Accelerator

#48

Post by thfuran »

hwers wrote: post #34
nynx wrote: post #3

Sounds like we need some new training methods. If training could take place locally and asynchronously instead of globally through backpropagation, the amount of energy could probably be significantly reduced.

Trying to reduce energy consumption for ML like this is so silly.

Why?

Re: Nvidia Hopper GPU Architecture and H100 Accelerator

#49

Post by TOMDM »

ml_hardware wrote: post #43
komuher wrote: post #7

1000 TFLOPS so i can run my GPT3 in under 100 ms locally :D If 1000 TFLOPS is possible to do in inference time then im speechless

At inference time it will be possible to do 4000 TFLOPS using sparse FP8 :) But keep in mind the model won't fit on a single H100 (80GB) because it's 175B params, and ~90GB even with sparse FP8 model weights, and then more needed for live activation memory. So you'll still want atleast 2+ H100s to run inference, and more realistically you would rent a 8xH100 cloud instance. But yeah the latency will be insanely fast…

So, we're about a 25-50% memory increase off of being able to run GPT3 on a single machine?

Sounds doable in a generation or two.

Re: Nvidia Hopper GPU Architecture and H100 Accelerator

#50

Post by jxy »

throw0101a wrote: post #40
TheRealSteel wrote: post #25

Earlier quoted context omitted.

Don't they provide them for their consumer cards too, just that it's a closed source binary blob?

And not just Linux: FreeBSD. * https://www.nvidia.com/en-us/drivers/unix/freebsd-x64-archiv... * https://www.freshports.org/x11/nvidia-driver Heck, Solaris : * https://www.nvidia.com/en-us/drivers/unix/solaris-display-ar... * https://www.nvidia.com/en-us/drivers/unix/

CUDA and related software/libraries only work on Linux or Windows.
Post reply on HN