Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Nvidia Hopper GPU Architecture and H100 Accelerator

anandtech.com

Post reply on HN
Print view View original

141–150 of 183 posts

Re: Nvidia Hopper GPU Architecture and H100 Accelerator

#141

Post by kovek »

chockchocschoir wrote: post #62
p1esk wrote: post #37

Earlier quoted context omitted.

You can buy A100s in a server today, a number of integrators will happily sell it to you.

As someone who've tried for some weeks, it really seems like it's out-of-stock literally everywhere. The demand seems to be a lot higher than the supply at the moment, so much that I'm considering buying one myself instead of renting servers with it.

Does it make sense that all the GPUs are bought out? They each provide a return for mining in the short-term. In the long term, they can be used to run A(G)I models, which will be very very useful

Re: Nvidia Hopper GPU Architecture and H100 Accelerator

#142

Post by my123 »

dragontamer wrote: post #129
bcatanzaro wrote: post #113

Earlier quoted context omitted.

It's a 32-bit format in memory and the additions are done with 32-bits.

I admit that I don't have the hardware to test your claims. But pretty much all the whitepapers I can find on TF32 explicitly state the 10-bit mantissa, suggesting that this is at best, a 19-bit format. 1-bit sign + 8-bit exponent + 10-bit mantissa. Yes, the system will read/write the 32-bit value to RAM. But if there's only 10-bits of mantissa in the circuits, you're only going to get 10-bits of precision (best case…

Yup, in a semi-related field, NVIDIA has 3xTF32 for cases needing higher precision: https://github.com/NVIDIA/cutlass/discussions/361

Re: Nvidia Hopper GPU Architecture and H100 Accelerator

#143

Post by fennecfoxen »

kovek wrote: post #141
chockchocschoir wrote: post #62

Earlier quoted context omitted.

As someone who've tried for some weeks, it really seems like it's out-of-stock literally everywhere. The demand seems to be a lot higher than the supply at the moment, so much that I'm considering buying one myself instead of renting servers with it.

Does it make sense that all the GPUs are bought out? They each provide a return for mining in the short-term. In the long term, they can be used to run A(G)I models, which will be very very useful

This is the GPU the parent is talking about https://www.nvidia.com/en-us/data-center/a100/

Re: Nvidia Hopper GPU Architecture and H100 Accelerator

#144

Post by jwuphysics »

moinnadeem wrote: post #71
nynx wrote: post #3

Sounds like we need some new training methods. If training could take place locally and asynchronously instead of globally through backpropagation, the amount of energy could probably be significantly reduced.

Disclosure: I work at MosaicML Yeah, I strongly agree. While Nvidia is working on better hardware (and they're doing a great job at it!), we believe that better training methods should be a big source of efficiency. We've released a new PyTorch library for efficient training at http://github.com/mosaicml/composer . Our combinations of methods can train CV models ~4x faster to the same accuracy on CV tasks, and ~2x fa…

I've been seeing a lot more about MosaicML on my Twitter feed. Just wanted to ask -- how are your priorities different than, say, Fastai?

Re: Nvidia Hopper GPU Architecture and H100 Accelerator

#145

Post by kovek »

fennecfoxen wrote: post #143
kovek wrote: post #141

Earlier quoted context omitted.

Does it make sense that all the GPUs are bought out? They each provide a return for mining in the short-term. In the long term, they can be used to run A(G)I models, which will be very very useful

This is the GPU the parent is talking about https://www.nvidia.com/en-us/data-center/a100/

This still makes sense! TPUs are useful for AI, which itself will be very very useful. It’s almost like it’s the best investment. That’s why smart players buy them all. Maybe I’m going out-of-topic.

Re: Nvidia Hopper GPU Architecture and H100 Accelerator

#146

Post by gautamcgoel »

This chip is capable of 2000 INT8 Tensor TOPS, or 1000 F16 Tensor TFLOPS. In other words, it is capable of performing over a quadrillion operations per second. Absolutely insane... I still have fond memories of installing my first NVidia gaming GPU, with just 512MB of RAM, probably capable of much less than a single teraflop of compute.

Re: Nvidia Hopper GPU Architecture and H100 Accelerator

#147

Post by asciimike »

chockchocschoir wrote: post #90
p1esk wrote: post #81

Earlier quoted context omitted.

Did you check Lambda or Exxact?

Yes, nor Lambda Labs or Exxact Corporation have them available last time I checked (last week). Both citing high demand as the reason for it being unavailable.

Howdy, I run [Crusoe Cloud](https://crusoecloud.com/) and we just launched an alpha of an A100 and A40 Cloud offering--we've got capacity at a reasonable price!

If you're interested in giving us a shot, feel free to shoot me an email at mike at crusoecloud dot com.

Re: Nvidia Hopper GPU Architecture and H100 Accelerator

#148

Post by touisteur »

dragontamer wrote: post #74
cjbgkagh wrote: post #30

Earlier quoted context omitted.

The main tensor op is a matmul intrinsic which is useful for way more than just deep learning. Edit; many of these speeds are low precision which is less useful outside of deep learning, but the higher precision matmul ops in the tensor cores are still very fast and very useful for wide variety of tasks.

> but the higher precision matmul ops in the tensor cores are still very fast and very useful for wide variety of tasks. The FP64 matrix-multiplication is only 60 TFlops, no where near the advertized 1000 TFlops. TF32 matrix-multiplication is a poorly named 16-bit operation.

There's a paper on getting fp32 accuracy using tf32 tensor cores and losing 3x efficiency. Can't wait to try it with cutlass... once I get how to use cutlass, woof.

Re: Nvidia Hopper GPU Architecture and H100 Accelerator

#149

Post by touisteur »

hughrr wrote: post #24
savant_penguin wrote: post #8

1 petaflop on a chip?? What is the catch?

700 watts so being NVidia it'll blow up in 6 months and you'll need to wait in a queue for 6 months to RMA it because all the miners had bought up the entire supply chain.

Those datacenter/hpc GPUs don't seem to get bought so much by the mining community? I don't have problems sourcing some through the usual channels (HPE, dell,...?). But you need somehow deep pockets.

Re: Nvidia Hopper GPU Architecture and H100 Accelerator

#150

Post by fancyfredbot »

The Tensor cores will be great for machine learning and the FP32/FP64 fantastic for HPC, but I'd be surprised if there were a lot of applications using both of these features at once. I wonder if there's room for a competitor to come in and sell another huge accelerator but with only one of these two features either at a lower price or with more performance? Perhaps the power density would be too high if everything was in use at once?
Post reply on HN