Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Nvidia DGX GH200 Whitepaper

resources.nvidia.com

Post reply on HN
Print view View original

21–30 of 45 posts

Re: Nvidia DGX GH200 Whitepaper

#21

Post by pclmulqdq »

lwneal wrote: post #19
wmf wrote: post #14

Earlier quoted context omitted.

A wafer of H100s uses far more electricity than a wafer of [Apple] A16s though.

An H100 uses up to 350 Watts, while an A16 has a TDP of only 8 W. But, the A16 is a smaller chip (about 108mm vs. the H100's 814mm) so you can fit more of them on a wafer. Since a wafer is 300mm in diameter, its area is 70685 mm^2, which would yield 86 H100's or 654 A16's. [1][2] However, that discounts the waste on the edges of the circular wafer, as well as the chip yield, which will both likely be worse for the la…

8 watts for the A16's TDP cannot be correct. Your phone CPU has a higher TDP. I saw 250 on Nvidia's website as a maximum.

Edit: Oh, you are talking about the Apple A16. Those chips are completely different in function, so sure.

Re: Nvidia DGX GH200 Whitepaper

#22

Post by wmf »

pclmulqdq wrote: post #20
wmf wrote: post #14

Earlier quoted context omitted.

A wafer of H100s uses far more electricity than a wafer of [Apple] A16s though.

I'm pretty sure that's actually wrong. Some math: A16 is 200 sq mm of silicon while an H100 is about 800. That means you get about 100-120 A16's on a wafer, while you only get ~30 H100's (see https://www.silicon-edge.co.uk/j/index.php/resources/die-per... ). Let's assume yield is 100% to make things easier. The rated max power of the A16 is about 250W, while the H100 is quoted at 700W. Thus, a wafer of A16's is about…

That's my point; if silicon demand shits from mobile to AI data centers you can't expect energy consumption to be the same.

Re: Nvidia DGX GH200 Whitepaper

#23

Post by danbruc »

luc4sdreyer wrote: post #8
tikkun wrote: post #6

As context: 1x dgx gh200 has 256x gh200s which each have 1x h100 and 1x grace cpu

Adding up to "1 exaFLOPS" (sparse FP8). For reference, the fastest FP64 supercomputer is the AMD-based Frontier supercomputer, at 1.1 exaFLOPS.

Does sparse mean anything other than we can not actually do as many FP8 operations per second as we just claimed? To me it sounds like they can do X matrix operations per second on sparse matrices using Y FP8 operations per second, but instead of just saying what Y is they tell us how many FP8 operations would be required if the matrices were not sparse. Is this pure marketing bullshit or is there some logic to this? How sparse do those matrices have to be? Or am I misunderstanding this claim?

Re: Nvidia DGX GH200 Whitepaper

#24

Post by wmf »

kvetching wrote: post #18
m3kw9 wrote: post #4

So basically 2x faster than H100

Was this upgrade known or is this out of left field and people that stocked up on H100s going to feel a little regret

It's been on the roadmap for a few years although there were no performance numbers. I assume GH200 is more expensive so the price/performance advantage may not be overwhelming. Worst case you order GH200s and then scalp your H100s on the used market.

Re: Nvidia DGX GH200 Whitepaper

#25

Post by jauntywundrkind »

mmaunder wrote: post #3

The memory and bandwidth numbers are mind blowing. Going to be very hard to catch Nvidia. It’s as if competitors are going through the motions for participation prizes.

AMD has been shipping 128x lanes of PCIe 5.0 on chip. That's 0.5TBps. Getting up to 0.9TBps isn't that crazy, but having big enough fabric & switches to attach to is a huge feat.

I have hope though. CXL switching is going to give the whole industry a very fresh look at interconnect fabrics, as a simpler to manage faster more direct alternative to PCIe. Should be good.

Personally I worry it's flogging a dead horse, has too many constraints, but Ethernet could be rumbling into action again too maybe. The hyperscalers & others created a new LinuxFoundation group "Ultra Ethernet Scaling" to scale up much faster. Still, even at 1Tbps, that's a bunch of lanes (7x) of that ultra Ethernet you'd need to get to NVlink's 0.9TBps GPU interconnect. More radical breaks with Ethernet are needed than line speed bumps, things that can make switches easier to scale out big, if this realm of tech is to be good systems fabric. https://www.linuxfoundation.org/press/announcing-ultra-ether...

One interesting note on the DGX GH200 architecture that is super interesting to me is that it's inverted the connectivity relationship. Typically a system would have NIC & GPU hanging off the processor bus, and interconnect would go over that bus (maybe optimizing with p2p-dma to skip going through main memory, if it's fancy). But here? GPUs have a 0.9TBps connection to the NVswitch. If the CPU wants to talk to the cluster, it uses nvlink c2c to send the data to the gpu that then used it's nvlink connection to the NVswitch to send it out. Interesting reversal, interesting flourish, and gee it sure makes sense to me; the GPU is the thing!

Also, past 256 GPUs, there are BlueField 3 devices for Ethernet or infiniband connectivity on DGX nodes. Which is a good but also pretty boring/standard smartnic based scale out strategy.

Re: Nvidia DGX GH200 Whitepaper

#26

Post by tikkun »

jacquesm wrote: post #9

I wonder how much this thing will cost, best I've been able to find so far is a 'low 8 digits' estimate in Anandtech article but nothing more specific than that. https://www.anandtech.com/show/18877/nvidia-grace-hopper-has...

Some private cloud execs I talked with ballparked it at $15-25mm [1].

[1]: (I wrote this) https://gpus.llm-utils.org/nvidia-h100-gpus-supply-and-deman...

Re: Nvidia DGX GH200 Whitepaper

#27

Post by smodad »

jiggawatts wrote: post #13
smodad wrote: post #12

What's funny is that even though the DGX GH200 is some of the most powerful hardware available, there's such a voracious demand that it's not gonna be enough to quench it. In fact, this is one of those cases where I think the demand will always outpace supply. Exciting stuff ahead. I heard Elon say something interesting during the discussion/launch of xAI: "My prediction is that we will go from an extreme silicon sho…

He’s just plain wrong about the electricity usage going up because of AI compute. To a first approximation, the amount of silicon wafers going through fabs globally is constant. We won’t suddenly increase chip manufacturing a hundredfold! There aren’t enough fabs or “tools” like the ASML EUV machines for that. Electricity is used for lots of things, not just compute, and within compute the AI fraction is tiny. We’re…

I bet you're right. Even if you take into account that a data center is a monster consumer of energy, in the grand scheme of things it's not that big. Some back of the envelope math:

Global electrical production in 2022 was ~30,000 TWh.[1]

If we over-estimate that a hyperscale data-center will consume about 100 MW of power, per year that would be around 876 GWh.[2]

Let's overestimate again and say that 1,000 new data centers spring up in a year, every year they would consume 876 TWh.

Which, is 2.92% of total electricity production. Which given the fact that I overestimated the energy consumption by more than an order of magnitude, I would say the term "rounding error" is accurate.

I think the main limiting factor in the near term is going to be chip production capacity. The fabs take so long to spin up, it's going to be a while before we can even consider "electricity production" being a limiting factor.

[1] https://yearbook.enerdata.net/electricity/world-electricity-... [2] https://cc-techgroup.com/data-center-energy-consumption/

Re: Nvidia DGX GH200 Whitepaper

#28

Post by flakiness »

mmaunder wrote: post #17
tuetuopay wrote: post #2

Why is this called a whitepaper, as this is more of a documentation and architecture overview of the cluster? Wow a CLOS topology for networking, very innovative. Details on NVLink would be great. For example, the needs and problems solved by their custom cables seemingly required by NVLink would be worth a whitepaper. Don't get me wrong, this is still great the general public can get a glimpse into Grace Hopper. And…

> Why is this called a whitepaper, as this is more of a documentation and architecture overview of the cluster? That’s what a marketing white paper is and does. It’s not an academic paper.

To be fair NVIDIA used to publish more detailed "white paper" for their GPUs ex. [1] and CPU textbooks like H&P [2] draws a lot of details from these. This less detailed "whitepaper" still has a scent of these old tradition.

[1] https://www.nvidia.com/content/PDF/nvidia-ampere-ga-102-gpu-...

[2] https://www.amazon.com/Computer-Architecture-Quantitative-Jo...

Re: Nvidia DGX GH200 Whitepaper

#29

Post by pclmulqdq »

wmf wrote: post #22
pclmulqdq wrote: post #20

Earlier quoted context omitted.

I'm pretty sure that's actually wrong. Some math: A16 is 200 sq mm of silicon while an H100 is about 800. That means you get about 100-120 A16's on a wafer, while you only get ~30 H100's (see https://www.silicon-edge.co.uk/j/index.php/resources/die-per... ). Let's assume yield is 100% to make things easier. The rated max power of the A16 is about 250W, while the H100 is quoted at 700W. Thus, a wafer of A16's is about…

That's my point; if silicon demand shits from mobile to AI data centers you can't expect energy consumption to be the same.

Demand right now is not shifting from mobile to datacenter, demand is shifting from "normal" datacenter compute to AI datacenter compute.

I think if you had said "AMD Epyc" rather than a mobile chip, that would be a much more apt comparison. The AI chips are somewhat more power intensive per box, but fairly similar on power/area. It turns out that these silicon processes are fairly uniform in terms of the power/area that they can sustain for any kind of workload.

Mobile chips are designed for <10% utilization and "rush-to-idle" workloads, and they are not remotely comparable to datacenter silicon (of any kind).

Re: Nvidia DGX GH200 Whitepaper

#30

Post by swyx »

smodad wrote: post #12

What's funny is that even though the DGX GH200 is some of the most powerful hardware available, there's such a voracious demand that it's not gonna be enough to quench it. In fact, this is one of those cases where I think the demand will always outpace supply. Exciting stuff ahead. I heard Elon say something interesting during the discussion/launch of xAI: "My prediction is that we will go from an extreme silicon sho…

i mean he's not the only one. sama's other big bet is on nuclear fusion. https://blog.samaltman.com/helion
Post reply on HN