Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Google Launches AI Supercomputer Powered by Nvidia H100 GPUs

tomshardware.com

Post reply on HN
Print view View original

21–30 of 182 posts

Re: Google Launches AI Supercomputer Powered by Nvidia H100 GPUs

#24

Post by Aissen »

metadat wrote: post #10
jleahy wrote: post #4

Earlier quoted context omitted.

250

Interesting, so what is the compute power of the 1000-node A100 super cluster my team has been allocated at work? I was expecting Google to be much bigger than us.

Back of the envelope math is that H100 is twice as fast as A100 (task may vary). So your 1000-node A100 very, very fast.

Now, the GPU-to-GPU links (NVLink) might often give them a big advantage for some workloads, letting them exchange data without going through the CPU, and virtually address more memory if your want to manipulate very large models.

So it's hard to answer properly without knowing the topology of your cluster.

Also, note that this "supercomputer", is probably "just" a DGX H100 in Google's DC.

Re: Google Launches AI Supercomputer Powered by Nvidia H100 GPUs

#25

Post by binkHN »

Technical detail:

> Each A3 supercomputer is packed with 4th generation Intel Xeon Scalable processors backed by 2TB of DDR5-4800 memory. But the real "brains" of the operation come from the eight Nvidia H100 "Hopper" GPUs, which have access to 3.6 TBps of bisectional bandwidth by leveraging NVLink 4.0 and NVSwitch.

Re: Google Launches AI Supercomputer Powered by Nvidia H100 GPUs

#27

Post by hoschicz »

sebzim4500 wrote: post #18
cubefox wrote: post #17

Earlier quoted context omitted.

TPUs do compete with GPUs for ML tasks, so yes, this is evidence that GPUs are winning. The only alternative I could imagine is that TPUs will "win" at supercomputers exclusivity aimed at inference (as opposed to training). Since TPUs excel at inference. The question is how much ML compute is used for inference as opposed to training. Not much, I guess, otherwise something like TPUs would be more popular.

I've heard estimates that the amount of compute used to train GPT-4 is equivalent to 8 months of usage and most models are used much less than GPT-4 is, although I guess they are also easier to train.

of "usage"? I never bought that claim as it's not clear what usage they mean – on 100x8 months or 10000x8 months?

Re: Google Launches AI Supercomputer Powered by Nvidia H100 GPUs

#28

Post by vlovich123 »

panarky wrote: post #16
cubefox wrote: post #15

Earlier quoted context omitted.

Either GPUs are better for most AI tasks or TPUs. Both being overall approximately equally good is very unlikely.

> most AI tasks Different workloads require different infrastructure. Can your workload saturate the TPU without getting throttled by memory or network? Great! Use TPUs and reduce training cost. But if your TPUs are idle 70% of the time because the constraint is getting data to them ... "A3 represents the first production-level deployment of its GPU-to-GPU data interface, which allows for sharing data at 200 Gbps whi…

But you could do an equivalent TPUTPU interlink. Surely that can’t be the reason.

Re: Google Launches AI Supercomputer Powered by Nvidia H100 GPUs

#29

Post by ilaksh »

I think the most interesting AI hardware stuff is about memristors or some type of compute-in-memory.

https://arxiv.org/pdf/2303.07470.pdf

https://ieeexplore.ieee.org/abstract/document/9669041

Maybe there will be something like transformers but more suited to crossbar arrays of memristors. If that actually makes sense.

Post reply on HN