Google Launches AI Supercomputer Powered by Nvidia H100 GPUs • Hacker News Board

Re: Google Launches AI Supercomputer Powered by Nvidia H100 GPUs

#91

Post by cubefox » Sat, May 13, 2023, 5:35 PM UTC

saiojd wrote: post #43

cubefox wrote: post #17

Earlier quoted context omitted.

TPUs do compete with GPUs for ML tasks, so yes, this is evidence that GPUs are winning. The only alternative I could imagine is that TPUs will "win" at supercomputers exclusivity aimed at inference (as opposed to training). Since TPUs excel at inference. The question is how much ML compute is used for inference as opposed to training. Not much, I guess, otherwise something like TPUs would be more popular.

Since you seem knowledgable on this topic, what is it that TPUs do differently than GPUs? Why are they better at inference?

Sorry, I actually don't know much about them.

Re: Google Launches AI Supercomputer Powered by Nvidia H100 GPUs

#92

Post by kcb » Sat, May 13, 2023, 5:36 PM UTC

ChuckNorris89 wrote: post #8

So this is why Nvidia isn't lowering the price on the GPUs despite them sitting on the shelves and not selling. They make enough money from customers in the data center and supercomputer businesses that gaming is just a small market.

Kind find the latest version of this but gaming is far from small market. People tend to seriously underestimate the size of the PC gaming market.

https://www.techspot.com/images2/news/bigimage/2021/08/2021-...

Re: Google Launches AI Supercomputer Powered by Nvidia H100 GPUs

#93

Post by qwertox » Sat, May 13, 2023, 5:37 PM UTC

AMD should be gifting their GPUs by the dozens to the most prolific Open Source contributors if they want a piece of the cake. Their lack of access to CUDA is really harming them badly.

Re: Google Launches AI Supercomputer Powered by Nvidia H100 GPUs

#94

Post by MuffinFlavored » Sat, May 13, 2023, 5:45 PM UTC

Aissen wrote: post #54

People want to rent the pricey NVIDIA DGX H100. So Google just put it in their DC, letting customers pay its full price it every ~3 months; plus they don't have to operate it, which is win (or is it ?).

they have to pay for the power and the staff to set them up/manage it

Re: Google Launches AI Supercomputer Powered by Nvidia H100 GPUs

#95

Post by wmf » Sat, May 13, 2023, 5:45 PM UTC

gitfan86 wrote: post #82

noogle wrote: post #77

Should we buy Nvidia stock then? The greatest technological advancement in recent years critically depends on the hardware from a single company with no competition. yet Nvidia stock is still below its 2021 peak. How so?

It is unknown how much pricing power NVDA has. Can they 3x the price of everything And still sell out?

It sounds like they already did that. A100 was very expensive and H100 is even more expensive.

Re: Google Launches AI Supercomputer Powered by Nvidia H100 GPUs

#96

Post by HybridCurve » Sat, May 13, 2023, 5:51 PM UTC

adam_arthur wrote: post #11

ChuckNorris89 wrote: post #8

So this is why Nvidia isn't lowering the price on the GPUs despite them sitting on the shelves and not selling. They make enough money from customers in the data center and supercomputer businesses that gaming is just a small market.

Gaming is a huge chunk of their revenue, around $2B in recent quarters, with datacenter around $3.5B. Despite the AI hype, Nvidia’s datacenter revenue was down QoQ and only up 10% YoY. It remains to be seen if the growth trajectory has changed meaningfully over the last quarter, because the stock is priced for massive earnings growth while their revenue and earnings have been actually shrinking. We’ll find out on the…

>Gaming is a huge chunk of their revenue, around $2B in recent quarters, with datacenter around $3.5B.

Is it? I remember hearing they didn't make much money from their consumer GPU products a few years back. This was one of the reasons why they tried to clamp down so aggressively on people using desktop GPUs for computing. They had made a number of driver changes which restricted the capabilities of anything but the tesla and quadro products. They were also restricting bulk purchases of their cards.

Re: Google Launches AI Supercomputer Powered by Nvidia H100 GPUs

#97

Post by blihp » Sat, May 13, 2023, 5:57 PM UTC

qwertox wrote: post #93

AMD should be gifting their GPUs by the dozens to the most prolific Open Source contributors if they want a piece of the cake. Their lack of access to CUDA is really harming them badly.

It's more than just that: for the money, their consumer GPUs don't compete in compute tasks (especially inference/training) and their Linux compute drivers are a pile of steaming garbage on consumer hardware. It's really interesting/depressing to watch as they've done a nice job of supplying good open source graphics drivers. They really seem to be lacking something at a leadership level in terms of understanding GPU compute outside of specific enterprise/scientific use cases.

Re: Google Launches AI Supercomputer Powered by Nvidia H100 GPUs

#98

Post by ksec » Sat, May 13, 2023, 6:04 PM UTC

Our_Benefactors wrote: post #39

ksec wrote: post #19

Going Slightly Off Topic. This is why Leading Edge Node will continue to be well funded. Consumer Electronics ( Mainly Smartphone ) Silicon usage has been the main push behind the development of Pure Play leading edge foundry in the past 10 years. Despite the predicted / expected drop of Smartphone sales, considering the potential shown by ChatGPT or Bard, GPU or Wafers dedicated for AI will continue to be in demand…

Can you elaborate on the supposed “wonky physics” that goes on when things get small? I’ve seen it thrown around that 3nm is “almost” the smallest size that can be made before different classes of physical errors are introduced due to the extremely small distance between gates.

Read [1] from 2020, I have replied there along with the economics issues I was referring to which AI demand will likely solve, or at least part of the solution.

[1] https://news.ycombinator.com/item?id=24618031

Re: Google Launches AI Supercomputer Powered by Nvidia H100 GPUs

#99

Post by zrm » Sat, May 13, 2023, 6:12 PM UTC

jocaal wrote: post #73

zrm wrote: post #68

Earlier quoted context omitted.

Interestingly the "4th generation Intel Xeon Scalable processors" themselves have up to 2.45 TBps in memory bandwidth, with the 8-socket configuration, or 2 TBps with 2-socket Xeon Max and HBM. If they'd make an 8-socket Xeon Max it would have 8 TBps. Considering that the Xeon Max 9462 is $8000 vs. the H100 going for north of $40,000, that could be interesting.

The throughput these gpu's have make the price pretty competitive, but I think AMD is working on a APU in their instinct lineup. That could be pretty competitive since Nvidia is overcharging for memory and you could just use sticks instead

A lot of this is workload-dependent. LLMs for example seem to be memory-bound, so a fast CPU with HBM or a large number of memory channels should do well.

Socket SP5 has 12 channels, which is 461 GBps per socket at DDR5-4800. Intel is getting 1 TBps from HBM, but then you're paying for HBM. $8000 for the cheapest Xeon Max vs. $3000 for the Epyc 9334 with the same number of cores or ~$1000 for the least expensive thing that will fit in the 12-channel socket. CPUs also have a cost advantage because then you don't need a CPU and a GPU.

Other things might be more compute bound. Then a fast GPU in a socket with a lot of memory channels worth of cheap sticks should be fun.

Re: Google Launches AI Supercomputer Powered by Nvidia H100 GPUs

#100

Post by electroly » Sat, May 13, 2023, 6:17 PM UTC

cloudking wrote: post #42

IanCal wrote: post #9

Kinda feels like the main thing google launches is waiting lists.

Yep, launching things slowly and testing them before releasing wide. Seems like a good practice when you're introducing a new technology to the world.

I think in this case they just know the demand is RED HOT and they don't have nearly the supply to go around. I don't think it's really the typical new product concerns on this one (product-market fit, are we covering use cases, are there technical problems, etc.). They know people want this and would rather have it right this second, problems and all, than wait for a slow rollout; Google just doesn't have the supply to go around.