Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Google Cloud TPU Multislice Training

cloud.google.com

Post reply on HN
Print view View original

11–20 of 49 posts

Re: Google Cloud TPU Multislice Training

#11

Post by smarterclayton »

jeffbee wrote: post #8

Something that doesn't seem worth bragging about is that the startup time increases linearly with the cluster size. Wouldn't you want it to be constant? What's the issue there?

Disclaimer: work associated with this team, didn't write or review the blog post

Article stated that it was throughput scheduling the pods on the clusters (from unrelated benchmarks that's usually ~300 pods/sec throughput for kube scheduler today) and then doing XLA compilation at pod launch, rather than amortizing once for all jobs.

Optimizing throughput of kube scheduler is a good general opportunity and something I believe we would like to see.

I believe AOT compilation just not a critical optimization for the test, we would recommend it when running large and long training jobs to AOT compile to keep pod start latency low for hardware failures and job restarts (from checkpoints).

> The start times we observed were impressive, but we believe we can improve these even further. We are working on areas such as optimizing scheduling in GKE to increase throughput and enabling ahead-of-time compilation in MaxText to avoid just-in-time compilations on the full cluster.

Re: Google Cloud TPU Multislice Training

#12

Post by latchkey »

sashank_1509 wrote: post #9

Ok so they claim in the article, 50000 TPU’s is equivalent to 10 exaflop floating point computations. That is equivalent to ~2,512 NVIDIA H100’s, which is like really small. Just shows the difference between TPU’s and GPU’s I guess. Inflection, a new LLM company created a 20,000 H100 cluster, I’m positive OpenAI, Tesla, Meta etc have orchestrated a job on more than 2500 H100 GPU’s.

It sounds like they partnered with CoreWeave to use their equipment and it is "only" 3500 gpus so far.

https://inflection.ai/inflection-ai-announces-1-3-billion-of...

https://inflection.ai/nvidia-coreweave-mlperf

Re: Google Cloud TPU Multislice Training

#13

Post by rwitten »

sashank_1509 wrote: post #9

Ok so they claim in the article, 50000 TPU’s is equivalent to 10 exaflop floating point computations. That is equivalent to ~2,512 NVIDIA H100’s, which is like really small. Just shows the difference between TPU’s and GPU’s I guess. Inflection, a new LLM company created a 20,000 H100 cluster, I’m positive OpenAI, Tesla, Meta etc have orchestrated a job on more than 2500 H100 GPU’s.

Hey! I'm an contributor on this (Rafi Witten), all opinions my own.

You're asking the right question but I think the math is off by a bit. The equivalent number on the H100's is 989 TFLOP/s/chip so the equivalent job is ~10K H100's = (10 * 10^18) / (989 * 10^12). (Both chips also have 8-bit acceleration!)

I believe this is the largest ML job both by exaflops and number of chips every demonstrated. Other companies own more chips or exaflops than we show in this job but getting all the hardware working at once on a single job is a different matter! :-)

Re: Google Cloud TPU Multislice Training

#14

Post by rwitten »

jeffbee wrote: post #8

Something that doesn't seem worth bragging about is that the startup time increases linearly with the cluster size. Wouldn't you want it to be constant? What's the issue there?

(Contributor on the blog post, all opinions my own)

Agreed with you and we definitely weren't trying to brag! This is fast compared to people's expectations in the space but slow compared to what we should be able to accomplish and will accomplish in the future.

Re: Google Cloud TPU Multislice Training

#15

Post by rwitten »

jrk wrote: post #4

As far as I can tell, the article notably never defines what "slices" are or what "multi-slice" means.

Great questions! Slices are a set of TPU chips that share a fast, private inter-chip-interconnect. Unlike the current GPU generation in clouds, the TPUs on different machines can communicate through this private network. Multislice means that we're using a hierarchical network, where there is both inter-chip-interconnect and normal data-center netowrking.

More details: https://cloud.google.com/tpu/docs/multislice-introduction

(P.S. - contributor on blog post, Google employee, all thoughts my own)

Re: Google Cloud TPU Multislice Training

#16

Post by marmaduke »

sashank_1509 wrote: post #9

Ok so they claim in the article, 50000 TPU’s is equivalent to 10 exaflop floating point computations. That is equivalent to ~2,512 NVIDIA H100’s, which is like really small. Just shows the difference between TPU’s and GPU’s I guess. Inflection, a new LLM company created a 20,000 H100 cluster, I’m positive OpenAI, Tesla, Meta etc have orchestrated a job on more than 2500 H100 GPU’s.

> That is equivalent

On what number or op for the h100?

Re: Google Cloud TPU Multislice Training

#17

Post by latchkey »

rwitten wrote: post #13
sashank_1509 wrote: post #9

Ok so they claim in the article, 50000 TPU’s is equivalent to 10 exaflop floating point computations. That is equivalent to ~2,512 NVIDIA H100’s, which is like really small. Just shows the difference between TPU’s and GPU’s I guess. Inflection, a new LLM company created a 20,000 H100 cluster, I’m positive OpenAI, Tesla, Meta etc have orchestrated a job on more than 2500 H100 GPU’s.

Hey! I'm an contributor on this (Rafi Witten), all opinions my own. You're asking the right question but I think the math is off by a bit. The equivalent number on the H100's is 989 TFLOP/s/chip so the equivalent job is ~10K H100's = (10 * 10^18) / (989 * 10^12). (Both chips also have 8-bit acceleration!) I believe this is the largest ML job both by exaflops and number of chips every demonstrated. Other companies own…

I'd love to hear more about the challenges of getting the hardware working.

Re: Google Cloud TPU Multislice Training

#18

Post by lern_too_spel »

sashank_1509 wrote: post #9

Ok so they claim in the article, 50000 TPU’s is equivalent to 10 exaflop floating point computations. That is equivalent to ~2,512 NVIDIA H100’s, which is like really small. Just shows the difference between TPU’s and GPU’s I guess. Inflection, a new LLM company created a 20,000 H100 cluster, I’m positive OpenAI, Tesla, Meta etc have orchestrated a job on more than 2500 H100 GPU’s.

This is a blog post from Google Cloud marketing. It's saying that you, too, could train an LLM on Google Cloud if you hand them enough money. You can't do that on Inflection's or Tesla's clusters. Similar marketing blog post from last year: https://cloud.google.com/blog/products/compute/calculating-1...

The PaLM paper linked in the blog post is about how to get something actually useful out of that compute.

Re: Google Cloud TPU Multislice Training

#19

Post by sashank_1509 »

rwitten wrote: post #13
sashank_1509 wrote: post #9

Ok so they claim in the article, 50000 TPU’s is equivalent to 10 exaflop floating point computations. That is equivalent to ~2,512 NVIDIA H100’s, which is like really small. Just shows the difference between TPU’s and GPU’s I guess. Inflection, a new LLM company created a 20,000 H100 cluster, I’m positive OpenAI, Tesla, Meta etc have orchestrated a job on more than 2500 H100 GPU’s.

Hey! I'm an contributor on this (Rafi Witten), all opinions my own. You're asking the right question but I think the math is off by a bit. The equivalent number on the H100's is 989 TFLOP/s/chip so the equivalent job is ~10K H100's = (10 * 10^18) / (989 * 10^12). (Both chips also have 8-bit acceleration!) I believe this is the largest ML job both by exaflops and number of chips every demonstrated. Other companies own…

I think your math is also slightly off, in the Google article, it claims “that is capable of achieving 10 exa-FLOPs (16-bit).” , so you should be comparing with 16 bit operations from a H100.

989 is TF32 core, for 16 bit it is 1979, so I guess around 5000 H100’s in a single training job would be equivalent to the training job mentioned in this article.

Either way I actually would not be surprised if OpenAI has launched a single job on more than 10k GPU’s, but I also am not very knowledgeable on practical scaling. Congrats on the feat!

Re: Google Cloud TPU Multislice Training

#20

Post by jeffbee »

smarterclayton wrote: post #11
jeffbee wrote: post #8

Something that doesn't seem worth bragging about is that the startup time increases linearly with the cluster size. Wouldn't you want it to be constant? What's the issue there?

Disclaimer: work associated with this team, didn't write or review the blog post Article stated that it was throughput scheduling the pods on the clusters (from unrelated benchmarks that's usually ~300 pods/sec throughput for kube scheduler today) and then doing XLA compilation at pod launch, rather than amortizing once for all jobs. Optimizing throughput of kube scheduler is a good general opportunity and something…

Thanks for the context. I remember recently reading a paper from I think Baidu where they claimed to have a container arrival rate in the millions per second, consequently it was practical to operate their whole site in the style of lambda/cloud functions.

Actually now that I am searching for that it seems Baidu has a number of papers on workload orchestration at scale specifically for learning.

Post reply on HN