Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Google Cloud TPU Multislice Training

cloud.google.com

Post reply on HN
Print view View original

21–30 of 49 posts

Re: Google Cloud TPU Multislice Training

#21

Post by aschleck »

sashank_1509 wrote: post #19
rwitten wrote: post #13

Earlier quoted context omitted.

Hey! I'm an contributor on this (Rafi Witten), all opinions my own. You're asking the right question but I think the math is off by a bit. The equivalent number on the H100's is 989 TFLOP/s/chip so the equivalent job is ~10K H100's = (10 * 10^18) / (989 * 10^12). (Both chips also have 8-bit acceleration!) I believe this is the largest ML job both by exaflops and number of chips every demonstrated. Other companies own…

I think your math is also slightly off, in the Google article, it claims “that is capable of achieving 10 exa-FLOPs (16-bit).” , so you should be comparing with 16 bit operations from a H100. 989 is TF32 core, for 16 bit it is 1979, so I guess around 5000 H100’s in a single training job would be equivalent to the training job mentioned in this article. Either way I actually would not be surprised if OpenAI has launch…

1979 16 bit flops on an H100 is with sparsity. See footnote 2 on https://www.nvidia.com/en-us/data-center/h100/. You should be halving it for non-sparse flops.

Re: Google Cloud TPU Multislice Training

#22

Post by lern_too_spel »

leumassuehtam wrote: post #5

Did they use this to train Gemini? Which raises the question, where is Gemini?

Unlikely. One reason Google Cloud is so terrible is that nobody in Google actually uses Google Cloud. It used to be that every time I mentioned this, somebody would jump in and say, "Well actually, Google Domains runs on Google Cloud," and we'd discuss whether Google Domains was a business critical part of Google. https://support.google.com/domains/answer/13689670?hl=en

Re: Google Cloud TPU Multislice Training

#23

Post by smarterclayton »

rwitten wrote: post #15
jrk wrote: post #4

As far as I can tell, the article notably never defines what "slices" are or what "multi-slice" means.

Great questions! Slices are a set of TPU chips that share a fast, private inter-chip-interconnect. Unlike the current GPU generation in clouds, the TPUs on different machines can communicate through this private network. Multislice means that we're using a hierarchical network, where there is both inter-chip-interconnect and normal data-center netowrking. More details: https://cloud.google.com/tpu/docs/multislice-int…

Also, I should point out that a set of machines hosting TPUs is referred to as a "pod", which is not the same thing as a Kubernetes pod (also referenced in this doc).

The term "pod" originated in early data center design and occasionally crosses over from HPC to broad use - i.e. nVidia calls the set of DGX machines a "pod" https://blogs.nvidia.com/blog/2021/03/05/what-is-a-cluster-p....

Kubernetes chose "pod" to represent a set of co-scheduled containers, like a "pod of whales". Other systems like Mesos and Google's Borg https://storage.googleapis.com/pub-tools-public-publication-... use "task" to refer to a single container but didn't have a concept for heterogenous co-scheduled tasks at the time.

Somewhat ironically, it now means TPUs on GKE are confusing because we have TPUs hosts organized into "pods", and "pods" for the software using the TPUs.

A Kubernetes pod using a TPU lands on a host which is part of a slice of a TPU pod.

Re: Google Cloud TPU Multislice Training

#24

Post by smarterclayton »

jeffbee wrote: post #20
smarterclayton wrote: post #11

Earlier quoted context omitted.

Disclaimer: work associated with this team, didn't write or review the blog post Article stated that it was throughput scheduling the pods on the clusters (from unrelated benchmarks that's usually ~300 pods/sec throughput for kube scheduler today) and then doing XLA compilation at pod launch, rather than amortizing once for all jobs. Optimizing throughput of kube scheduler is a good general opportunity and something…

Thanks for the context. I remember recently reading a paper from I think Baidu where they claimed to have a container arrival rate in the millions per second, consequently it was practical to operate their whole site in the style of lambda/cloud functions. Actually now that I am searching for that it seems Baidu has a number of papers on workload orchestration at scale specifically for learning.

I will note that a trend I have observed with recent ML - as we increasingly use accelerators and models correspondingly grow in size, we are returning to a "one machine, one workload" paradigm for the biggest training and inference jobs. You might have 8k accelerators, but only 1000 machines, and if you have one container per host 300 schedules / second is fast.

While at the same time as you note we have functional models for container execution that are approaching millions of dispatches for highly partitionable work, especially in data engineering and ETL.

Re: Google Cloud TPU Multislice Training

#25

Post by DavidSJ »

Question for rwitten or anyone else involved in this project:

I see a per-device batch size of 6 for the 16B model. With 256x199 = 50944 TPUs and a sequence length of 2048, this works out to 104M tokens per batch. This is much larger than typical for training runs of dense LMs of this size, which are usually closer to ~4M tokens per batch.

Was your critical batch size really this large? In other words, did you really see a benefit as compared to a much smaller batch size (and probably many fewer TPUs)? Did you use some special learning rate schedule or optimizer to achieve this?

Re: Google Cloud TPU Multislice Training

#26

Post by amf12 »

lern_too_spel wrote: post #22
leumassuehtam wrote: post #5

Did they use this to train Gemini? Which raises the question, where is Gemini?

Unlikely. One reason Google Cloud is so terrible is that nobody in Google actually uses Google Cloud. It used to be that every time I mentioned this, somebody would jump in and say, "Well actually, Google Domains runs on Google Cloud," and we'd discuss whether Google Domains was a business critical part of Google. https://support.google.com/domains/answer/13689670?hl=en

> Unlikely. One reason Google Cloud is so terrible is that nobody in Google actually uses Google Cloud.

Well, actually, Google Cloud is just an abstraction on top of internal Google infra, so this isn't the right question. So, it depends on what you want to infer/compare.

Re: Google Cloud TPU Multislice Training

#27

Post by jeffbee »

smarterclayton wrote: post #23
rwitten wrote: post #15

Earlier quoted context omitted.

Great questions! Slices are a set of TPU chips that share a fast, private inter-chip-interconnect. Unlike the current GPU generation in clouds, the TPUs on different machines can communicate through this private network. Multislice means that we're using a hierarchical network, where there is both inter-chip-interconnect and normal data-center netowrking. More details: https://cloud.google.com/tpu/docs/multislice-int…

Also, I should point out that a set of machines hosting TPUs is referred to as a "pod", which is not the same thing as a Kubernetes pod (also referenced in this doc). The term "pod" originated in early data center design and occasionally crosses over from HPC to broad use - i.e. nVidia calls the set of DGX machines a "pod" https://blogs.nvidia.com/blog/2021/03/05/what-is-a-cluster-p... . Kubernetes chose "pod" to rep…

As your second link mentions in section 2.4, Borg has "allocs" which are basically pods.

Re: Google Cloud TPU Multislice Training

#28

Post by milesward »

lern_too_spel wrote: post #22
leumassuehtam wrote: post #5

Did they use this to train Gemini? Which raises the question, where is Gemini?

Unlikely. One reason Google Cloud is so terrible is that nobody in Google actually uses Google Cloud. It used to be that every time I mentioned this, somebody would jump in and say, "Well actually, Google Domains runs on Google Cloud," and we'd discuss whether Google Domains was a business critical part of Google. https://support.google.com/domains/answer/13689670?hl=en

Lots of stuff inside Alphabet runs on GCP. And yes, Google Domains pisses me off.

Re: Google Cloud TPU Multislice Training

#29

Post by p1esk »

DavidSJ wrote: post #25

Question for rwitten or anyone else involved in this project: I see a per-device batch size of 6 for the 16B model. With 256x199 = 50944 TPUs and a sequence length of 2048, this works out to 104M tokens per batch. This is much larger than typical for training runs of dense LMs of this size, which are usually closer to ~4M tokens per batch. Was your critical batch size really this large? In other words, did you really…

I’m confused - why do you multiply number of chips by sequence length? Shouldn’t the total batch size be 50k x6?

Re: Google Cloud TPU Multislice Training

#30

Post by smarterclayton »

jeffbee wrote: post #27
smarterclayton wrote: post #23

Earlier quoted context omitted.

Also, I should point out that a set of machines hosting TPUs is referred to as a "pod", which is not the same thing as a Kubernetes pod (also referenced in this doc). The term "pod" originated in early data center design and occasionally crosses over from HPC to broad use - i.e. nVidia calls the set of DGX machines a "pod" https://blogs.nvidia.com/blog/2021/03/05/what-is-a-cluster-p... . Kubernetes chose "pod" to rep…

As your second link mentions in section 2.4, Borg has "allocs" which are basically pods.

True. The pod’s monotonic and atomic lifecycle across containers is a significant difference, but you can broadly accomplish similar behaviors with an alloc for sharing resources.
Post reply on HN