Google Cloud TPU Multislice Training
cloud.google.com
Google Cloud TPU Multislice Training
1–10 of 49 posts
Re: Google Cloud TPU Multislice Training
#2Full title: "Google Cloud demonstrates the world’s largest distributed training job for large language models across 50000+ TPU v5e chips"
Summary from Bard: "This article is about training large language models (LLMs) on Google Cloud TPUs. It discusses the challenges of training LLMs at scale, and how Google Cloud TPU Multislice Training addresses these challenges. The article also details the results of a recent experiment in which Google trained a 128B parameter LLM on 50,944 TPU v5e chips. This experiment is the largest publicly disclosed LLM distributed training job to date."
Re: Google Cloud TPU Multislice Training
#3Can someone ELI5 this?
Re: Google Cloud TPU Multislice Training
#4As far as I can tell, the article notably never defines what "slices" are or what "multi-slice" means.
Re: Google Cloud TPU Multislice Training
#5Did they use this to train Gemini? Which raises the question, where is Gemini?
Re: Google Cloud TPU Multislice Training
#6Can someone ELI5 this?
Google has a megawatt-scale TPUv5 cluster that they can spare for stunts, and they are flexing it.
Re: Google Cloud TPU Multislice Training
#7Can someone ELI5 this?
tl;dr - google has big computer (a lot of TPUs)
they just showed they could indeed make 50k TPUs do some flops.
With no paper this is just a marketing press release - the only takeaway is that existing tech stacks can utilize it probably.
Re: Google Cloud TPU Multislice Training
#8Something that doesn't seem worth bragging about is that the startup time increases linearly with the cluster size. Wouldn't you want it to be constant? What's the issue there?
Re: Google Cloud TPU Multislice Training
#9Ok so they claim in the article, 50000 TPU’s is equivalent to 10 exaflop floating point computations. That is equivalent to ~2,512 NVIDIA H100’s, which is like really small. Just shows the difference between TPU’s and GPU’s I guess. Inflection, a new LLM company created a 20,000 H100 cluster, I’m positive OpenAI, Tesla, Meta etc have orchestrated a job on more than 2500 H100 GPU’s.
Re: Google Cloud TPU Multislice Training
#10Ok so they claim in the article, 50000 TPU’s is equivalent to 10 exaflop floating point computations. That is equivalent to ~2,512 NVIDIA H100’s, which is like really small. Just shows the difference between TPU’s and GPU’s I guess. Inflection, a new LLM company created a 20,000 H100 cluster, I’m positive OpenAI, Tesla, Meta etc have orchestrated a job on more than 2500 H100 GPU’s.
It's worth noting that just because an H100 has a higher flops number doesn't mean your program is actually hitting that number of flops. Modern TPUs are surprisingly competitive with Nvidia on a perf/$ metric, if you're doing cloud ML they are absolutely worth a look. We have been keeping costs down by racking our own GPUs but TPUs are so cost effective that we need to do some thinking about changing our approach.
I'm not certain but I think part of this is that XLA (for example) is a mountain of chip-specific optimizations between your code and the actual operations. So comparing your throughput between GPU and TPU is not just flops-to-flops.