Cloud TPU VMs are generally available
cloud.google.com
Cloud TPU VMs are generally available
1–10 of 53 posts
Re: Cloud TPU VMs are generally available
#2TPU VMs are an incredible, incredible product. If you were put off by the previous TPU Node architecture, you should definitely come back and give this a shot. The dev/debug loop is just so much nicer now that the TPU is local to your VM.
That said, the one thing I'm missing on these is orchestration. GKE supports TPU Nodes but it does not support TPU VMs. It seems to me like this is a must-have feature so I'm sure it's on the roadmap.
If anyone from the team is here, do you have updates on this? And if anyone reading this has orchestration techniques they use and recommend for TPU VMs, please let me know what they are!
Re: Cloud TPU VMs are generally available
#3Does this solve the data-loading issues raised by sillysaurus in this prior thread?
Re: Cloud TPU VMs are generally available
#4First thing you should explain is what the hell a TPU is/
Re: Cloud TPU VMs are generally available
#5TPU VMs are an incredible, incredible product. If you were put off by the previous TPU Node architecture, you should definitely come back and give this a shot. The dev/debug loop is just so much nicer now that the TPU is local to your VM. That said, the one thing I'm missing on these is orchestration. GKE supports TPU Nodes but it does not support TPU VMs. It seems to me like this is a must-have feature so I'm sure i…
Thanks for the very kind feedback! We've wanted to provide TPU VMs since the beginning of the Cloud TPU program, and I'm delighted that you're enjoying them. Many people across Google contributed to this launch.
We're definitely working on GKE integration, and we intend to provide more support for orchestration over time since individual Cloud TPU workloads are getting bigger and bigger. We'd love to hear any more detailed feedback you might have.
Re: Cloud TPU VMs are generally available
#6Does this solve the data-loading issues raised by sillysaurus in this prior thread? https://news.ycombinator.com/item?id=24721229
Yes, TPU VMs dramatically improve the Cloud TPU user experience. You now have direct access to the VM on each TPU host whether you are using JAX, PyTorch, or TensorFlow, which provides a lot more flexibility and control and can often improve performance.
Re: Cloud TPU VMs are generally available
#7First thing you should explain is what the hell a TPU is/
Here's a quick overview of TPUs from last year's Google I/O keynote: https://www.youtube.com/watch?v=XFFrahd05OM&t=1565s
Re: Cloud TPU VMs are generally available
#8Does this solve the data-loading issues raised by sillysaurus in this prior thread? https://news.ycombinator.com/item?id=24721229
Yes, TPU VMs dramatically improve the Cloud TPU user experience. You now have direct access to the VM on each TPU host whether you are using JAX, PyTorch, or TensorFlow, which provides a lot more flexibility and control and can often improve performance.
I struggle to understand precisely what you mean by user experience and ‘often improved performance‘.
Previously, there was no actual support for crucial features of the TPU related to data loading when using PyTorch, say. In turn, using a TPU over a GPU on that setup was frequently not worth it due to that exact issue. Your answer suggests it might be different now: are TF, Jax and PyTorch now on par in all stages?
Re: Cloud TPU VMs are generally available
#9Founder of the Cloud TPU program here. If you'd like to experiment with TPU VMs for free and are willing to share your work with the world somehow (e.g. via publications or open-source projects), you can apply to participate in the TPU Research Cloud (TRC) program here:
https://sites.research.google/trc/
Re: Cloud TPU VMs are generally available
#10Earlier quoted context omitted.
Yes, TPU VMs dramatically improve the Cloud TPU user experience. You now have direct access to the VM on each TPU host whether you are using JAX, PyTorch, or TensorFlow, which provides a lot more flexibility and control and can often improve performance.
I struggle to understand precisely what you mean by user experience and ‘often improved performance‘. Previously, there was no actual support for crucial features of the TPU related to data loading when using PyTorch, say. In turn, using a TPU over a GPU on that setup was frequently not worth it due to that exact issue. Your answer suggests it might be different now: are TF, Jax and PyTorch now on par in all stages?
In the previous Cloud TPU architecture, PyTorch and JAX users had to create a separate CPU VM for every remote TPU host and arrange for these CPU hosts to communicate indirectly with the TPU hosts via gRPC. This was cumbersome and made debugging difficult.
With TPU VMs, none of this is necessary. You can SSH directly into each TPU host machine and install arbitrary software on a VM there to handle data loading and other tasks with much greater flexibility.
The blog post provides an example of training cost improvement using PyTorch / XLA on TPU VMs in the "Local execution of input pipeline" section. Hopefully we will be able to provide more tutorials on using PyTorch / XLA with TPU VMs soon.
With TPU VMs, workloads that require lots of CPU-TPU communication can now do that communication locally instead of going over the network, which can improve performance.