41–48 of 48 posts
Earlier quoted context omitted.
What I'm really asking is: how much effective compute throughput are you able to get during Transformer training relative to the amount of theoretical raw compute available?
That's a great question. I don't have that analysis handy, but it would definitely be worth doing.
It's disappointing there is no attempt at normalizing results. It's not perf/$, or perf/W, or perf/chip, or anything that might be useful - it seems to just be perf/(the largest machine google/nvidia could afford to put together for a given benchmark). Seriously. Transformer: 1024 TPUs are twice as fast as 480 GPUs. Resnet50: 1536 GPUs are about as fast as 1024 TPUs. SSD: 1024 TPUs are twice as fast as 240 GPUs. Grea…
Results are disappointing. 2x speed up over general purpose GPUs doesn’t justify a whole new hardware architecture.
Now the real problem is, can you actually get a TPU pod in practice?
Earlier quoted context omitted.
Results are disappointing. 2x speed up over general purpose GPUs doesn’t justify a whole new hardware architecture.
For actual practitioners, TPUs are incredible. The cost/performance combo is unmatched. Now the real problem is, can you actually get a TPU pod in practice?
48 to 1 hour would have allowed us to run a few experiment per work day. That would be huge.
If you told me that the next generation of TPUs could run the current 48 hour GPU training set in 1 hour, that would be something.
But 2x faster won’t make a difference. Generally, startups for a new chip need to demonstrate 100x improvement over the general purpose approach to get funding. So this seems like another Google vanity project.
Earlier quoted context omitted.
For actual practitioners, TPUs are incredible. The cost/performance combo is unmatched. Now the real problem is, can you actually get a TPU pod in practice?
My personal experience with crnns and lstms was that cutting training latency from 48 hours to 24 hours didn’t make a difference in our progress. Some architectures took 4x longer to run. 48 to 1 hour would have allowed us to run a few experiment per work day. That would be huge. If you told me that the next generation of TPUs could run the current 48 hour GPU training set in 1 hour, that would be something. But 2x f…
If you really wanted to get from 48 hours to multiple experiments per work day, you could (probably) do that right now by scaling horizontally. However, the cost becomes a major issue when you do that on GPUs. This is where the TPU shines since you can scale out without cost crippling you.
I work on training on GPUs and the TPU is definitely an incredibly useful piece of hardware, not a vanity project.
Earlier quoted context omitted.
All you see is 8 server racks with colored network cables, switches and power supplies. The TPU ASICs themselves make up just a tiny part of this datacenter, you also have the printed circuit boards, cooling fins, 8x48 metal boxes, power and network cables, DC/DC or AC/DC converters at the bottom, fans or water tubes for cooling and airgaps. My startup is trying to develop wafer scale integration where you collapse 2…
> would be suprised how hard it is to get funding from them to develop it. That's probably because there has been no evidence that wafer scale integration can actually work.
Earlier quoted context omitted.
My personal experience with crnns and lstms was that cutting training latency from 48 hours to 24 hours didn’t make a difference in our progress. Some architectures took 4x longer to run. 48 to 1 hour would have allowed us to run a few experiment per work day. That would be huge. If you told me that the next generation of TPUs could run the current 48 hour GPU training set in 1 hour, that would be something. But 2x f…
I don't understand. Time-to-train is always a product of model and the amount of hardware you use. TPUs are fast and cheap allowing you to use more hardware for the same cost. The TPU design also makes it scale quite well. If you really wanted to get from 48 hours to multiple experiments per work day, you could (probably) do that right now by scaling horizontally. However, the cost becomes a major issue when you do t…
But, TPUs are not standard, can’t be used for any other usecase. So those savings might not actually be realizable when everything is taken into account.
Earlier quoted context omitted.
I don't understand. Time-to-train is always a product of model and the amount of hardware you use. TPUs are fast and cheap allowing you to use more hardware for the same cost. The TPU design also makes it scale quite well. If you really wanted to get from 48 hours to multiple experiments per work day, you could (probably) do that right now by scaling horizontally. However, the cost becomes a major issue when you do t…
According to the post, it’s only 2x cheaper. So for the same dollars spent, I would cut my time from 48 hours to 24. But, TPUs are not standard, can’t be used for any other usecase. So those savings might not actually be realizable when everything is taken into account.
TPUs do have the memory capacity advantage though (over 2080Ti).
Earlier quoted context omitted.
I don't understand. Time-to-train is always a product of model and the amount of hardware you use. TPUs are fast and cheap allowing you to use more hardware for the same cost. The TPU design also makes it scale quite well. If you really wanted to get from 48 hours to multiple experiments per work day, you could (probably) do that right now by scaling horizontally. However, the cost becomes a major issue when you do t…
According to the post, it’s only 2x cheaper. So for the same dollars spent, I would cut my time from 48 hours to 24. But, TPUs are not standard, can’t be used for any other usecase. So those savings might not actually be realizable when everything is taken into account.