Earlier quoted context omitted.
The story I like to tell for the Newton is that it was launched before the technology was ready yet. Like the Sega Game Gear. Old video phones. All those tablets that launched before the iPad. They’re good ideas, but they shipped a few years too early, and the technology to make them work well at a good price point wasn’t available until later. Like, the Sega Game Gear had a cool active matrix LCD screen, but it took…
>and the batteries only lasted like four hours Still more than the OG Steam Deck today :)
Google's First Tensor Processing Unit: Architecture
131–140 of 197 posts
Re: Google's First Tensor Processing Unit: Architecture
#132Earlier quoted context omitted.
They don't sell any TPUs, do they? Besides the, now ancient, coral toy-TPUs.
Has there been any development? The last update is from 2021 [0], but it is not officially killed by google(.com) [0] https://coral.ai/news/updates-07-2021
Re: Google's First Tensor Processing Unit: Architecture
#133> However, although tensors describe the relationship between arbitrary higher-dimensional arrays, in practice the TPU hardware that we will consider is designed to perform calculations associated with one and two-dimensional arrays. Or, more specifically, vector and matrix operations. I still don’t understand why the term “tensor” is used if it’s only vectors and matrices.
It says: tensors describe the relationship between high-d arrays It does not say: tensors “only” describe the relationship between high-d arrays The term “tensor” is used because it covers all cases: scalars, vectors, matrices, and higher-dimensional arrays. Tensors are still a generalization of vectors and matrices. Note the context: In ML and computer science, they are considered a generalization. From a strict pur…
Before 1900, the use of the word tensor was consistent with its etymology, because it was used only for symmetric matrices, which correspond to affine transformations that stretch or compress a body in certain directions.
The square matrix that corresponds to a general affine transformation can be decomposed into the product of a tensor (a symmetric matrix which stretches) and a versor (a rotation matrix, which is antisymmetric and which rotates).
When Ricci-Curbastro and Levi-Civitta have published the first theory of what now are called tensors, they did not define any new word for the concept of a multidimensional array with certain rules of transformation when the coordinate system is changed, which is now called tensor.
When Einstein has published the Theory of General Relativity during WWI in which he used what is now called tensor theory, for an unknown reason and without any explanation for this choice he has begun to use the word "tensor" with the current meaning, in contrast with all previous physics publications.
Because Einstein has become extremely popular immediately after WWI, his usage of the word "tensor" has spread everywhere, including in mathematics (and including in the American translations of the works of Ricci and Levi-Civita, where the word tensor has been introduced everywhere, despite the fact that it did not exist in the original).
Nevertheless, for many years the word "tensor" could not be used for arbitrary multi-dimensional arrays, but only for those which observe the tensor transformation rules with respect to coordinate changes.
The use of the word "tensor" as a synonym for the word "array", like in ML/AI, is a recent phenomenon.
Previously, e.g. in all early computer literature, the word "array" (or "table" in COBOL literature) was used to cover all cases, from scalars, vectors and matrices to arrays with an arbitrary number of dimensions, so no new words are necessary.
Re: Google's First Tensor Processing Unit: Architecture
#134Earlier quoted context omitted.
The way I see, NVidia only has a few advantages ordered from most important to least: 1. Reserved fab space. 2. Highly integrated software. 3. Hardware architecture that exists today. 4. Customer relationships. but all of these aspects are weak in one way or another: For #1, fab space is tight, and NVidia can strangle its consumer GPU market if it means selling more AI chips at a higher price. This advantage is gone…
Nvidia's datacenter AI chips don't have raytracing or rasterization. Heck, for all we know the new blackwell chip is almost exclusively tensor cores. They gave no numbers for regular CUDA perf.
Re: Google's First Tensor Processing Unit: Architecture
#135On the podcast interview now Groq CEO Jonathon Ross did[1] he talked about the creation of the original TPUs (which he built at Google). Apparently originally it was a FPGA he did in his 20% time because he sat near the team who was having inference speed issues. They got it working, then Jeff Dean did the math and the decided to do an ASIC. Now of course Google should spin off the TPU team as a separate company. It'…
The way I see, NVidia only has a few advantages ordered from most important to least: 1. Reserved fab space. 2. Highly integrated software. 3. Hardware architecture that exists today. 4. Customer relationships. but all of these aspects are weak in one way or another: For #1, fab space is tight, and NVidia can strangle its consumer GPU market if it means selling more AI chips at a higher price. This advantage is gone…
On GPUs, ML "just works" (inference and training) and are always order of magnitude faster than whatever CPU you have. TPUs work very well for some model architectures (old ones that they were optimized and designed for) and on some novel others can be actually slower than a CPU (because of gathers and similar) - this was my experience working on ML stuff as an ML Researcher at Google till 2022, maybe it got better but I doubt. Older TPUs were ok only for inference of those specific models and useless for training. And anything new I tried (fundamental part of research...) - the compiler would sonetimes just break with an internal error, most of the time just produce terrible and slow code, and bugs filed against it would stay open for years.
GPU is so much more than a matrix multiplier - it's a fully general, programmable processor. With excellent compilers, but most importantly - low level access that you don't need to rely on proprietary compiler engineers (like TPU ones) and anyone can develop something like Flash Attention. And as a side note: while a Transformer might be mostly matrix multiplication, many other models are not.
Re: Google's First Tensor Processing Unit: Architecture
#136Earlier quoted context omitted.
Probably many orders of magnitude greater than those buying TPU's for personal use...
Technically correct, but only because TPUs aren't for sale. H100s cost like 30,000 USD, if you can even get one.
Re: Google's First Tensor Processing Unit: Architecture
#137Earlier quoted context omitted.
They have Vulcan, which is cross-compatible. And AMD has ROCm. pytorch is standard and pytorch has ROCm support. And the Google TPU v5 also has pytorch support. We do have a couple of H100's, but I'd love to replace them with AMD's
If AMD fixes or open sources their proprietary firmware blob[0]. Geohot streamed all weekend on Twitch, reverse engineering the AMD firmware. It was quite entertaining learning about how that low level hardware firmware works[1] and his rants about AMD of course. [0] https://www.phoronix.com/news/Tinybox-Radeon-Again-UMR [1] https://www.twitch.tv/georgehotz
Re: Google's First Tensor Processing Unit: Architecture
#138Earlier quoted context omitted.
The way I see, NVidia only has a few advantages ordered from most important to least: 1. Reserved fab space. 2. Highly integrated software. 3. Hardware architecture that exists today. 4. Customer relationships. but all of these aspects are weak in one way or another: For #1, fab space is tight, and NVidia can strangle its consumer GPU market if it means selling more AI chips at a higher price. This advantage is gone…
Seems you have not worked with ML workloads, but base your comment on "internet wisdom", or worse, business analysts (I am sorry if that's inaccurate). On GPUs, ML "just works" (inference and training) and are always order of magnitude faster than whatever CPU you have. TPUs work very well for some model architectures (old ones that they were optimized and designed for) and on some novel others can be actually slower…
Re: Google's First Tensor Processing Unit: Architecture
#139Are they simple enough that "older outdated less in demand" fabs can produce them?
I know Apple and Nvidea has a lock on a lot of fab space?
Re: Google's First Tensor Processing Unit: Architecture
#140Earlier quoted context omitted.
Where can I buy a TPU v5 to install in my server? If the answer is “cloud”: that’s why NVidia is wiping the floor.
How many people are out there buying H100s for their personal use?
And that's all fixed capital expenditure - there's no risk a code bug or typo by an inexperienced student will lead to a huge bill.
Also, if you're looking for an alternative to CUDA because you dislike vendor lock-in, switching to something only available in GCP would be an absurd choice.