31–40 of 197 posts
> However, although tensors describe the relationship between arbitrary higher-dimensional arrays, in practice the TPU hardware that we will consider is designed to perform calculations associated with one and two-dimensional arrays. Or, more specifically, vector and matrix operations. I still don’t understand why the term “tensor” is used if it’s only vectors and matrices.
Well, in the transformer forward pass there are a bunch of 4-dimensional arrays being used.
Earlier quoted context omitted.
How is that an added benefit if the hardware doesn’t actually support n-dimensional arrays (other the n = 1 and 2)? And, strictly speaking, a vector can be considered a 1x n (or n x1) matrix, so Matrix Processing Unit would have been fine.
it’s an abstraction, just like 2d arrays
Earlier quoted context omitted.
It's branding (see: TensorFlow); also, pretty much anything (linear) you would do with an arbitrarily ranked tensor can be expressed in terms of vector ops and matmuls
"Fixed-Function Matrix Accelerator" just doesn't have the same buzzy ring to it.
FixMax™ or Maxxelerator™ would be good brands.
On the podcast interview now Groq CEO Jonathon Ross did[1] he talked about the creation of the original TPUs (which he built at Google). Apparently originally it was a FPGA he did in his 20% time because he sat near the team who was having inference speed issues. They got it working, then Jeff Dean did the math and the decided to do an ASIC. Now of course Google should spin off the TPU team as a separate company. It'…
Given the size of the market and its near-monopoly situation, I strongly think this has the potential to (almost immediately) surpass the Pixel hardware business. But the problem here is that TPU is a relatively scarce computing resource even inside Google and it's very likely that Google has a hard time to meet its internal demands...
Earlier quoted context omitted.
You'd hate particle physics then. "Spin" and "action" and so on are terrible names, but scientists live with them, because convention. Convention dominates most of what we do. I'm not sure there's a good way around this. Most conventions suck, but they were established back before there was a clear idea of what the best long-term convention should be.
At least in physics you can understand how the terms came about historically, where at some point they made sense. But “tensor” here, as note in sibling comments, seems to have been chosen primarily for marketing reasons.
chuckle CISC/RISC has gone from astute observation, to research program, to revolutionary technology, to marketing buzzwords....and finally to being just completely meaningless sounds.
I suppose it's the terminological circle of life.
Earlier quoted context omitted.
I was confused as hell for a long time when I first got into ML, until I figured out how to think about tensors in a visual way. You're right: fundamentally ML is about vector and matrix operations (1D and 2D). So then why are most ML programs 3D, 4D, and in a transformer sometimes up to 6D (?!) One reasonable guess is that the third dimension is time. Actually not. It turns out that time is pretty rare in ML, and it…
Thanks for the background! I still don’t think it’s appropriate to call a batch of matrices a tensor.