How is it that Google invented the TPU and Google Research came up with the paper on LLM and NVDA and AI startup companies have captured ~100% of the value
For historical precedent see Xerox Parc.
Google's First Tensor Processing Unit: Architecture
51–60 of 197 posts
Re: Google's First Tensor Processing Unit: Architecture
#52Broadcom did the TPU
Re: Google's First Tensor Processing Unit: Architecture
#53Sigh...learning about TPUs a decade ago made me invest heavily in $GOOG for the coming AI revolution...got that one 100% wrong. +400% over 10 years isn't bad but I can't help but feel shortchanged seeing nvidia/etc
An out and out cherrypicker complaining about missing out on gains.. couldnt write a better script
Re: Google's First Tensor Processing Unit: Architecture
#54> However, although tensors describe the relationship between arbitrary higher-dimensional arrays, in practice the TPU hardware that we will consider is designed to perform calculations associated with one and two-dimensional arrays. Or, more specifically, vector and matrix operations. I still don’t understand why the term “tensor” is used if it’s only vectors and matrices.
Well, in the transformer forward pass there are a bunch of 4-dimensional arrays being used.
Came in to say this.
The Einsum notation makes it desirable to formulate your model/layer as multi-dimensional arrays connected by (loosely) named axes, without worrying too much about breaking it down to primitives yourself. Once you get used to it, the terseness is liberating.
Re: Google's First Tensor Processing Unit: Architecture
#55Earlier quoted context omitted.
You’re seeming to imply the number of instructions available is what distinguishes CISC, but it never has been.
Guys....what are the instructions? The on-chip memory they are talking about is essentially...a big register set. So we have load from main memory into registers, store from registers into main memory, multiply matrices--source and dest are stored in registers.... We have a 20 instruction, load-store cpu....how is this not RISC? At least RISC how we used the term in 1995?
I think the "multiply matrices" instruction is the one that makes it a cisc
Re: Google's First Tensor Processing Unit: Architecture
#56Re: Google's First Tensor Processing Unit: Architecture
#57> However, although tensors describe the relationship between arbitrary higher-dimensional arrays, in practice the TPU hardware that we will consider is designed to perform calculations associated with one and two-dimensional arrays. Or, more specifically, vector and matrix operations. I still don’t understand why the term “tensor” is used if it’s only vectors and matrices.
I was confused as hell for a long time when I first got into ML, until I figured out how to think about tensors in a visual way. You're right: fundamentally ML is about vector and matrix operations (1D and 2D). So then why are most ML programs 3D, 4D, and in a transformer sometimes up to 6D (?!) One reasonable guess is that the third dimension is time. Actually not. It turns out that time is pretty rare in ML, and it…
higher dimensional vectors or matrices are still not tensors.
Re: Google's First Tensor Processing Unit: Architecture
#58Earlier quoted context omitted.
At least in physics you can understand how the terms came about historically, where at some point they made sense. But “tensor” here, as note in sibling comments, seems to have been chosen primarily for marketing reasons.
It comes from the maths, where tensors are generalisations of matrices/vectors. They got cribbed, because the ML stuff directly used a bunch of the underlying maths. It’s a novel term, it sounds cool, not surprised it also then got promoted up into a marketing term.
> tensors are generalisations of matrices/vectors.
Is that what they are though? Because that really is not my understanding. Tensors are mappings which not all matrices and vectors are. Maybe the matrices in ML layers are all mappings, but a matrix in general is not, not is a vector always a mapping. So tensors aren’t generalizations of matrices and vectors.
Re: Google's First Tensor Processing Unit: Architecture
#59How is it that Google invented the TPU and Google Research came up with the paper on LLM and NVDA and AI startup companies have captured ~100% of the value
It's far too early to suggest Google will not capture value from AI. They have plenty of opportunity to integrate AI into their products.
Re: Google's First Tensor Processing Unit: Architecture
#60How is it that Google invented the TPU and Google Research came up with the paper on LLM and NVDA and AI startup companies have captured ~100% of the value
Because Google can’t focus on a product for more than 18 months if it isn’t generating several billion in PROFIT. They are punch drunk on advertising.