Interestingly, this might be well answered by the LLMs built on the technology you’re interested in.
Ask HN: What is an A.I. chip and how does it work?
61–70 of 95 posts
Re: Ask HN: What is an A.I. chip and how does it work?
#62There's a lot of information here about chips which are mostly built for training neural networks.
It's worth noting there are very widely deployed chips primarily built for inference (running the network) especially on mobile phones.
Depending on the device and manufacturer sometimes this is implemented as part of the CPU itself, but functionally it's the same idea.
The Apple Neural Engine is a good example of this. This is separate to the GPU which is also on the CPU.
Further information is here: https://machinelearning.apple.com/research/neural-engine-tra...
The Google Tensor CPU used in the pixel has a similar coprocessor called the EdgeTPU.
Re: Ask HN: What is an A.I. chip and how does it work?
#63I want to latch on this question a bit -- which company out there is primed to bring us a CUDA competitor. AMD has failed, so any wise words from the people in the industry?
Re: Ask HN: What is an A.I. chip and how does it work?
#64It may help your digging and search if you have in mind what those chips really try to do: Accelerate numerical linear algebra calculations. If you are familiar with linear algebra these specialized chips literally etch silicon so as to perform vector (and more general multi-array or tensor) computations faster than a general purpose CPU. They do that by loading and operating a whole set of numbers (a chunk of a vect…
>* The speedup in a one-off gain, the death of Moore's law is equally dead for "AI chips" and CPU's
This does not seem to be true in reality. The H100 is about 2.5x 'better' for AI than the A100 (obviously depends exactly what you are doing), and they released about 2 years apart. That is roughly in line with Moore's law.
Re: Ask HN: What is an A.I. chip and how does it work?
#65I've worked in this space for the past five years. The chips are essentially highly parallel processors. There's no unifying architecture. You have the graph-based / hpc-simulator chips like Cerebras, Graphcore, etc which are basically a stick-as-many-cores-as-possible situation with a high-speed networking fabric. You have the 'tensor' cores like Groq where the chip operates as a whole and is just well suited for tensor processing (parallelizable, high-speed memory, etc).
At the end of the day, it's matrix multiplication acceleration mostly, and then IO optimization. Literally most of the optimization has nothing to do with compute. We can compute faster than we can ingest.
Re: Ask HN: What is an A.I. chip and how does it work?
#66Earlier quoted context omitted.
A few finer points: 1 - It's RTX 4070, not GTX 4070 2 - the 30 TFLOPS you mention are at the very top when overclocked, they go for 22 normally. 3 - Also those are single precision TFLOPS, as in 32 bit. What really matter nowadays is double precision. And in double precision a 4070 is 0.35 TFLOPS (or 350 GFLOPS). 2 orders of magnitude lower, still impressive though
For neural nets it's actually the opposite - half-precision bfloat16 is enough. You need large range, but not much accuracy. Yes, the exact numbers are going to vary, but just giving a data point to indicate the magnitude of the numbers. If you want to quibble there's CPU SIMD too.
For gaming do matter those double precision. And we were talking about a certain GPU, which is used for gaming, not AI. Hence why the AI chips exists in the first place - dedicated hardware for dedicated tasks (or ASIC for short)
Re: Ask HN: What is an A.I. chip and how does it work?
#67On top of what others have said here about TPUs and their kin, you can make things really scream by taping out an ASIC for a specific frozen neural network (i.e. including the weights and parameters).
If you never have to change the network - for instance to do image segmentation or object recognition - then you can’t get any more efficient than a custom silicon design that bakes in the weights as transistors.
Re: Ask HN: What is an A.I. chip and how does it work?
#68It may help your digging and search if you have in mind what those chips really try to do: Accelerate numerical linear algebra calculations. If you are familiar with linear algebra these specialized chips literally etch silicon so as to perform vector (and more general multi-array or tensor) computations faster than a general purpose CPU. They do that by loading and operating a whole set of numbers (a chunk of a vect…
>* The speedup in a one-off gain, the death of Moore's law is equally dead for "AI chips" and CPU's This does not seem to be true in reality. The H100 is about 2.5x 'better' for AI than the A100 (obviously depends exactly what you are doing), and they released about 2 years apart. That is roughly in line with Moore's law.
The difference seems to be: more units for parallel calculation, but the speed of calculation in itself doesn't double anymore. In other words: Moore's law has stopped for raw speed and perhaps other areas, but is still alive in other areas. This has some weird consequences: Some models can't be processed in smaller chips (because swapping in parts is too slow to be useful), but after a threshold is crossed, suddenly the large models run efficiently.
Probably we will see usage of large AI models in smaller devices the next years because there's another way to optimize: use more efficient representations of the model weights. I think about posits, a different floating point system where even 6 bits are perhaps usable. When models can switch to 6 bit posits from f16 (half floats), hardware can load more than three times larger models. We will see whether hardware for this will be mass-produced.
Re: Ask HN: What is an A.I. chip and how does it work?
#69It may help your digging and search if you have in mind what those chips really try to do: Accelerate numerical linear algebra calculations. If you are familiar with linear algebra these specialized chips literally etch silicon so as to perform vector (and more general multi-array or tensor) computations faster than a general purpose CPU. They do that by loading and operating a whole set of numbers (a chunk of a vect…
> If you are familiar with linear algebra Could you please write what are some common day to day life applications of linear algebra in computing?
Linear Algebra is like a lot of math in CS; you don't necessarily see it initially, but once you get some familiarity you start seeing it everywhere.
Others have commented on computer graphics, but (as it turns out) the exact same algorithms apply for Collision Detection in 3D space. And since games already are manipulating graphics, you add on another set of Linear Algebra transforms that change the position/rotation/shear of those vertices. In a similar way, science (especially physics) use linear algebra to build simulations of all kinds of systems.
One surprising use is in Advertising (and other user preference aggregators). Turns out a preference acts like a magnitude of a one-dimensional vector. String N preference vectors together and you get an N-Dimension Vector that you can perform Linear Algebra operations upon. One common application is the Dot Product, which is a fancy way of taking two N-Dimension vectors and measuring how close those vectors point in the same direction in a [1, -1] range.
Yet another common place to find Linear Algebra is in computer science papers. Most of the time this is simply notation; a lot of common programming forms can be represented by MxN matrices. However some of those algorithms will use LA as a way of parsing and breaking down a problem. You will see this in compiler papers often, but its transferable to many other domains.
As a final and personal observation, I found that Linear Algebra helped me grasp Functional Programming. In both cases I am applying a transform to some input, and often stringing together many such transforms. Also in both cases, the transformations are sensitive to their order, and a bad ordering produces nonsense just like garbage data.
Re: Ask HN: What is an A.I. chip and how does it work?
#70Earlier quoted context omitted.
> If you are familiar with linear algebra Could you please write what are some common day to day life applications of linear algebra in computing?
Linear Algebra is like a lot of math in CS; you don't necessarily see it initially, but once you get some familiarity you start seeing it everywhere. Others have commented on computer graphics, but (as it turns out) the exact same algorithms apply for Collision Detection in 3D space. And since games already are manipulating graphics, you add on another set of Linear Algebra transforms that change the position/rotatio…
> One common application is the Dot Product , which is a fancy way of taking two N Dimension vectors and measuring how close those vectors point in the same direction in a [1, 1] range.
That's cosine similarity, or normalised dot product. The dot product can take any value when the vectors are not unit norm.