Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: What is an A.I. chip and how does it work?

news.ycombinator.com

Post reply on HN
Print view View original

41–50 of 95 posts

Re: Ask HN: What is an A.I. chip and how does it work?

#41

Post by wongarsu »

bhu1st wrote: post #40
nologic01 wrote: post #11

It may help your digging and search if you have in mind what those chips really try to do: Accelerate numerical linear algebra calculations. If you are familiar with linear algebra these specialized chips literally etch silicon so as to perform vector (and more general multi-array or tensor) computations faster than a general purpose CPU. They do that by loading and operating a whole set of numbers (a chunk of a vect…

> If you are familiar with linear algebra Could you please write what are some common day to day life applications of linear algebra in computing?

The obvious fields are computer graphics (all kinds: 2d, 3d rasterized and 3d raytraced are all heavy on linear algebra, though 3d rasterized is the easiest to speed up with the lockstep SIMD architectures we call GPUs) and neural networks.

Computer graphics mostly because you can view the real world as 3d space and the screen as 2d space, and linear algebra gives you all the tools to manipulate something in 3d space and project it into 2d space. Neural networks because you can treat them as matrix multiplications.

Re: Ask HN: What is an A.I. chip and how does it work?

#43

Post by psychphysic »

Google's TPU which they sell via Coral is just a systolic array of multiply-accumulates arranged in a grid.

Here's a decent overview from the horse's mouth. https://cloud.google.com/blog/products/ai-machine-learning/a...

It's called a systolic array because the data moves through it in waves similar to what an engineer imagines the heart looks like :)

Re: Ask HN: What is an A.I. chip and how does it work?

#44

Post by dogma1138 »

binarymax wrote: post #9

I’d start with CUDA, because knowing what a chip does won’t click until you see how it can be programmed to do massive parallel computation and matmul. I read the first book in this list about 10 years ago, and though it’s pretty old the concepts are solid. https://developer.nvidia.com/cuda-books-archive

CUDA abstracts most of the parallelism, the magic of CUDA is it gave developers a C/C++ API or language if you will that doesn’t really requires them to think about that they can continue writing their problems as they did when programming for mostly single core single threaded CPUs back in the day and CUDA takes care of the rest.

Even “manual” CUDA optimizations deal more with concurrency and data residency than parallelism and even those are usually limited to following the compute guide for your specific hardware and feature set and the driver does the majority of the heavy lifting.

Re: Ask HN: What is an A.I. chip and how does it work?

#45

Post by unnouinceput »

HarHarVeryFunny wrote: post #36

Modern AI/ML is increasingly about neural nets (deep learning), whose performance is based on floating point math - mostly matrix multiplication and multiply-and-add operations. These neural nets are increasingly massive, e.g. GPT-3 has 175 billion parameters, meaning that each pass thru the net (each word generated) is going to involve in excess of 175B floating point multiplications! When you're multiplying two lar…

A few finer points:

1 - It's RTX 4070, not GTX 4070

2 - the 30 TFLOPS you mention are at the very top when overclocked, they go for 22 normally.

3 - Also those are single precision TFLOPS, as in 32 bit. What really matter nowadays is double precision. And in double precision a 4070 is 0.35 TFLOPS (or 350 GFLOPS). 2 orders of magnitude lower, still impressive though

Re: Ask HN: What is an A.I. chip and how does it work?

#46

Post by benreesman »

microtonal wrote: post #25
benreesman wrote: post #21

Earlier quoted context omitted.

While a given generation of accelerator can only target model architectures that are comparatively proven out, and there’s a lag time, it’s measured in years not decades. I remember when NVIDIA didn’t have hardware for ReLU. The fact of the matter on Moore’s law is that we’ve got transistors, but not TDP to burn and have for years. These stupid big L3 caches are just: “fuck it, I’ve got die to burn”. This is an old s…

I remember when NVIDIA didn’t have hardware for ReLU. Could you elaborate? ReLU is max(x,0) and CUDA had fmaxf since CUDA 1.0 (2007).

I could easily be wrong, but IIUC the software/hardware stack was, as of 2014 or so, fusing down to a specific set of circuits for broadcast activation functions under the broader banner of “Tensor Cores” (along with a bunch of devilish FMA hot-pathing under the sheets).

This is a fairly random press piece off Google but there are a ton of them. Hard to tell whether it’s the hardware, the blob, or the process node unless you work there.

https://developer.nvidia.com/blog/accelerating-relu-and-gelu...

Re: Ask HN: What is an A.I. chip and how does it work?

#47

Post by HarHarVeryFunny »

unnouinceput wrote: post #45
HarHarVeryFunny wrote: post #36

Modern AI/ML is increasingly about neural nets (deep learning), whose performance is based on floating point math - mostly matrix multiplication and multiply-and-add operations. These neural nets are increasingly massive, e.g. GPT-3 has 175 billion parameters, meaning that each pass thru the net (each word generated) is going to involve in excess of 175B floating point multiplications! When you're multiplying two lar…

A few finer points: 1 - It's RTX 4070, not GTX 4070 2 - the 30 TFLOPS you mention are at the very top when overclocked, they go for 22 normally. 3 - Also those are single precision TFLOPS, as in 32 bit. What really matter nowadays is double precision. And in double precision a 4070 is 0.35 TFLOPS (or 350 GFLOPS). 2 orders of magnitude lower, still impressive though

For neural nets it's actually the opposite - half-precision bfloat16 is enough. You need large range, but not much accuracy.

Yes, the exact numbers are going to vary, but just giving a data point to indicate the magnitude of the numbers. If you want to quibble there's CPU SIMD too.

Re: Ask HN: What is an A.I. chip and how does it work?

#48

Post by nottorp »

An "AI" chip is marketing. But as other posts say, "linear algebra coprocessor" doesn't roll of the tongue as well.

Incidentally there used to be a proper "AI" chip. The original perceptron was intended to be implemented in hardware. But general purpose chips evolved much faster.

https://en.wikipedia.org/wiki/Perceptron

Re: Ask HN: What is an A.I. chip and how does it work?

#49

Post by juujian »

bhu1st wrote: post #40
nologic01 wrote: post #11

It may help your digging and search if you have in mind what those chips really try to do: Accelerate numerical linear algebra calculations. If you are familiar with linear algebra these specialized chips literally etch silicon so as to perform vector (and more general multi-array or tensor) computations faster than a general purpose CPU. They do that by loading and operating a whole set of numbers (a chunk of a vect…

> If you are familiar with linear algebra Could you please write what are some common day to day life applications of linear algebra in computing?

I have really lost touch with common day to day life at this point, but Excel would be an excellent example. If you have a column with a million data points and do some basic calculation -- subtract or multiply another column -- whether or not the software you use can translate that into a vectorized operation under the hood using linear algebra can significantly speed up your operation.
Post reply on HN