Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: What is an A.I. chip and how does it work?

news.ycombinator.com

Post reply on HN
Print view View original

21–30 of 95 posts

Re: Ask HN: What is an A.I. chip and how does it work?

#21

Post by benreesman »

nologic01 wrote: post #11

It may help your digging and search if you have in mind what those chips really try to do: Accelerate numerical linear algebra calculations. If you are familiar with linear algebra these specialized chips literally etch silicon so as to perform vector (and more general multi-array or tensor) computations faster than a general purpose CPU. They do that by loading and operating a whole set of numbers (a chunk of a vect…

While a given generation of accelerator can only target model architectures that are comparatively proven out, and there’s a lag time, it’s measured in years not decades.

I remember when NVIDIA didn’t have hardware for ReLU.

The fact of the matter on Moore’s law is that we’ve got transistors, but not TDP to burn and have for years. These stupid big L3 caches are just: “fuck it, I’ve got die to burn”.

This is an old story, things migrate in and out of the “CPU”, but the current outlook is that we’ll be targeting specialized hardware more rather than less for the foreseeable future.

Re: Ask HN: What is an A.I. chip and how does it work?

#22

Post by juujian »

nologic01 wrote: post #11

It may help your digging and search if you have in mind what those chips really try to do: Accelerate numerical linear algebra calculations. If you are familiar with linear algebra these specialized chips literally etch silicon so as to perform vector (and more general multi-array or tensor) computations faster than a general purpose CPU. They do that by loading and operating a whole set of numbers (a chunk of a vect…

Good explanation. That also gives you an idea why GPUs are decent at acceleration computations for neural networks (think cuda) -- they are already optimized for doing many small computations in parallel rather with slower processors and have a lot of dedicated ram (VRAM).

Re: Ask HN: What is an A.I. chip and how does it work?

#23

Post by kmeisthax »

jmrm wrote: post #17
ajb117 wrote: post #16

Earlier quoted context omitted.

I'm pretty sure you can't buy TPUs, but people usually buy GPUs instead. If you're building a personal rig, these days, you can get an Nvidia RTX 3090 for about $720 USD on ebay used, which is pretty cheap for 24GB VRAM. There's also the A6000 with 48GB VRAM but that'll cost about $5000 on Amazon. Of course, there's new cards that are faster with more VRAM like the 4090 and RTX 6000, but they're also more expensive.…

AFAIK Google Coral is an inexpensive TPU you can buy right now: https://coral.ai/products/accelerator/

The problem is that this is an "inferencing" accelerator - i.e. it can only execute pretrained models. You cannot train a model on one of these, you need a training accelerator. And pretty much all of those are either NVidia GPUs or cloud-only offerings.

Re: Ask HN: What is an A.I. chip and how does it work?

#24

Post by VeninVidiaVicii »

nologic01 wrote: post #11

It may help your digging and search if you have in mind what those chips really try to do: Accelerate numerical linear algebra calculations. If you are familiar with linear algebra these specialized chips literally etch silicon so as to perform vector (and more general multi-array or tensor) computations faster than a general purpose CPU. They do that by loading and operating a whole set of numbers (a chunk of a vect…

Is this why my M1 MacBook Air can run the same R code at least 10x faster than my giant Linux tower?

Re: Ask HN: What is an A.I. chip and how does it work?

#25

Post by microtonal »

benreesman wrote: post #21
nologic01 wrote: post #11

It may help your digging and search if you have in mind what those chips really try to do: Accelerate numerical linear algebra calculations. If you are familiar with linear algebra these specialized chips literally etch silicon so as to perform vector (and more general multi-array or tensor) computations faster than a general purpose CPU. They do that by loading and operating a whole set of numbers (a chunk of a vect…

While a given generation of accelerator can only target model architectures that are comparatively proven out, and there’s a lag time, it’s measured in years not decades. I remember when NVIDIA didn’t have hardware for ReLU. The fact of the matter on Moore’s law is that we’ve got transistors, but not TDP to burn and have for years. These stupid big L3 caches are just: “fuck it, I’ve got die to burn”. This is an old s…

I remember when NVIDIA didn’t have hardware for ReLU.

Could you elaborate? ReLU is max(x,0) and CUDA had fmaxf since CUDA 1.0 (2007).

Re: Ask HN: What is an A.I. chip and how does it work?

#26

Post by microtonal »

VeninVidiaVicii wrote: post #24
nologic01 wrote: post #11

It may help your digging and search if you have in mind what those chips really try to do: Accelerate numerical linear algebra calculations. If you are familiar with linear algebra these specialized chips literally etch silicon so as to perform vector (and more general multi-array or tensor) computations faster than a general purpose CPU. They do that by loading and operating a whole set of numbers (a chunk of a vect…

Is this why my M1 MacBook Air can run the same R code at least 10x faster than my giant Linux tower?

Apple Silicon Macs have special matrix multiplication units (AMX) that can do matrix multiplication fast and with low energy requirements [1]. These AMX units can often beat matrix multiplication on AMD/Intel CPUs (especially those without a very large number of cores). Since a lot of linear algebra code uses matrix multiplication and using the AMX units is only a matter of linking against Accelerate (for its BLAS interface), a lot of software that uses BLAS is faster o Apple Silicon Macs.

That said, the GPUs in your M1 Mac are faster than the AMX units and any reasonably modern NVIDIA GPU will wipe the floor with the AMX units or Apple Silicon GPUs in raw compute. However, a lot of software does not use CUDA by default and for small problem sets AMX units or CPUs with just AVX can be faster because they don't incur the cost of data transfers from main memory to GPU memory and vice versa.

[1] Benchmarks:

https://github.com/danieldk/gemm-benchmark#example-results

https://explosion.ai/blog/metal-performance-shaders (scroll down a bit for AMX and MPS numbers)

Re: Ask HN: What is an A.I. chip and how does it work?

#27

Post by fulafel »

mongol wrote: post #15

What would be an affordable/ cheap way to get hands on with this type of hardware? Right now I have zero knowledge.

Apple and Google consumer hardware have specialized ML compute features.

https://www.tomsguide.com/news/google-pixel-7s-most-critical...

https://www.macobserver.com/tips/deep-dive/what-is-apple-neu...

Re: Ask HN: What is an A.I. chip and how does it work?

#28

Post by stephencanon »

microtonal wrote: post #26
VeninVidiaVicii wrote: post #24

Earlier quoted context omitted.

Is this why my M1 MacBook Air can run the same R code at least 10x faster than my giant Linux tower?

Apple Silicon Macs have special matrix multiplication units (AMX) that can do matrix multiplication fast and with low energy requirements [1]. These AMX units can often beat matrix multiplication on AMD/Intel CPUs (especially those without a very large number of cores). Since a lot of linear algebra code uses matrix multiplication and using the AMX units is only a matter of linking against Accelerate (for its BLAS in…

> That said, the GPUs in your M1 Mac are faster than the AMX units

Not for double, which is what R mostly uses IIRC.

Re: Ask HN: What is an A.I. chip and how does it work?

#29

Post by lexicality »

mongol wrote: post #15

What would be an affordable/ cheap way to get hands on with this type of hardware? Right now I have zero knowledge.

Depending on your definition of affordable, the windows dev kit 2023 makes a big deal out of their NPU but you'll have to deal with windows 11 to access it unfortunately

Re: Ask HN: What is an A.I. chip and how does it work?

#30

Post by microtonal »

stephencanon wrote: post #28
microtonal wrote: post #26

Earlier quoted context omitted.

Apple Silicon Macs have special matrix multiplication units (AMX) that can do matrix multiplication fast and with low energy requirements [1]. These AMX units can often beat matrix multiplication on AMD/Intel CPUs (especially those without a very large number of cores). Since a lot of linear algebra code uses matrix multiplication and using the AMX units is only a matter of linking against Accelerate (for its BLAS in…

> That said, the GPUs in your M1 Mac are faster than the AMX units Not for double, which is what R mostly uses IIRC.

Ah, thanks for the correction! I never use R, so I assumed that it uses/supports single-precision floating point.
Post reply on HN