Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: What is an A.I. chip and how does it work?

news.ycombinator.com

Post reply on HN
Print view View original

11–20 of 95 posts

Re: Ask HN: What is an A.I. chip and how does it work?

#11

Post by nologic01 »

It may help your digging and search if you have in mind what those chips really try to do: Accelerate numerical linear algebra calculations.

If you are familiar with linear algebra these specialized chips literally etch silicon so as to perform vector (and more general multi-array or tensor) computations faster than a general purpose CPU. They do that by loading and operating a whole set of numbers (a chunk of a vector or a matrix) simultaneously (whereas the CPU would operate mostly serially - one at a time).

The advantage is (in a nutshell) that you can get a significant speedup. How much depends on the problem and how big a chunk you can process simultaneously but it can be a significant factor.

There are disadvantages that people ignore in the current AI hype:

* The speedup in a one-off gain, the death of Moore's law is equally dead for "AI chips" and CPU's

* It is extremely specialized and fine-tuned software you need to develop and run and it only applies to the above linear algebra problems.

* In the past such specialized numerical algebra hardware was the domain of HPC (high performance computing). Many a supercomputer vendor went bankrupt in the past because the cost versus market size was not there.

Re: Ask HN: What is an A.I. chip and how does it work?

#13

Post by cinntaile »

formerly_proven wrote: post #10
neom wrote: post #7

Earlier quoted context omitted.

The video is about analog chips for ML/NNs. His profile of this company was particularly interesting: https://mythic.ai/ OP asked about chips for AI.

Do companies pay him or does he do these ads for free?

He gets paid. It says so at the start of the video, but I guess that could depend on where you live.

Re: Ask HN: What is an A.I. chip and how does it work?

#14

Post by phh »

There are very different architectures in the wild. Some are simply standard GPUs (maybe with additional support for bf16/float16) (Rockchip RK1808 has one like that). You give it a list of instructions pretty much like a CPU (except massively parallel), and it'll execute it. BTW when I say standard GPU, I'm not saying "kinda like GPU, but really literally GPU architecture. Linux mainline support for Amlogic A311D2's NPU is 10 lines, and it's declaring a no-output vivante GPU.

Some are just hardware pipeline to compute 2d/3d convolutions + activation function (Rockchip RK3588 has one like that). You give it the memory address + dimensions of the input matrix, the memory address + dimensions of the weights, memory address + dimensions of the output, which activation function you want (there are only like 4 supported), then you tell it RUN, you wait for a bit, and you have the result at the output memory address.

(I took Rockchip example to show that even in one microcosm it can change a lot)

And then you can imagine any architecture in-between.

AFAIK they all work with some RAM as input and some RAM as output, but some may have its own RAM, some may share RAM with the system, some might have mixed usage (RK3588 has some SRAM, and when you ask it to compute the convolution, you can tell it either to write to SRAM or system RAM)

It's possible that there are some components that are border line between ISP (Image Signal Porcessing) and a NPU, where the input is the direct camera stream, but my guess is that they do some very small processing on the camera stream, then dump it to RAM, then do all the heavy work from RAM to RAM. I think that Pixel 4-5 had something like that.

Re: Ask HN: What is an A.I. chip and how does it work?

#16

Post by ajb117 »

mongol wrote: post #15

What would be an affordable/ cheap way to get hands on with this type of hardware? Right now I have zero knowledge.

I'm pretty sure you can't buy TPUs, but people usually buy GPUs instead. If you're building a personal rig, these days, you can get an Nvidia RTX 3090 for about $720 USD on ebay used, which is pretty cheap for 24GB VRAM. There's also the A6000 with 48GB VRAM but that'll cost about $5000 on Amazon. Of course, there's new cards that are faster with more VRAM like the 4090 and RTX 6000, but they're also more expensive.

Of course, this is all pretty expensive still. If your models are small enough you can get away with even older GPUs with less VRAM like a GTX 1080 Ti. And then of course there's services like Google Collab and vast.ai where you can rent a TPU or GPU in the cloud.

I'd check out Tim Dettmers' guide for buying GPUs: https://timdettmers.com/2023/01/30/which-gpu-for-deep-learni...

Re: Ask HN: What is an A.I. chip and how does it work?

#17

Post by jmrm »

ajb117 wrote: post #16
mongol wrote: post #15

What would be an affordable/ cheap way to get hands on with this type of hardware? Right now I have zero knowledge.

I'm pretty sure you can't buy TPUs, but people usually buy GPUs instead. If you're building a personal rig, these days, you can get an Nvidia RTX 3090 for about $720 USD on ebay used, which is pretty cheap for 24GB VRAM. There's also the A6000 with 48GB VRAM but that'll cost about $5000 on Amazon. Of course, there's new cards that are faster with more VRAM like the 4090 and RTX 6000, but they're also more expensive.…

AFAIK Google Coral is an inexpensive TPU you can buy right now: https://coral.ai/products/accelerator/

Re: Ask HN: What is an A.I. chip and how does it work?

#18

Post by [deleted] »

nologic01 wrote: post #11

It may help your digging and search if you have in mind what those chips really try to do: Accelerate numerical linear algebra calculations. If you are familiar with linear algebra these specialized chips literally etch silicon so as to perform vector (and more general multi-array or tensor) computations faster than a general purpose CPU. They do that by loading and operating a whole set of numbers (a chunk of a vect…

[deleted]

Re: Ask HN: What is an A.I. chip and how does it work?

#19

Post by worldsayshi »

jmrm wrote: post #17
ajb117 wrote: post #16

Earlier quoted context omitted.

I'm pretty sure you can't buy TPUs, but people usually buy GPUs instead. If you're building a personal rig, these days, you can get an Nvidia RTX 3090 for about $720 USD on ebay used, which is pretty cheap for 24GB VRAM. There's also the A6000 with 48GB VRAM but that'll cost about $5000 on Amazon. Of course, there's new cards that are faster with more VRAM like the 4090 and RTX 6000, but they're also more expensive.…

AFAIK Google Coral is an inexpensive TPU you can buy right now: https://coral.ai/products/accelerator/

Very cool! Although seems to have no memory to speak of so many use cases like LLM goes away because of that I guess?

Re: Ask HN: What is an A.I. chip and how does it work?

#20

Post by muxamilian »

worldsayshi wrote: post #19
jmrm wrote: post #17

Earlier quoted context omitted.

AFAIK Google Coral is an inexpensive TPU you can buy right now: https://coral.ai/products/accelerator/

Very cool! Although seems to have no memory to speak of so many use cases like LLM goes away because of that I guess?

It has 8 MB of memory. It also supports live streaming the neural network to the chip, although that is slower than when it is cached in the memory.
Post reply on HN