Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: What is an A.I. chip and how does it work?

news.ycombinator.com

Post reply on HN
Print view View original

51–60 of 95 posts

Re: Ask HN: What is an A.I. chip and how does it work?

#51

Post by dreamcompiler »

bhu1st wrote: post #40
nologic01 wrote: post #11

It may help your digging and search if you have in mind what those chips really try to do: Accelerate numerical linear algebra calculations. If you are familiar with linear algebra these specialized chips literally etch silicon so as to perform vector (and more general multi-array or tensor) computations faster than a general purpose CPU. They do that by loading and operating a whole set of numbers (a chunk of a vect…

> If you are familiar with linear algebra Could you please write what are some common day to day life applications of linear algebra in computing?

Nvidia exists because linear algebra is essential for 3D graphics. They got a second boost from crypto-currency, which isn't strictly about linear algebra but it can put those computation units to good use. Now Nvidia is riding high again because neural nets are all about linear algebra, and LLMs are big neural nets.

Re: Ask HN: What is an A.I. chip and how does it work?

#52

Post by nologic01 »

Q6T46nT668w6i3m wrote: post #39
amelius wrote: post #38

Earlier quoted context omitted.

> Accelerate numerical linear algebra calculations. Like compute eigenvalues/eigenvectors of large matrices, compute SVDs, solve large sparse systems of equations, etc?

Nothing that fancy. Usually matrix-matrix and matrix-scalar multiplication.

This is indeed the bread-and-butter, but there is use of all sorts of standard linear algebra algorithms. You can check various xla-related (accelerated linear algebra) folders in tensorflow or torch folders in pytorch to see the list of what is used [1],[2]

[1] https://github.com/tensorflow/tensorflow/tree/8d9b35f442045b...

[2] https://github.com/pytorch/pytorch/blob/6e3e3dd477e0fb9768ee...

Re: Ask HN: What is an A.I. chip and how does it work?

#53

Post by reesul »

grozzle wrote: post #34
phh wrote: post #14

There are very different architectures in the wild. Some are simply standard GPUs (maybe with additional support for bf16/float16) (Rockchip RK1808 has one like that). You give it a list of instructions pretty much like a CPU (except massively parallel), and it'll execute it. BTW when I say standard GPU, I'm not saying "kinda like GPU, but really literally GPU architecture. Linux mainline support for Amlogic A311D2's…

The RK3588 seems like a beast for performance per dollar, I have one running desktop Linux for emulators and games, but I haven't used the NPU yet, because I haven't taken the time to figure out how to get OpenCV to talk to it. Is Rockchip's stuff "good", as NPUs go? I'm thinking of buying another 3588 SoC for my robotics hobby - you seem like you'd know if that's a decent idea or not.

Yeah it’s performance vs cost is honestly nuts. 6 TOPS is a pretty solid NPU, but I don’t know what their software is like. Programming those accelerators is often difficult, especially if you’re a small time customer.

Curious if anyone can weigh in on their SW usability. A quick search for their user level tools showed examples/documentation in Chinese(?)

Re: Ask HN: What is an A.I. chip and how does it work?

#54

Post by thrtythreeforty »

strohwueste wrote: post #8
neximo64 wrote: post #5

Short story, CPU's can do calculations, they can do them one at a time. Think of something like 1+1, = 2. If you had 1 million equations like these, CPU's will generally do them one at a go, i.e the first one, then the second, etc. GPUs were optimised to draw, so were able to do dozens of these at a go. So these can be used for AI/ML in both gradient descent and inference (forward passes). Because you can do many at…

Do you have any sources on those informations? I find it really hard to find stuff for what you describe. Also do you know about the detail of producing those Asics? Are they CMOS or flash (in-memory-compute?)

All current AI accelerators (that aren't a research project) are ordinary CMOS. Google published some papers about TPUv3. You should read them if you want to know more about the architecture of these kinds of chips.

Re: Ask HN: What is an A.I. chip and how does it work?

#55

Post by zoogeny »

There is a YouTube channel TechTechPotato [1] that has a podcast on AI hardware called "The AI Hardware Show". Pretty small and it gives you a view on how niche this market is - but if you want the 10k foot view from young budding tech journalists then I think this fits the bill.

Some random examples of video titles from the last 6 months of the channel:

* A Deep Dive into IBM's New Machine Learning Chip

* Does my PC actually use Machine Learning?

* Intel's Next-Gen 2023 Max CPU and Max GPU

* A Deep Dive into Avant, the new chip from Lattice Semiconductor (White Paper Video)

* The AI Hardware Show 2023, Episode 1: TPU, A100, AIU, BR100, MI250X

I think the podcasters background is actually in HPC (High Performance Computing), i.e. super computers. But that overlaps just enough with AI hardware that he saw an opportunity to capitalize on the new AI hype.

1. https://www.youtube.com/c/TechTechPotato

Re: Ask HN: What is an A.I. chip and how does it work?

#56

Post by wyldfire »

mongol wrote: post #15

What would be an affordable/ cheap way to get hands on with this type of hardware? Right now I have zero knowledge.

Qualcomm has an SDK [1] where you can run software on a DSP/NSP simulator.

[1] https://developer.qualcomm.com/software/hexagon-dsp-sdk

Re: Ask HN: What is an A.I. chip and how does it work?

#57

Post by anthomtb »

JoeAltmaier wrote: post #35
benreesman wrote: post #21

Earlier quoted context omitted.

While a given generation of accelerator can only target model architectures that are comparatively proven out, and there’s a lag time, it’s measured in years not decades. I remember when NVIDIA didn’t have hardware for ReLU. The fact of the matter on Moore’s law is that we’ve got transistors, but not TDP to burn and have for years. These stupid big L3 caches are just: “fuck it, I’ve got die to burn”. This is an old s…

There have been some really strange instruction sets conceived. As a student at Stanford we had a time-share system that was home-grown (as I remember). It had opcodes to reverse bits in a bitstring! And odder things. Somebody needed that for some research project I guess. And then repurposed the damn thing for timeshare. It was pretty sad timeshare as I recall. The only machine(s?) available to a population of what?…

> It had opcodes to reverse bits in a bitstring

Bit reversal is used in Fast Fourier Transforms. Its not entirely surprising to me that you'd have specialized hardware for that operation.

Ref: https://en.wikipedia.org/wiki/Bit-reversal_permutation#Appli...

Re: Ask HN: What is an A.I. chip and how does it work?

#60

Post by grozzle »

reesul wrote: post #53
grozzle wrote: post #34

Earlier quoted context omitted.

The RK3588 seems like a beast for performance per dollar, I have one running desktop Linux for emulators and games, but I haven't used the NPU yet, because I haven't taken the time to figure out how to get OpenCV to talk to it. Is Rockchip's stuff "good", as NPUs go? I'm thinking of buying another 3588 SoC for my robotics hobby - you seem like you'd know if that's a decent idea or not.

Yeah it’s performance vs cost is honestly nuts. 6 TOPS is a pretty solid NPU, but I don’t know what their software is like. Programming those accelerators is often difficult, especially if you’re a small time customer. Curious if anyone can weigh in on their SW usability. A quick search for their user level tools showed examples/documentation in Chinese(?)

Yeah it's the first under $100 system I've tried, (and I'm a fan of the genre) that's truly a desktop replacement in terms of being a snappy responsive desktop even with lots of browser tabs, etc.

I do know they have their own special sauce to talk to the NPU. I was discouraged from making the effort myself because their special sauce to talk to the VPU has barely any ffmpeg support, it p much only uses gstreamer, and I'm neither a masochist nor French so that's a non-starter.

Post reply on HN