Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: What is an A.I. chip and how does it work?

news.ycombinator.com

Post reply on HN
Print view View original

91–95 of 95 posts

Re: Ask HN: What is an A.I. chip and how does it work?

#91

Post by HarHarVeryFunny »

unnouinceput wrote: post #90
HarHarVeryFunny wrote: post #83

Earlier quoted context omitted.

The NVIDIA cards are all dual-use for gaming and compute/ML. Some features like the RTX 4070's Tensor Cores (incl. bfloat16) are there primarily for ML, and other features like ray tracing are there for gaming.

The NVIDIA cards are for mining crypto-coins too, and they successfully did that for years, before being made obsolete in that area by ASICs. Now it's time for the same thing in AI/ML too, hence why AI chips are being developed, they are the ASICs for this domain. That's the big picture. In 2 to 3 years none is going to use NVIDIA gaming cards for AI/ML anymore, no matter how many GFLOPS future 5000/6000 series are g…

ASICs aren't magic - they are just chips designed to do a single function fast (e.g run a crypto mining algorithm) as an alternative to using a general purpose CPU/GPU whose generality comes at the cost of some performance overhead.

If your application calls for generality - like a gaming card's need to run custom shaders, or an ML model's need to run custom compute kernels, then an ASIC won't help you. These applications still need a general purpose processor, just one that provides huge parallelism.

It seems you may be thinking that all an ML chip does is matrix multiplication, and so a specialized ASIC would make sense, but that's not the case - an ML chip needs to run the entire model - think of it as a PyTorch accelerator, not a matmul accelerator.

Finally, the market for consumer (vs data center) ML cards is tiny relative to the gaming market, and these chips/cards are expensive to develop. Unless this changes, it doesn't make sense for companies like NVIDA to develop ML-only cards when with minimal effort they can leverage their data center designs and build dual-use GPU/compute consumer cards.

Re: Ask HN: What is an A.I. chip and how does it work?

#92

Post by phh »

grozzle wrote: post #34
phh wrote: post #14

There are very different architectures in the wild. Some are simply standard GPUs (maybe with additional support for bf16/float16) (Rockchip RK1808 has one like that). You give it a list of instructions pretty much like a CPU (except massively parallel), and it'll execute it. BTW when I say standard GPU, I'm not saying "kinda like GPU, but really literally GPU architecture. Linux mainline support for Amlogic A311D2's…

The RK3588 seems like a beast for performance per dollar, I have one running desktop Linux for emulators and games, but I haven't used the NPU yet, because I haven't taken the time to figure out how to get OpenCV to talk to it. Is Rockchip's stuff "good", as NPUs go? I'm thinking of buying another 3588 SoC for my robotics hobby - you seem like you'd know if that's a decent idea or not.

Well, I already said it :P RK3588's NPU can only do convolutions and a small list of activation functions. Realistically it's usable only on image (2D, 3D data). The software side is pretty weird: It's amazing the amount of models they support despite the limited fixed-hardware function, but there has been literally 0 development in the last 6 months and it does have a lot of bugs that doesn't seem hard to fix. Even when it comes to images, it doesn't support changing the resolution of the input (you need to ""recompile the model"" for that), which is super weird since hardware pipeline doesn't care much about the size.

Anyway, I really don't recommend it, unless you're making your own model, you know before-hand what's supported and what isn't, and your input is fixed resolution (which is a pretty fair usage in an embedded system) (fixed = doesn't change at every frame. handling hotplug from one webcam with a resolution to another with another resolution is fine)

I think looking at the examples give you a reasonable show of what it can do: https://github.com/rockchip-linux/rknn-toolkit/tree/master/e... It's mobilenet, yolov3, resnet50. There aren't more examples because they didn't had more examples. There aren't more examples because that's pretty much all you can reasonably run.

As far as I can tell, modern image models using transformer/vit won't be runnable on it. (it acts enough as a coprocessor that it's possible to do some parts in CPU some parts in NPU - and Rockchip framework handles that -, so maybe it's somehow possible)

(Note: I say this as a huge Rockchip lover, their mainline support is top-notch, they make very durable product (their 2015's RK3288 is still far from obsolete), and I bought a RK3588 SBC to play with a NPU accelerator (whose full specification is publicly available btw), in the hope to have a self-hosted LLM voice assistant)

Re: Ask HN: What is an A.I. chip and how does it work?

#94

Post by rramadass »

gojo_dog wrote: post #88

There's a five part series outlining AI accelerator chips that came out last year. Starting with Introduction, Motivation, Technical Foundations, and Existing Solutions: https://medium.com/@adi.fu7/ai-accelerators-part-i-intro-822...

Excellent; just what i was looking for!

Please do submit this as a top-level HN submission so that it gets greater visibility.

Re: Ask HN: What is an A.I. chip and how does it work?

#95

Post by anon291 »

rramadass wrote: post #80
anon291 wrote: post #65

I've worked in this space for the past five years. The chips are essentially highly parallel processors. There's no unifying architecture. You have the graph-based / hpc-simulator chips like Cerebras, Graphcore, etc which are basically a stick-as-many-cores-as-possible situation with a high-speed networking fabric. You have the 'tensor' cores like Groq where the chip operates as a whole and is just well suited for te…

>The chips are essentially highly parallel processors Right. AFAIK we already were doing SIMD, Vector Processing, VLIW etc. to speed up parallel processing for numerical calculations in AI/ML. What then is the reason for the explosion of these different categories of chips ? Are they just glorified ASICs designed for their specific domains or FPGAs programmed accordingly? If so what is their architecture i.e. what ar…

Realistically, a good AI chip will have provisions for high throughput IO (the most important thing, and the key differentiator) and the actual processing really doesn't matter because with enough engineering effort you'll be able to saturate the chip.

GPUs have a high speed output in the form of an HDMI link. However, there is no high speed input. Reads/writes to/from the GPU are slow. The Cerebras wafer chip for example has 8-16 FPGA driven IO chips that directly read from TCP/IP onto the chip and off again in parallel. Each FPGA connects to its own ethernet port. So you can get the data on/off the chip as fast as possible. That's it really.

As for the processing engines. They're usually just standard cores with a high speed interconnect and maybe some matrix multiplication optimizations. Some, like Groq, have a high speed fabric with specialized processors at various locations.

Post reply on HN