With RAM price inflation, I wonder if FPGAs can be used to offload inference processing without keeping weights in RAM? The available RAM would be for activations, KV Cache, context but not static weights. Weights could be streamed from disk. This approach is not for tokens/second but throughput at a lower cost. Possibly better answers/kHh? I've started researching this, but wonder if others have considered/tried this?
Ask HN: Who is using FPGA for ML inference?
1–10 of 15 posts
Re: Ask HN: Who is using FPGA for ML inference?
#2FPGAs are far more expensive than GPU+DRAM, even at today's inflated prices.
Re: Ask HN: Who is using FPGA for ML inference?
#3FPGAs are far more expensive than GPU+DRAM, even at today's inflated prices.
Tang nano FPGAs are around $10, the idea being to couple with a $25 ARM or RISC-V dev board and offload the math from a streaming store. Coupling this with an older 8GB VRAM board can theoretically run a model whose weights do not fit. The weights get repeatedly run through the MCU/FPGA and the VRAM is used for KV Cache and context. If you connected several via USB and streamed a MoE model expert per FPGA, you could achieve parallelism. Won't be fast, but, many requests could be processed in parallel, as each request shares the large static weights.
The reason for this is, model weights do not need to be randomly accessed. So why store them in expensive RAM.
Cerebras and Qrok seem to be using a very different approach than NVIDIA to get orders of magnitudes speed ups. I'm trying to explore other alternative approaches.
Re: Ask HN: Who is using FPGA for ML inference?
#4Train circuits, not weights; then use an FPGA
Re: Ask HN: Who is using FPGA for ML inference?
#5No one because FPGAs are 10-20x less dense than ASICs actually designed for purpose. The tooling is also complete shit. Signed someone that did (part of) their PhD on this topic.
Re: Ask HN: Who is using FPGA for ML inference?
#6FPGAs are far more expensive than GPU+DRAM, even at today's inflated prices.
Tang nano FPGAs are around $10, the idea being to couple with a $25 ARM or RISC-V dev board and offload the math from a streaming store. Coupling this with an older 8GB VRAM board can theoretically run a model whose weights do not fit. The weights get repeatedly run through the MCU/FPGA and the VRAM is used for KV Cache and context. If you connected several via USB and streamed a MoE model expert per FPGA, you could…
I predict that the FPGA adds no value in this scenario. Just process inference on the CPU.
Re: Ask HN: Who is using FPGA for ML inference?
#7Earlier quoted context omitted.
Tang nano FPGAs are around $10, the idea being to couple with a $25 ARM or RISC-V dev board and offload the math from a streaming store. Coupling this with an older 8GB VRAM board can theoretically run a model whose weights do not fit. The weights get repeatedly run through the MCU/FPGA and the VRAM is used for KV Cache and context. If you connected several via USB and streamed a MoE model expert per FPGA, you could…
I predict that the FPGA adds no value in this scenario. Just process inference on the CPU.
I am doing the initial work on many core dual CPUs. I already have systems with large system RAM and older GPUs with less VRAM, and large SAS HDs, fast SSDs running large open weight LLMs that do not really fit any GPU I own (they offload and/or run on CPU and system RAM, but slowly).
I have the MCUs and FPGAs (in a drawer) and I am retired, and this is my idea of fun.
I am trying to generalize an approach to use large MoE models (with possibly small quants) to run many agents in parallel without spending more on more or bigger GPUs.
I am also doing some edge ML (bird species recognition near the camera) using NPUs (in design phase, yet untested). I have an electronics lab, and I've emulated soft CPUs and built software that runs on FPGAs and in my emulators.
Instead of assuming my approach won't work or is too expensive, I choose to be optimistic. Also, failures are educational. I'm trying to gain more FPGA experience.
Re: Ask HN: Who is using FPGA for ML inference?
#8No one because FPGAs are 10-20x less dense than ASICs actually designed for purpose. The tooling is also complete shit. Signed someone that did (part of) their PhD on this topic.
But I can afford my already purchased drawer full of cheap FPGAs for experiments.
I guess I am too risk adverse to bet $10,000 on an ASIC run (having no experience doing that and no desire to go that route).
If I can demonstrate a proof-of-concept in a reproducible research way, others can then advance to the ASIC level.
This is just a hobbyist experiment looking for other hobbyists who can afford a cheap FPGA and have some free time and interest.
Re: Ask HN: Who is using FPGA for ML inference?
#9Train circuits, not weights; then use an FPGA
Can you suggest what circuits? examples?
Re: Ask HN: Who is using FPGA for ML inference?
#10You could build small ML model like talos-v2. But larger model/LLM requires much more engineering cost than you expected. Optimizing HDL has too many control knob/param to solve by RL.