11–15 of 15 posts
My understanding (as a non-FPGA expert) is that currently FPGA beats generic hardware (CPU,GPU) for "small size algorithm" (i.e that do not need GB of weights), while enabling a certain flexibility vs ASIC.
My guess is that you cannot bake all the weights into the circuit topology, so you are still bound by the memory transfer speed (to be double checked).
I thought the same when I saw that the financial industry was hiring FPGA people for low-latency algorithms. My understanding (as a non-FPGA expert) is that currently FPGA beats generic hardware (CPU,GPU) for "small size algorithm" (i.e that do not need GB of weights), while enabling a certain flexibility vs ASIC. My guess is that you cannot bake all the weights into the circuit topology, so you are still bound by th…
I'm trying to understand the benefits of streaming expert weights through hardware that offloads the math and avoids storing all of the weights in RAM at once.
But that's not the only thing that can be streamed and offloaded. '
Finally, I'm trying to come up with approaches to reuse old hardware, old GPUs, old RAM instead of paying today's prices for GPU VRAM or unified memory. Even if I do not end up showing any particular FPGA benefit, I might be able to better run very large models on systems without GPUs or without unified RAM.
You could build small ML model like talos-v2. But larger model/LLM requires much more engineering cost than you expected. Optimizing HDL has too many control knob/param to solve by RL.
I have a homelab, electronics lab, and software dev experience (electronics is a hobby, and an early career before I pivoted to O/S development). Before the recent RAM/GPU price jumps I had invested in a lot of used ECC RAM and many older GPUs (and some new GPUs) on many older servers that I refurbished and upgraded. It is hard to justify the prices of new GPUs/RAM going forward. It is cheaper to upgrade my CPUs (and I have) to have more cores.
Yes, my older hardware is slow by today's standards, but it is at least affordable. Yes, old hardware is power inefficient, but I justify that by using solar panels. I cannot do anything about the current supply/demand problems, but I can perhaps help with reuse and upcycling. Maybe my work will help students and junior programmers learn on old/used hardware.
ML inference using FPGA's is an active area of research, https://arxiv.org/html/2512.01463v1 and have been successfully used in LHC/HEP detectors such as CMS and ATLAS.