Crazy how little traction this kind of project gets on here. Posts about squeezing a 1+T model to seconds/token and you have this wave of optimism like "It's the effort that counts! We'll get there!". Sure, this particular project isn't really scalable in the same sense (PL fabric/use what ya got/cost/power) but IMO it's conceptually a brilliant thing to showcase comparatively. I have a strange feeling a decent chunk of people dismissing this project are the same who spent small fortunes on hobby llm inference setups/investments and see this as a useless exercise. Meanwhile dozens of $$$M startups in the CIM/analog compute/etc have been R&D'ing for years now that will make this same outcome a reality before we know it (crazy inference speeds on usable models within local reach). Anyways kudos to OP and really enjoyed the documentation and findings of this!
Show HN: A tiny LLM running at 21,000 tok/s on a $250 FPGA (Live Demo)
21–30 of 30 posts
Re: Show HN: A tiny LLM running at 21,000 tok/s on a $250 FPGA (Live Demo)
#22Earlier quoted context omitted.
FPGAs are not power efficient at all vs GPUs and ASICs anyway, which is going to be especially true when they are fully saturated by LLM inference.
Nothing can be as power efficient as an ASIC, which is designed for a specific purpose, instead of being a programmable device intended to be suitable for a large class of applications. A GPU is much more efficient than an FPGA for what a GPU does. On the other hand for applications for which the set of primitive operations implemented in hardware by a GPU is not a good fit, an FPGA can be much more power efficient t…
Yeah but then it's basically an AI ASIC with an FPGA block inside it. Basically the less FPGA-like an FPGA is, ie the more dedicated silicon in the FPGA for the task in question, the more power efficient it is, because custom logic in an FPGA is done in LUTs which is RAM and RAM is way way more power hungry than actual logic gates, and the fabric is apparently power hungry too. It's unfortunate to me because I like FPGAs and wish they weren't so niche, but they are inherently limited in this way.
Re: Show HN: A tiny LLM running at 21,000 tok/s on a $250 FPGA (Live Demo)
#23Hi everyone, I’m a friend of Mike’s; he’s having issues replying to the post at the moment, but hopes to post a thorough reply to the comments as soon as possible
Just a quick test to see if i'm back
Re: Show HN: A tiny LLM running at 21,000 tok/s on a $250 FPGA (Live Demo)
#24Crazy how little traction this kind of project gets on here. Posts about squeezing a 1+T model to seconds/token and you have this wave of optimism like "It's the effort that counts! We'll get there!". Sure, this particular project isn't really scalable in the same sense (PL fabric/use what ya got/cost/power) but IMO it's conceptually a brilliant thing to showcase comparatively. I have a strange feeling a decent chunk…
Appreciated. and yeah, agreed the interesting comparison isn't "is this scalable as-is" (it isn't, PL fabric, cost, power), it's that the CIM / analog-compute startups you mention are chasing exactly this endpoint with real money and years of R&D, the bar for making something useful is brutally high.
However, if no-one made anything that was useless on the same thesis, a lot of these concepts would have never got off the ground. I would hazard a guess that people like taalas would have started with a (much much bigger) fpga to validate whether the approach was possible before committing to designing a chip big enough to fit an 8B model in it.
I just nerd sniped myself...
VP1902 could fit around a 500m model in, whereas a cadence protium rack of them could squeeze in a ~6B at 8bit, or a ~13B at 4bit. So accounting for the headroom of distributed compute, Llama 3.1 8B at 4bit. I don't want to even estimate how long synthesis and place and route would take on that!
Re: Show HN: A tiny LLM running at 21,000 tok/s on a $250 FPGA (Live Demo)
#25conceptually it's a cool idea. practically the results seem about as coherent as import random; print(random.choice(list(my_dict))) ..but way slower is there a practical use to a model this small?
I 100% agree the model is basically useless. that was never the deliverable. the artifact here is the inference engine, not the model living in it. 3.16M params at character level is just what fits in ~3MB of on-chip SRAM.
Plus I treated it as a good learning experience to get better with FPGA's but also system design.
Re: Show HN: A tiny LLM running at 21,000 tok/s on a $250 FPGA (Live Demo)
#26Earlier quoted context omitted.
You're correct, the flat line is aggregate only. the fabric is saturated from a few dozen active clients onward, so extra connections can't buy throughput, they just queue. per-user p50/p95 across that same sweep: 17ms/30ms solo, 450ms/545ms at 100, ~2s/2.4s at 500, 3.8s/4.4s at 1000, 6.3s/9.4s at 2000. zero errors or drops at every stage. It degrades as a well-behaved queue, not a cliff, but nobody would call 6s at…
does the reflash actually stall every live connection, or just the ones whose request lands during that window? if the whole board goes dark for the full ~25s while any request is queued behind it, you could probably hide most of that behind partial reconfiguration, reflashing only the region holding the model weights while the sequencer and I/O logic on the rest of the fabric stay live and keep draining the queue. t…
[deleted]
Re: Show HN: A tiny LLM running at 21,000 tok/s on a $250 FPGA (Live Demo)
#27Earlier quoted context omitted.
You're correct, the flat line is aggregate only. the fabric is saturated from a few dozen active clients onward, so extra connections can't buy throughput, they just queue. per-user p50/p95 across that same sweep: 17ms/30ms solo, 450ms/545ms at 100, ~2s/2.4s at 500, 3.8s/4.4s at 1000, 6.3s/9.4s at 2000. zero errors or drops at every stage. It degrades as a well-behaved queue, not a cliff, but nobody would call 6s at…
does the reflash actually stall every live connection, or just the ones whose request lands during that window? if the whole board goes dark for the full ~25s while any request is queued behind it, you could probably hide most of that behind partial reconfiguration, reflashing only the region holding the model weights while the sequencer and I/O logic on the rest of the fabric stay live and keep draining the queue. t…
[dead]
Re: Show HN: A tiny LLM running at 21,000 tok/s on a $250 FPGA (Live Demo)
#28I started this about 10 weeks ago when the Taalas chatjimmy demo first did the rounds, Llama 8B baked into custom silicon, 17k tok/s for a single user. Their whole thesis is that inference is bound by reading the weights, so stop fetching them from far away. I wanted to see how far that idea stretches on a 'consumer hardware': every weight resident in SRAM, zero DRAM My chip only gives you ~3 MB to live in, so the mo…
IPO when?
Seriously, great stuff!
Re: Show HN: A tiny LLM running at 21,000 tok/s on a $250 FPGA (Live Demo)
#29Earlier quoted context omitted.
Nothing can be as power efficient as an ASIC, which is designed for a specific purpose, instead of being a programmable device intended to be suitable for a large class of applications. A GPU is much more efficient than an FPGA for what a GPU does. On the other hand for applications for which the set of primitive operations implemented in hardware by a GPU is not a good fit, an FPGA can be much more power efficient t…
Yeah but then it's basically an AI ASIC with an FPGA block inside it. Basically the less FPGA-like an FPGA is, ie the more dedicated silicon in the FPGA for the task in question, the more power efficient it is, because custom logic in an FPGA is done in LUTs which is RAM and RAM is way way more power hungry than actual logic gates, and the fabric is apparently power hungry too. It's unfortunate to me because I like F…
I agree.
Re: Show HN: A tiny LLM running at 21,000 tok/s on a $250 FPGA (Live Demo)
#30Really cool project. I wonder what the future of LLM inference will look like. The Talaas demo is promising, but using an ASIC with weights in ROM means you can’t update the model (weights or architecture) without replacing the entire chip. SRAM isn’t dense enough to store model weights, but DRAM has bandwidth issues unless you use HBM which is expensive. Maybe novel memory technologies are the future (there are a number of emerging technologies in R&D), but they likely require breakthroughs to become commercially viable. Systolic arrays could work, one could imagine architectures where routing (architecture) is fixed but weights are programmable, or architectures where the weights and routing are programmable but the compute units are fixed function (coarse grained architecturally reprogrammable), or maybe the weights are in ROM but can be hot swapped easily with into fixed compute elements. Definitely an interesting and emerging field.