Show HN: A tiny LLM running at 21,000 tok/s on a $250 FPGA (Live Demo)
1–10 of 13 posts
Re: Show HN: A tiny LLM running at 21,000 tok/s on a $250 FPGA (Live Demo)
#2[dead]
Re: Show HN: A tiny LLM running at 21,000 tok/s on a $250 FPGA (Live Demo)
#3I didn't expect the 2,000 connection sweep to stay flat, since all of them are sharing one stream. What does per user latency look like at that end of the sweep?
Re: Show HN: A tiny LLM running at 21,000 tok/s on a $250 FPGA (Live Demo)
#4[dead]
Re: Show HN: A tiny LLM running at 21,000 tok/s on a $250 FPGA (Live Demo)
#5I didn't expect the 2,000 connection sweep to stay flat, since all of them are sharing one stream. What does per user latency look like at that end of the sweep?
You're correct, the flat line is aggregate only. the fabric is saturated from a few dozen active clients onward, so extra connections can't buy throughput, they just queue. per-user p50/p95 across that same sweep: 17ms/30ms solo, 450ms/545ms at 100, ~2s/2.4s at 500, 3.8s/4.4s at 1000, 6.3s/9.4s at 2000. zero errors or drops at every stage. It degrades as a well-behaved queue, not a cliff, but nobody would call 6s at the top end snappy.
The benchmark sweep is 2,000 concurrent active requesters hammering it constantly, real traffic is mostly lurkers, which cost a file descriptor and nothing else. the interactive feel actually gives out earlier than the queue math. The speculative-typing UI wants sub-second replies, and that budget blows around 100–150 simultaneous typists.
I've been logging the stats since it went live, unfortunately it didn't hit FP. Peak was 10 concurrent connections (13 uniques in the busiest half hour), ~580 requests and ~37k tokens served, and at no point did two people actually have an inference in flight at the same moment which would have been the real test for the queue, every visitor got the fabric to themselves, p50 ~23ms. so the 2,000-conn drill was not stressed today. the one blemish: a single window with p95 ~57s, which lines up with the model-rotation FPGA reconfigure rather than load. A request that arrives mid-reflash waits out the ~25s swap. if this thread sends 50× more people, the queue math above says it holds.
I need to discard the requests that overlap the model changeover for a truer result.
Re: Show HN: A tiny LLM running at 21,000 tok/s on a $250 FPGA (Live Demo)
#6But the problem is not that your model is fast.
Sure, you can go ASIC and go even faster, but the thing around GPU is that they scale well for both training and inference, and the technical floor is low.
The level to get into FPGA design is insanely high, you've got to read timing diagrams, you need to know combinatorial and sequential logics and good sense of boolean algebra, you need to have an asynchronous signal based mindset which is vastly different from CPU/GPU, you need to know netlist and you need to endure the time it takes for the EDA to finish generating it. Yosys is still years behind Xilinx
There is a reason GPUs are called accelerators; it sacrifices and does not try to really specialize on one particular thing, except high parallel dataflow and branch-free calculation. Otherwise we will all be using DSPs
Re: Show HN: A tiny LLM running at 21,000 tok/s on a $250 FPGA (Live Demo)
#7But the problem is not that your model is fast. Sure, you can go ASIC and go even faster, but the thing around GPU is that they scale well for both training and inference, and the technical floor is low. The level to get into FPGA design is insanely high, you've got to read timing diagrams, you need to know combinatorial and sequential logics and good sense of boolean algebra, you need to have an asynchronous signal…
FPGAs are not power efficient at all vs GPUs and ASICs anyway, which is going to be especially true when they are fully saturated by LLM inference.
Re: Show HN: A tiny LLM running at 21,000 tok/s on a $250 FPGA (Live Demo)
#8Ignore the naysayers!
Any article, even the really good ones on HN, while they get positive comments, for whatever reason, always get a lot of negative ones, too...
That is, the negative comments are absolutely unavoidable, even for people accomplishing great things!
I personally think that what you've done is brilliant, absolutely brilliant!
I can't wait to see more in this space...
Brilliant, absolutely brilliant!
Re: Show HN: A tiny LLM running at 21,000 tok/s on a $250 FPGA (Live Demo)
#9Ignore the naysayers! Any article, even the really good ones on HN, while they get positive comments, for whatever reason, always get a lot of negative ones, too... That is, the negative comments are absolutely unavoidable, even for people accomplishing great things! I personally think that what you've done is brilliant, absolutely brilliant! I can't wait to see more in this space... Brilliant, absolutely brilliant!
I am also very cautious with people who tell me something impossible when I can trust my engineering skills and get a good sense that there is potentially a good outcome. In my experience, it simply means they don’t know how to do it, or are frustrated they couldn’t do it themselves and get into the spotlight.
Re: Show HN: A tiny LLM running at 21,000 tok/s on a $250 FPGA (Live Demo)
#10Hi everyone, I’m a friend of Mike’s; he’s having issues replying to the post at the moment, but hopes to post a thorough reply to the comments as soon as possible