11–20 of 30 posts
I didn't expect the 2,000 connection sweep to stay flat, since all of them are sharing one stream. What does per user latency look like at that end of the sweep?
[0] https://news.ycombinator.com/item?id=49244312
@dang, the creator of this idea is having his comments killed off for some reason.
But the problem is not that your model is fast. Sure, you can go ASIC and go even faster, but the thing around GPU is that they scale well for both training and inference, and the technical floor is low. The level to get into FPGA design is insanely high, you've got to read timing diagrams, you need to know combinatorial and sequential logics and good sense of boolean algebra, you need to have an asynchronous signal…
FPGAs are not power efficient at all vs GPUs and ASICs anyway, which is going to be especially true when they are fully saturated by LLM inference.
Anything else is added and unintentional benefits.
Ignore the naysayers! Any article, even the really good ones on HN, while they get positive comments, for whatever reason, always get a lot of negative ones, too... That is, the negative comments are absolutely unavoidable, even for people accomplishing great things! I personally think that what you've done is brilliant, absolutely brilliant! I can't wait to see more in this space... Brilliant, absolutely brilliant!
But model(s) implemented are clearly too small to be useful as a 'chat partner'. Tried a couple of sentences - replies is just some gibberish coming out.
This really needs a bigger FPGA, or some other application(s) where a tiny LLM does actually useful work. Barring that, generated tokens/sec is kind of a meaningless measure imho.
Earlier quoted context omitted.
FPGAs are not power efficient at all vs GPUs and ASICs anyway, which is going to be especially true when they are fully saturated by LLM inference.
That said, FPGA do provide a middle ground, but using it for speed and power efficient is not a forte, and the true value exactly comes from this focus alone: it allows you do emulate systhesis and verify that your logic is correct before you do full ASIC tapeout, e.g. building softcores for CPU validation Anything else is added and unintentional benefits.
I didn't expect the 2,000 connection sweep to stay flat, since all of them are sharing one stream. What does per user latency look like at that end of the sweep?
You're correct, the flat line is aggregate only. the fabric is saturated from a few dozen active clients onward, so extra connections can't buy throughput, they just queue. per-user p50/p95 across that same sweep: 17ms/30ms solo, 450ms/545ms at 100, ~2s/2.4s at 500, 3.8s/4.4s at 1000, 6.3s/9.4s at 2000. zero errors or drops at every stage. It degrades as a well-behaved queue, not a cliff, but nobody would call 6s at…
practically the results seem about as coherent as
import random; print(random.choice(list(my_dict)))
is there a practical use to a model this small?
But the problem is not that your model is fast. Sure, you can go ASIC and go even faster, but the thing around GPU is that they scale well for both training and inference, and the technical floor is low. The level to get into FPGA design is insanely high, you've got to read timing diagrams, you need to know combinatorial and sequential logics and good sense of boolean algebra, you need to have an asynchronous signal…
The hardest bit is probably SystemVerilog - it's just such a terrible language for hardware design. Full of footguns and gotchas and weird limitations and undocumented or tool-dependent stuff that you have to just know (like what is synthesizable).
Approximately nobody uses Yosys.
But the problem is not that your model is fast. Sure, you can go ASIC and go even faster, but the thing around GPU is that they scale well for both training and inference, and the technical floor is low. The level to get into FPGA design is insanely high, you've got to read timing diagrams, you need to know combinatorial and sequential logics and good sense of boolean algebra, you need to have an asynchronous signal…
FPGAs are not power efficient at all vs GPUs and ASICs anyway, which is going to be especially true when they are fully saturated by LLM inference.
A GPU is much more efficient than an FPGA for what a GPU does. On the other hand for applications for which the set of primitive operations implemented in hardware by a GPU is not a good fit, an FPGA can be much more power efficient than a GPU.
For applications that involve a massive amount of computations with FP32, FP16 or BF16 numbers, for which GPUs have special hardware execution units, i.e. for training and for inference with non-quantized models, there is no chance for an FPGA to be more efficient.
If the GPU is recent enough to have good support for more heavily quantized data types, e.g. INT8, FP8, NVFP4 etc. an FPGA also does not have chances to be competitive.
An FPGA could be more efficient than a GPU if either it is some special AI-oriented FPGA, which instead of having traditional arithmetic units oriented for DSP applications, has execution units implementing the quantized data types popular in ML/AI, or if it implements inference using some new not yet standardized data type, for which GPUs do not have dedicated support yet.
But the problem is not that your model is fast. Sure, you can go ASIC and go even faster, but the thing around GPU is that they scale well for both training and inference, and the technical floor is low. The level to get into FPGA design is insanely high, you've got to read timing diagrams, you need to know combinatorial and sequential logics and good sense of boolean algebra, you need to have an asynchronous signal…
FPGAs aren't that difficult to program. Waves / timing diagrams are trivial to understand, they can just be very tedious to read. The different execution model to CPUs also isn't very hard to understand IMO. Hardware people love to say that it is, but it really isn't. The hardest bit is probably SystemVerilog - it's just such a terrible language for hardware design. Full of footguns and gotchas and weird limitations…