1-Bit AI Infrastructure
arxiv.org
1-Bit AI Infrastructure
1–10 of 33 posts
Re: 1-Bit AI Infrastructure
#2Later a4.8 quantization by some of the same team:
Re: 1-Bit AI Infrastructure
#3[deleted]
Re: 1-Bit AI Infrastructure
#4[deleted]
Re: 1-Bit AI Infrastructure
#5Later a4.8 quantization by some of the same team: https://news.ycombinator.com/item?id=42092724 https://arxiv.org/abs/2411.04965
and the repo for this project: https://github.com/microsoft/BitNet
Re: 1-Bit AI Infrastructure
#6I have read about it quite a few weeks ago the first time and I found it very interesting.
Now that I have done more than enough CPU design inside FPGAs, I wanted to try something new, some computation heavy things that could benefit from an FPGA. Does anyone here know how feasable it'd be to implement something like that on an FPGA? I only have rather small chips (artix-7 35T and polarfire SoC with 95k logic slices). So I know I won't be able to press a full LLM into that, but something should be possible.
Maybe I should refresh the fundamentals though and start with MNIST. But the question is rather: What is a realistic goal that I could possibly reach with these small FPGAs? Performance might be secondary, I am rather interested in what's possible regarding complexity/features on a small device.
Also has anyone here compiled openCL (or GL?) kernels for FPGAs and can give me a starting point? I was wondering if it's possible to have a working backend for something like tinygrad[1]. I think this would be a good way to learn all the different layers on how such frameworks actually work
Re: 1-Bit AI Infrastructure
#7I would expect research along this way to pick up quite a bit if we confirm the pretrain stage is not scaling as previous expected, thus the scale and architecture would be more stable in the near future, especially if the focus shifts to inference time scaling.
Re: 1-Bit AI Infrastructure
#8Later a4.8 quantization by some of the same team: https://news.ycombinator.com/item?id=42092724 https://arxiv.org/abs/2411.04965
and the repo for this project: https://github.com/microsoft/BitNet
The demo they showed was full of repeated sentences. The 3B model looks quite dense, TBH. Did they just want to show the speed?
Re: 1-Bit AI Infrastructure
#9I have read about it quite a few weeks ago the first time and I found it very interesting. Now that I have done more than enough CPU design inside FPGAs, I wanted to try something new, some computation heavy things that could benefit from an FPGA. Does anyone here know how feasable it'd be to implement something like that on an FPGA? I only have rather small chips (artix-7 35T and polarfire SoC with 95k logic slices)…
Couldn't you implement a bitnet kernel, and use that as a co-processor to a PC? Or is the I/O bandwidth so low that it won't be worth it?
Re: 1-Bit AI Infrastructure
#10I have read about it quite a few weeks ago the first time and I found it very interesting. Now that I have done more than enough CPU design inside FPGAs, I wanted to try something new, some computation heavy things that could benefit from an FPGA. Does anyone here know how feasable it'd be to implement something like that on an FPGA? I only have rather small chips (artix-7 35T and polarfire SoC with 95k logic slices)…
You gain in potential parallelism with FPGA, so with very small "at the edge" models they could speed things up, right? But the models are always going to be large, so memory bandwidth is going to be a bottle neck unless some v fancy FPGA memory "fabric" is possible. Perhaps for extremely low latency classification tasks? I'm having trouble picturing that application though.
The code itself is surprisingly small/tight. I'm been playing with llama.cpp for the last few days. The CPU only archive is like 8Mb on gitlab, and there is no memory allocation during run time. My ancient laptop (as in 2014!) is sweating but producing spookily good output with quantized 7B models.
(I'm mainly commenting to have someone correct me, by the way, since I'm interested in this question too!)