It seems like arxiv replaced 'bitnet.cpp' with a link 'this http url', even though '.cpp' is clearly not a tld. Poor regex?
1-Bit AI Infrastructure
11–20 of 33 posts
Re: 1-Bit AI Infrastructure
#12It seems like arxiv replaced 'bitnet.cpp' with a link 'this http url', even though '.cpp' is clearly not a tld. Poor regex?
Sort of. And not on the author’s side.
https://academia.stackexchange.com/questions/132315/how-to-a...
Re: 1-Bit AI Infrastructure
#13I have read about it quite a few weeks ago the first time and I found it very interesting. Now that I have done more than enough CPU design inside FPGAs, I wanted to try something new, some computation heavy things that could benefit from an FPGA. Does anyone here know how feasable it'd be to implement something like that on an FPGA? I only have rather small chips (artix-7 35T and polarfire SoC with 95k logic slices)…
You gain in potential parallelism with FPGA, so with very small "at the edge" models they could speed things up, right? But the models are always going to be large, so memory bandwidth is going to be a bottle neck unless some v fancy FPGA memory "fabric" is possible. Perhaps for extremely low latency classification tasks? I'm having trouble picturing that application though. The code itself is surprisingly small/tigh…
> Perhaps for extremely low latency classification tasks? I'm having trouble picturing that application though.
Possibly, yes. I have no concrete plans yet. Maybe language models are the wrong area though. Some general either image classification or object detection would be neat (say lane detection with a camera or something like that)
Re: 1-Bit AI Infrastructure
#14I have read about it quite a few weeks ago the first time and I found it very interesting. Now that I have done more than enough CPU design inside FPGAs, I wanted to try something new, some computation heavy things that could benefit from an FPGA. Does anyone here know how feasable it'd be to implement something like that on an FPGA? I only have rather small chips (artix-7 35T and polarfire SoC with 95k logic slices)…
Couldn't you implement a bitnet kernel, and use that as a co-processor to a PC? Or is the I/O bandwidth so low that it won't be worth it?
Since I don't have a board with PCIe port the fastest I could get is 100MBit ethernet, i think. Or rather use the Microchip board which has a hard RISC-V quad core processor on it connected via an AXI-Bus with the FPGA fabric. The CPU itself run at only 625MHz, so there is huge potential to speed up some fancy computation
Re: 1-Bit AI Infrastructure
#15It's technically not 1-bit, but 2-bit.
Anyway, I wonder if there is some HW support in modern CPUs/GPUs for linear algebra (like matrix multiplication) over Z_2^n ? I think it would be useful for SAT solving.
Re: 1-Bit AI Infrastructure
#16Earlier quoted context omitted.
and the repo for this project: https://github.com/microsoft/BitNet
The demo they showed was full of repeated sentences. The 3B model looks quite dense, TBH. Did they just want to show the speed?
3B models, especially in quantized state, almost always behave like this.
Re: 1-Bit AI Infrastructure
#17It's technically not 1-bit, but 2-bit. Anyway, I wonder if there is some HW support in modern CPUs/GPUs for linear algebra (like matrix multiplication) over Z_2^n ? I think it would be useful for SAT solving.
not CPU/GPU but on FPGA finite field arithmetic is a thing; plenty of stuff like this around https://ieeexplore.ieee.org/document/4392002
Re: 1-Bit AI Infrastructure
#18It's technically not 1-bit, but 2-bit. Anyway, I wonder if there is some HW support in modern CPUs/GPUs for linear algebra (like matrix multiplication) over Z_2^n ? I think it would be useful for SAT solving.
There's carry-less multiplication (https://en.m.wikipedia.org/wiki/CLMUL_instruction_set), introduced by Intel in 2010.
Re: 1-Bit AI Infrastructure
#19I have read about it quite a few weeks ago the first time and I found it very interesting. Now that I have done more than enough CPU design inside FPGAs, I wanted to try something new, some computation heavy things that could benefit from an FPGA. Does anyone here know how feasable it'd be to implement something like that on an FPGA? I only have rather small chips (artix-7 35T and polarfire SoC with 95k logic slices)…
This submission should help you:
Re: 1-Bit AI Infrastructure
#20Earlier quoted context omitted.
Couldn't you implement a bitnet kernel, and use that as a co-processor to a PC? Or is the I/O bandwidth so low that it won't be worth it?
Since I don't have a board with PCIe port the fastest I could get is 100MBit ethernet, i think. Or rather use the Microchip board which has a hard RISC-V quad core processor on it connected via an AXI-Bus with the FPGA fabric. The CPU itself run at only 625MHz, so there is huge potential to speed up some fancy computation
Even with a PCIe FPGA card you're still going to be memory bound during inference. When running LLama.cpp on straight CPU memory bandwidth, not CPU power, is always the bottleneck.
Now if the FPGA card had a large amount of GPU tier memory then that would help.