BitNet: Inference framework for 1-bit LLMs
1–10 of 179 posts
Re: BitNet: Inference framework for 1-bit LLMs
#2headline hundred billion parameter, none of the official models are over 10 billion parameters. Curious.
Re: BitNet: Inference framework for 1-bit LLMs
#3but there is no trained 100b param model? "can run a 100B BitNet" is about the inference implementation, not about the existence of any such model
Re: BitNet: Inference framework for 1-bit LLMs
#4Misleading title but this is pretty exciting. Interesting how this is based on llama cpp. Its nice to see some momentum since they released the paper in 2023
Re: BitNet: Inference framework for 1-bit LLMs
#5I'm curious if 1-bit params can be compared to 4- or 8-bit params. I imagine that 100B is equivalent to something like a 30B model? I guess only evals can say. Still, being able to run a 30B model at good speed on a CPU would be amazing.
Re: BitNet: Inference framework for 1-bit LLMs
#6The title is misleading — there's no trained 100B model, just an inference framework that claims to handle one. But the engineering is worth paying attention to.
I run quantized 70B models locally (M2 Max 96GB, llama.cpp + LiteLLM), and memory bandwidth is always the bottleneck. The 1.58-bit approach is interesting because ternary weights turn matmuls into additions — a fundamentally different compute profile on commodity CPUs. If 5-7 tok/s on a single CPU for 100B-class models is reproducible, that's a real milestone for on-device inference.
Framework is ready. Now we need someone to actually train the model.
Re: BitNet: Inference framework for 1-bit LLMs
#7> bitnet.cpp is the official inference framework for 1-bit LLMs (e.g., BitNet b1.58). It offers a suite of optimized kernels, that support fast and lossless inference of 1.58-bit models on CPU and GPU (NPU support will coming next).
One bit or one trit? I am confused!
Re: BitNet: Inference framework for 1-bit LLMs
#8> bitnet.cpp is the official inference framework for 1-bit LLMs (e.g., BitNet b1.58). It offers a suite of optimized kernels, that support fast and lossless inference of 1.58-bit models on CPU and GPU (NPU support will coming next). One bit or one trit? I am confused!
"1-bit LLMs" is just marketing. The Shannon entropy of one letter with a 3 symbol alphabet (-1, 0, 1) is 1.58.
Re: BitNet: Inference framework for 1-bit LLMs
#9https://github-production-user-asset-6210df.s3.amazonaws.com...
demo shows a huge love for water, this AI knows its home
Re: BitNet: Inference framework for 1-bit LLMs
#10headline hundred billion parameter, none of the official models are over 10 billion parameters. Curious.
The project is an inference framework which should support 100B parameter model at 5-7tok/s on CPU. No one has quantized a 100B parameter model to 1 trit, but this existing is an incentive for someone to do so.