Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

BitNet: Inference framework for 1-bit LLMs

github.com

Post reply on HN
Print view View original

81–90 of 179 posts

Re: BitNet: Inference framework for 1-bit LLMs

#81

Post by naasking »

StilesCrisis wrote: post #67

The output from this model is horrible! It's GPT-2 level babble and repeats entire paragraphs verbatim. It also reuses the same fake citation `(Jenkins, 2010)` over and over again. From the start of their video (which scrolls by fast enough that you don't see the slop clearly...) ``` Ecosystem Services and their impact on the Ecosystem Ecosystem services refer to the services provided by ecosystems to the human socie…

It's a two year old base model that's only 3B parameters, trained on only 100B tokens. It's still a research project at this point.

Re: BitNet: Inference framework for 1-bit LLMs

#83

Post by ActivePattern »

WithinReason wrote: post #25
LuxBennu wrote: post #6

The title is misleading — there's no trained 100B model, just an inference framework that claims to handle one. But the engineering is worth paying attention to. I run quantized 70B models locally (M2 Max 96GB, llama.cpp + LiteLLM), and memory bandwidth is always the bottleneck. The 1.58-bit approach is interesting because ternary weights turn matmuls into additions — a fundamentally different compute profile on comm…

> a fundamentally different compute profile on commodity CPU In what way? On modern processors, a Fused Multiply-Add (FMA) instruction generally has the exact same execution throughput as a basic addition instruction

The win is in how many weights you process per instruction and how much data you load.

So it's not that individual ops are faster — it's that the packed representation lets each instruction do more useful work, and you're moving far less data from memory to do it.

Re: BitNet: Inference framework for 1-bit LLMs

#84

Post by webXL »

verdverm wrote: post #65
webXL wrote: post #62

Earlier quoted context omitted.

It comes from (intentionally?) misleading docs: https://github.com/microsoft/BitNet/issues/391 (only suggesting that it's intentional because it's been there so long)

That issue appears to be the one that's wrong. From the technical report > We evaluated bitnet.cpp in terms of both inference speed and energy cost. Comprehensive tests were conducted on models with various parameter sizes, ranging from 125M to 100B. specific configurations for each model are detailed in the Appendix A.

Thanks for pointing that out. I'll ask the issue creator if they've considered that. Would be nice if the maintainer would handle that (sigh) and link to the actual models used for testing (double sigh).

Re: BitNet: Inference framework for 1-bit LLMs

#85

Post by cubefox »

Arcuru wrote: post #31

It's good to see this getting some continued development. I looked into it last year[1] and I thought it showed a lot of promise so I've been very disappointed that I never saw a newer model. [1] - https://jackson.dev/post/dont-sleep-on-bitnet/

I think this approach is not so interesting because it's just quantization of a full precision model. So it speeds up inference (at a quality penalty) but not training. It would be more interesting to train an actually binary model directly, without any floating point multiplication, like in this paper: https://proceedings.neurips.cc/paper_files/paper/2024/hash/7...

Re: BitNet: Inference framework for 1-bit LLMs

#86

Post by webXL »

152334H wrote: post #3

but there is no trained 100b param model? "can run a 100B BitNet" is about the inference implementation, not about the existence of any such model

I think they used a dummy model or else they would have linked to it. Just google '1-bit 100b model' and you'll only see references to this project without any download links.

Re: BitNet: Inference framework for 1-bit LLMs

#87

Post by cubefox »

regularfry wrote: post #72
radarsat1 wrote: post #5

I'm curious if 1-bit params can be compared to 4- or 8-bit params. I imagine that 100B is equivalent to something like a 30B model? I guess only evals can say. Still, being able to run a 30B model at good speed on a CPU would be amazing.

At some point you hit information limits. With conventional quantisation you see marked capability fall-off below q5. All else being equal you'd expect an N-parameter 5-bit quant to be roughly comparable to a 3N-parameter ternary, if they are trained to the same level, just in terms of the amount of information they can possibly hold. So yes, 100B ternary would be within the ballpark of a 30B q5 conventional model, w…

I assume that theoretically, 1-bit models could be most efficient because modern models switched from 32 bit to 16 bit to 8 bit per parameter (without quantization).

Re: BitNet: Inference framework for 1-bit LLMs

#89

Post by herf »

https://arxiv.org/pdf/2310.11453 The original paper [fig 1, bottom-right] seems to say it needs about 4-5x the parameters of a fp16 model. You can build it and run some models, but the selection is limited because it has to be trained from scratch. I imagine inference speed is faster compared with modern PTQ (4- and 8-bit quants) though.

Re: BitNet: Inference framework for 1-bit LLMs

#90

Post by monocasa »

wongarsu wrote: post #15
LuxBennu wrote: post #6

The title is misleading — there's no trained 100B model, just an inference framework that claims to handle one. But the engineering is worth paying attention to. I run quantized 70B models locally (M2 Max 96GB, llama.cpp + LiteLLM), and memory bandwidth is always the bottleneck. The 1.58-bit approach is interesting because ternary weights turn matmuls into additions — a fundamentally different compute profile on comm…

I've also always though that it's an interesting opportunity for custom hardware. Two bit addition is incredibly cheap in hardware, especially compared to anything involving floating point. You could make huge vector instructions on the cheap, then connect it to the fastest memory you can buy, and you have a capable inference chip. You'd still need full GPUs for training, but for inference the hardware would be order…

These are trits, which provide their own efficiencies.

Interestingly, a trit x float multiplier is cheaper than a trit x integer multiplier in hardware if you're willing to ignore things like NaNs.

0 and 1 are trivial, just a mux for identity and zero. But because floats are sign-magnitude, multiply by -1 is just an inverter for the sign bit, where as for integers you need a bitwise inverter and full incrermenter.

Post reply on HN