Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

BitNet: Inference framework for 1-bit LLMs

github.com

Post reply on HN
Print view View original

101–110 of 179 posts

Re: BitNet: Inference framework for 1-bit LLMs

#101

Post by SoftTalker »

K0balt wrote: post #51
vova_hn2 wrote: post #34

Earlier quoted context omitted.

First they claimed that if you use em dashes you are not human And I did not speak out Because I was not using em dashes Then they claimed that if you're crammar is to gud you r not hmuan And I did not spek aut Because mi gramar sukcs Then they claimed that if you actually read the article that you are trying to discuss you are not human...

I’ve been rounded up for things I wrote two decades ago because of my em dashes lol. The pitchfork mentality gives me little hope for how things are going to go once we have hive mind AGI robots pervasive in society.

I once spent some time learning the proper usage of em-dashes, en-dashes, and hyphens, and tried to be conscientious about using them properly in my writing. Little did I know it would be wasted effort in the LLM era, when competent writing actually became a negative.

Not only are we losing the ability to communicate clearly without the assistance of computers, those who can are being punished for it.

Re: BitNet: Inference framework for 1-bit LLMs

#102

Post by ismailmaj »

WithinReason wrote: post #94
ismailmaj wrote: post #68

Earlier quoted context omitted.

You drop the memory throughput requirements because of the packed representation of bits so an FMA can become the bottleneck, and you bypass the problem of needing to upscale the bits to whatever FP the FMA instruction needs. typically for 1-bit matmul, you can get away with xors and pop_counts which should have a better throughput profile than FMA when taking into account the SIMD nature of the inputs/outputs.

yes but this is not 1 bit matmul, it's 1.58 bits with expensive unpacking

The title and the repo uses 1-bit when it means 1.58 bits tertiary values, it doesn't change any of my arguments (still xors and pop_counts).

Re: BitNet: Inference framework for 1-bit LLMs

#104

Post by buo »

monocasa wrote: post #90
wongarsu wrote: post #15

Earlier quoted context omitted.

I've also always though that it's an interesting opportunity for custom hardware. Two bit addition is incredibly cheap in hardware, especially compared to anything involving floating point. You could make huge vector instructions on the cheap, then connect it to the fastest memory you can buy, and you have a capable inference chip. You'd still need full GPUs for training, but for inference the hardware would be order…

These are trits, which provide their own efficiencies. Interestingly, a trit x float multiplier is cheaper than a trit x integer multiplier in hardware if you're willing to ignore things like NaNs. 0 and 1 are trivial, just a mux for identity and zero. But because floats are sign-magnitude, multiply by -1 is just an inverter for the sign bit, where as for integers you need a bitwise inverter and full incrermenter.

Do you know a good reference to learn more about this (quantizing weigths to 1.58 bits, and trit arithmetic)?

Re: BitNet: Inference framework for 1-bit LLMs

#105

Post by deepsquirrelnet »

LuxBennu wrote: post #6

The title is misleading — there's no trained 100B model, just an inference framework that claims to handle one. But the engineering is worth paying attention to. I run quantized 70B models locally (M2 Max 96GB, llama.cpp + LiteLLM), and memory bandwidth is always the bottleneck. The 1.58-bit approach is interesting because ternary weights turn matmuls into additions — a fundamentally different compute profile on comm…

The title being misleading is important as well, because this has landed on the front page, and the only thing that would be the only notable part of this submission.

The "new" on huggingface banner has weights that were uploaded 11 months ago, and it's 2B params. Work on this in the repo is 2 years old.

The amount of publicity compared to the anemic delivery for BitNet is impressive.

Re: BitNet: Inference framework for 1-bit LLMs

#107

Post by bigyabai »

DrBazza wrote: post #100
LuxBennu wrote: post #6

The title is misleading — there's no trained 100B model, just an inference framework that claims to handle one. But the engineering is worth paying attention to. I run quantized 70B models locally (M2 Max 96GB, llama.cpp + LiteLLM), and memory bandwidth is always the bottleneck. The 1.58-bit approach is interesting because ternary weights turn matmuls into additions — a fundamentally different compute profile on comm…

> memory bandwidth is always the bottleneck I'm hoping that today's complaints are tomorrow's innovations. Back when 1Mb hard drive was $100,000, or when Gates said 640kb is enough. Perhaps some 'in the (chip) industry' can comment on what RAM manufacturers are doing at the moment - better, faster, larger? Or is there not much headroom left and it's down to MOBO manufacturers, and volume?

For larger contexts, the bottleneck is probably token prefill instead of memory bandwidth. Supposedly prefill is faster on the M5+ GPUs, but still a big hurdle for pre-M5 chips.

Re: BitNet: Inference framework for 1-bit LLMs

#108

Post by perfmode »

patchnull wrote: post #106

[flagged]

The quality cliff question is the right one to be asking. There's a pattern in systems work where something that scales cleanly in theory hits emergent failure modes at production scale that weren't visible in smaller tests. The loss landscape concern is exactly that kind of thing, and nobody has actually run the experiment.

That said, I think the comparison to improving GGUF quantization isn't quite apples to apples. Post-training quantization is compressing a model that already learned its representations in high precision. Native ternary training is making an architectural bet that the model can learn equally expressive representations under a much tighter constraint from the start. Those are different propositions with different scaling characteristics. The BitNet papers suggest the native approach wins at small scale, but that could easily be because the quantization baselines they compared against (Llama 3 at 1.58 bits) were just bad. A full-precision model wasn't designed to survive that level of compression.

The real tell will be whether anyone with serious compute (not Microsoft, apparently) decides the potential inference cost savings justify a full training run. The framework existing lowers one barrier, but the more important barrier is that a failed 100B training run is extremely expensive, and right now there's not enough evidence to derisk it. Two years of framework polish without a flagship model is a notable absence.

Re: BitNet: Inference framework for 1-bit LLMs

#110

Post by andai »

patchnull wrote: post #106

[flagged]

>Meanwhile GGUF Q2 and Q3 quantizations on llama.cpp keep getting better

Can you tell me more about this? It's been about a year since I looked into it, but it looked like performance dropped hard below Q4. I'd love to see more about this.

Also what's a good way to run them? I mostly use Ollama which only goes down to Q4. I think it supports HF urls though?

Post reply on HN