Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

BitNet: Inference framework for 1-bit LLMs

github.com

Post reply on HN
Print view View original

71–80 of 179 posts

Re: BitNet: Inference framework for 1-bit LLMs

#71

Post by logicallee »

It might interest you to know that one or two months ago, I had Claude port BitNet to WebGPU from the reference implementation, so that it runs right in your browser as a local model. After some debugging, the port seemed to work, but the model didn't function as well as the reference implementation so I'll have to work on it for a while. You can see a debugging session livestreamed here[1]. The released model file was about a gigabyte, it fits in most people's GPU's. We were also able to successfully fine-tune it right in the browser.

There's a lot that you can do when the model size is that small, yet still powerful.

Our next step is that we want to put up a content distribution network for it where people can also share their diffs for their own fine-tuned model. I'll post the project if we finish all the parts.

[1] https://www.youtube.com/live/x791YvPIhFo?is=NfuDFTm9HjvA3nzN

Re: BitNet: Inference framework for 1-bit LLMs

#72

Post by regularfry »

radarsat1 wrote: post #5

I'm curious if 1-bit params can be compared to 4- or 8-bit params. I imagine that 100B is equivalent to something like a 30B model? I guess only evals can say. Still, being able to run a 30B model at good speed on a CPU would be amazing.

At some point you hit information limits. With conventional quantisation you see marked capability fall-off below q5. All else being equal you'd expect an N-parameter 5-bit quant to be roughly comparable to a 3N-parameter ternary, if they are trained to the same level, just in terms of the amount of information they can possibly hold. So yes, 100B ternary would be within the ballpark of a 30B q5 conventional model, with a lot of hand-waving and sufficiently-smart-training

Re: BitNet: Inference framework for 1-bit LLMs

#73

Post by regularfry »

simonw wrote: post #42

Anyone know how hard it would be to create a 1-bit variant of one of the recent Qwen 3.5 models?

There are q2 and q1 quants, if you want an idea of how much performance you'd drop. Not quite the same implementation-wise, but probably equivalent in terms of smarts.

Re: BitNet: Inference framework for 1-bit LLMs

#74

Post by observationist »

embedding-shape wrote: post #16
LuxBennu wrote: post #6

The title is misleading — there's no trained 100B model, just an inference framework that claims to handle one. But the engineering is worth paying attention to. I run quantized 70B models locally (M2 Max 96GB, llama.cpp + LiteLLM), and memory bandwidth is always the bottleneck. The 1.58-bit approach is interesting because ternary weights turn matmuls into additions — a fundamentally different compute profile on comm…

> Framework is ready. Now we need someone to actually train the model. If Microslop aren't gonna train the model themselves to prove their own thesis, why would others? They've had 2 years (I think?) to prove BitNet in at least some way, are you really saying they haven't tried so far? Personally that makes it slightly worrisome to just take what they say at face value, why wouldn't they train and publish a model the…

So is it finally time for a Beowulf cluster to do something amazing?

Re: BitNet: Inference framework for 1-bit LLMs

#75

Post by Schlagbohrer »

embedding-shape wrote: post #45
throwaw12 wrote: post #39

Earlier quoted context omitted.

Because this is Microsoft, experimenting and failing is not encouraged, taking less risky bets and getting promoted is. Also no customer asked them to have 1-bit model, hence PM didn't prioritize it. But it doesn't mean, idea is worthless. You could have said same about Transformers, Google released it, but didn't move forward, turns out it was a great idea.

> You could have said same about Transformers, Google released it, but didn't move forward, I don't think you can, Google looked at the research results, and continued researching Transformers and related technologies, because they saw the value for it particularly in translations. It's part of the original paper, what direction to take, give it a read, it's relatively approachable for being a machine learning paper…

Google had been working on a big LLM but they wanted to resolve all the safety concerns before releasing it. It was only when OpenAI went "YOLO! Check this out!" that Google then internally said, "Damn the safety concerns, full speed ahead!" and now we find ourselves in this breakneck race in which all safety concerns have been sidelined.

Re: BitNet: Inference framework for 1-bit LLMs

#76

Post by naasking »

rustyhancock wrote: post #12
LuxBennu wrote: post #6

The title is misleading — there's no trained 100B model, just an inference framework that claims to handle one. But the engineering is worth paying attention to. I run quantized 70B models locally (M2 Max 96GB, llama.cpp + LiteLLM), and memory bandwidth is always the bottleneck. The 1.58-bit approach is interesting because ternary weights turn matmuls into additions — a fundamentally different compute profile on comm…

Yes. I had to read it over twice, it does strike me as odd that there wasn't a base model to work with. But it seems the biggest model available is 10B? Somewhat unusual and does make me wonder just how challenging it will be to train any model in the 100B order of magnitude.

What's unusual about it? It seems pretty standard to train small models to validate an approach, and then show that training scales with model size to 8B to 14B parameter models, which is what they did.

Re: BitNet: Inference framework for 1-bit LLMs

#77

Post by RandomTeaParty »

LuxBennu wrote: post #6

The title is misleading — there's no trained 100B model, just an inference framework that claims to handle one. But the engineering is worth paying attention to. I run quantized 70B models locally (M2 Max 96GB, llama.cpp + LiteLLM), and memory bandwidth is always the bottleneck. The 1.58-bit approach is interesting because ternary weights turn matmuls into additions — a fundamentally different compute profile on comm…

> The 1.58-bit approach

can we stop already with these decimals and just call it "1 trit" which it exactly is?

Re: BitNet: Inference framework for 1-bit LLMs

#79

Post by Jowsey »

cubefox wrote: post #17
LuxBennu wrote: post #6

The title is misleading — there's no trained 100B model, just an inference framework that claims to handle one. But the engineering is worth paying attention to. I run quantized 70B models locally (M2 Max 96GB, llama.cpp + LiteLLM), and memory bandwidth is always the bottleneck. The 1.58-bit approach is interesting because ternary weights turn matmuls into additions — a fundamentally different compute profile on comm…

LLM account

Agreed. This is becoming an issue, see also: https://news.ycombinator.com/item?id=47259308

Re: BitNet: Inference framework for 1-bit LLMs

#80

Post by gardnr »

Schlagbohrer wrote: post #75
embedding-shape wrote: post #45

Earlier quoted context omitted.

> You could have said same about Transformers, Google released it, but didn't move forward, I don't think you can, Google looked at the research results, and continued researching Transformers and related technologies, because they saw the value for it particularly in translations. It's part of the original paper, what direction to take, give it a read, it's relatively approachable for being a machine learning paper…

Google had been working on a big LLM but they wanted to resolve all the safety concerns before releasing it. It was only when OpenAI went "YOLO! Check this out!" that Google then internally said, "Damn the safety concerns, full speed ahead!" and now we find ourselves in this breakneck race in which all safety concerns have been sidelined.

Scaling seemed like the important idea that everyone was chasing. OpenAI used to be a lot more safety minded because it was in their non profit charter, now they’ve gone for-profit and weaponized their tech for the USA military. Pretty wild turnaround. Saying OpenAI was cavalier with safety in the early days is inaccurate. It was a skill issue. Remember Bard? Google was slow.
Post reply on HN