If this stuff was so revolutionary, don't you guys think Qwen/DeepSeek would have snapped it up already? Both those teams are highly innovative, picking up and inventing new techniques all the time. Hell, Deepseek-v3 was one of the first to do large scale fp8 training.
BitNet: Inference framework for 1-bit LLMs
141–150 of 179 posts
Re: BitNet: Inference framework for 1-bit LLMs
#142[flagged]
Re: BitNet: Inference framework for 1-bit LLMs
#143Earlier quoted context omitted.
> You could have said same about Transformers, Google released it, but didn't move forward, I don't think you can, Google looked at the research results, and continued researching Transformers and related technologies, because they saw the value for it particularly in translations. It's part of the original paper, what direction to take, give it a read, it's relatively approachable for being a machine learning paper…
Google had been working on a big LLM but they wanted to resolve all the safety concerns before releasing it. It was only when OpenAI went "YOLO! Check this out!" that Google then internally said, "Damn the safety concerns, full speed ahead!" and now we find ourselves in this breakneck race in which all safety concerns have been sidelined.
They thought people might prefer quality and safety.
Re: BitNet: Inference framework for 1-bit LLMs
#144Why did you call it a 100b parameter model? it is not 100b parameters. they published a 1b parameter and a 2b parameter model.
Furthermore, it was published 11 months ago, it's not a new release.
Re: BitNet: Inference framework for 1-bit LLMs
#145Earlier quoted context omitted.
> You could have said same about Transformers, Google released it, but didn't move forward, I don't think you can, Google looked at the research results, and continued researching Transformers and related technologies, because they saw the value for it particularly in translations. It's part of the original paper, what direction to take, give it a read, it's relatively approachable for being a machine learning paper…
What OpenAI did was train increasingly large transformer model instances. which was sensible because transformers allowed for a scaling up of training compared to earlier models. The resulting instances (GPT) showed good understanding of natural language syntax and generation of mostly sensible text (which was unprecedented at the time) so they made ChatGPT by adding new stages of supervised fine tuning and RLHF to t…
I miss having the completion models like davinci-003 since it gained in performance where it lacked simplicity to get what you want out.
It was fun to come up with creative ways to get it to answer your question or generate data by setting up a completion scenario.
I guess "chat" became the universal completion scenario. But I still feel like it could be "smarter" without the RLHF layer of distortion.
Re: BitNet: Inference framework for 1-bit LLMs
#146Earlier quoted context omitted.
> Framework is ready. Now we need someone to actually train the model. If Microslop aren't gonna train the model themselves to prove their own thesis, why would others? They've had 2 years (I think?) to prove BitNet in at least some way, are you really saying they haven't tried so far? Personally that makes it slightly worrisome to just take what they say at face value, why wouldn't they train and publish a model the…
So is it finally time for a Beowulf cluster to do something amazing?
Maybe! And it looks like Windows 11 will finally make this the year of the Linux desktop...
Re: BitNet: Inference framework for 1-bit LLMs
#147So Microsoft is actually using 2 bits instead of 1.58. In this case they could represent -1, 0, 1, 2. As inhibitory synapses account for 20%-30%, this could map well to how biological brains are structured. Does that make sense?
Can you explain your third statement? > As inhibitory synapses account for 20%-30%, this could map well to how biological brains are structured.
[deleted]
Re: BitNet: Inference framework for 1-bit LLMs
#148The title is misleading — there's no trained 100B model, just an inference framework that claims to handle one. But the engineering is worth paying attention to. I run quantized 70B models locally (M2 Max 96GB, llama.cpp + LiteLLM), and memory bandwidth is always the bottleneck. The 1.58-bit approach is interesting because ternary weights turn matmuls into additions — a fundamentally different compute profile on comm…
> memory bandwidth is always the bottleneck I'm hoping that today's complaints are tomorrow's innovations. Back when 1Mb hard drive was $100,000, or when Gates said 640kb is enough. Perhaps some 'in the (chip) industry' can comment on what RAM manufacturers are doing at the moment - better, faster, larger? Or is there not much headroom left and it's down to MOBO manufacturers, and volume?
It might be advantageous to have a different memory structure altogether, bespoke to the specific task.
Re: BitNet: Inference framework for 1-bit LLMs
#149Earlier quoted context omitted.
Yeah, "1.58 bit" is 1 trit with three states, since log2(3)≈1.58. So it's not a inference framework for 1-bit models (two states per parameter) but for 1.58 bit models (three states per parameter). Annoying that they try to mix up the two.
I always hope for "just a bunch of if statements" ... this is not it.
Symbolic AI?
I had some AI courses in uni in early 2010s and we were the last cohort that had Prolog and Logic Based AI.
Most interesting project was the final semester where we competed in teams to create the best team of bots for UE3 CTF.
Re: BitNet: Inference framework for 1-bit LLMs
#150One of the things I often wonder is "what will be the minimally viable LLM" that can work from just enough information that if it googles the rest it can provide reasonable answers? I'm surprised something like Encyclopedia Britanica hasn't yet (afaik) tried to capitalize on AI by selling their data to LLMs and validating outputs for LLM companies, it would make a night and day difference in some areas I would think.…
Here's a short clip of Karpathy speaking on this subject.
Also this is the direction the small LLMs are moving in already. They are too small for general knowledge, but getting quite good at tool use (incl. Googling).
Now we just need them to be very strict about what they know and don't know! (I think this is still an open problem, even with big ones.)