Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Microsoft BitNet: inference framework for 1-bit LLMs

github.com

Post reply on HN
Print view View original

21–30 of 36 posts

Re: Microsoft BitNet: inference framework for 1-bit LLMs

#21

Post by danielmarkbruce »

newfocogi wrote: post #3

I'm enthusiastic about BitNet and the potential of low-bit LLMs - the papers show impressive perplexity scores matching full-precision models while drastically reducing compute and memory requirements. What's puzzling is we're not seeing any major providers announce plans to leverage this for their flagship models, despite the clear efficiency gains that could theoretically enable much larger architectures. I suspect…

People are almost certainly working on it. The people who are actually serious and think about things like this are less likely to just spout out "WE ARE BUILDING A CHIP OPTIMIZED FOR 1-BIT" or "WE ARE TRAINING A MODEL USING 1-BIT" etc, before actually being quite sure they can make it work at the required scale. It's still pretty researchy.

Re: Microsoft BitNet: inference framework for 1-bit LLMs

#22

Post by danielmarkbruce »

alkh wrote: post #14

Sorry for a stupid question but to clarify, even though it is a 1-bit model, it is supposed to be working with any types of embeddings, even taken from larger LLMs(in their example, they use HF1BitLLM/Llama3-8B-1.58-100B-tokens). I.e. it doesn't have an embedding layer built-in and relies on embedding provided separately?

No. You can't put any type of embedding in.

Re: Microsoft BitNet: inference framework for 1-bit LLMs

#23

Post by summerlight »

wwwtyro wrote: post #7

Can anyone help me understand how this works without special bitnet precision-specific hardware? Is special hardware unnecessary? Maybe it just doesn't reach the full bitnet potential without it? Or maybe it does, with some fancy tricks? Thanks!

The major benefit would be its significant decrease in memory consumption, rather than the compute itself. The major bottleneck of the current LLM infra is typically memory bandwidth and that's the reason why those chip industries are going crazy on HBM. Surely compute optimization helps but this is useful even without any hardware changes.

Re: Microsoft BitNet: inference framework for 1-bit LLMs

#24

Post by [deleted] »

newfocogi wrote: post #3

I'm enthusiastic about BitNet and the potential of low-bit LLMs - the papers show impressive perplexity scores matching full-precision models while drastically reducing compute and memory requirements. What's puzzling is we're not seeing any major providers announce plans to leverage this for their flagship models, despite the clear efficiency gains that could theoretically enable much larger architectures. I suspect…

[deleted]

Re: Microsoft BitNet: inference framework for 1-bit LLMs

#25

Post by az226 »

summerlight wrote: post #23
wwwtyro wrote: post #7

Can anyone help me understand how this works without special bitnet precision-specific hardware? Is special hardware unnecessary? Maybe it just doesn't reach the full bitnet potential without it? Or maybe it does, with some fancy tricks? Thanks!

The major benefit would be its significant decrease in memory consumption, rather than the compute itself. The major bottleneck of the current LLM infra is typically memory bandwidth and that's the reason why those chip industries are going crazy on HBM. Surely compute optimization helps but this is useful even without any hardware changes.

Inference speeds go brrrr as well.

Re: Microsoft BitNet: inference framework for 1-bit LLMs

#26

Post by layer8 »

faragon wrote: post #12

I'm glad Microsoft uses Bash in the example, instead of their own Windows shells. As a user I would like having something like "Git Bash" for Windows built in the system, as default shell.

Just install Cygwin.

Not sure what you mean by “default shell”. The default shell on Windows is this: https://en.wikipedia.org/wiki/Windows_shell. I don’t suppose you mean booting into Bash. Windows doesn’t have any other notion of a default shell.

Re: Microsoft BitNet: inference framework for 1-bit LLMs

#29

Post by Datagenerator »

diggan wrote: post #20
throwaway314155 wrote: post #18

Earlier quoted context omitted.

Probably because it has relatively painless GPU sharing with pass through. As far as I know that sort of feature requires a hypervisor-level VM, which is not something you get with VirtualBox.

Someone correct me if I'm wrong, but I think you can use a KVM or QEMU backend for VirtualBox and that way get GPU pass-through. Probably not out of the box though.

Close Windows, many doors open

Re: Microsoft BitNet: inference framework for 1-bit LLMs

#30

Post by faragon »

layer8 wrote: post #26
faragon wrote: post #12

I'm glad Microsoft uses Bash in the example, instead of their own Windows shells. As a user I would like having something like "Git Bash" for Windows built in the system, as default shell.

Just install Cygwin. Not sure what you mean by “default shell”. The default shell on Windows is this: https://en.wikipedia.org/wiki/Windows_shell . I don’t suppose you mean booting into Bash. Windows doesn’t have any other notion of a default shell.

I used Cygwin for more than a decade. I prefer Git Bash (msys-based).
Post reply on HN