Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Microsoft BitNet: inference framework for 1-bit LLMs

github.com

Post reply on HN
Print view View original

11–20 of 36 posts

Re: Microsoft BitNet: inference framework for 1-bit LLMs

#11

Post by eightysixfour »

wwwtyro wrote: post #7

Can anyone help me understand how this works without special bitnet precision-specific hardware? Is special hardware unnecessary? Maybe it just doesn't reach the full bitnet potential without it? Or maybe it does, with some fancy tricks? Thanks!

While fancy hardware would make it faster, what you are comparing it to is a bunch of floating point and large number multiplication. I believe in this case they just use a look up table:

If one value is 0, it is 0.

If the signs are different, it is -1.

If the signs are the same, it is 1.

I’m sure those can be done with relatively few instructions using far less power hungry hardware.

Re: Microsoft BitNet: inference framework for 1-bit LLMs

#13

Post by waynenilsen »

newfocogi wrote: post #3

I'm enthusiastic about BitNet and the potential of low-bit LLMs - the papers show impressive perplexity scores matching full-precision models while drastically reducing compute and memory requirements. What's puzzling is we're not seeing any major providers announce plans to leverage this for their flagship models, despite the clear efficiency gains that could theoretically enable much larger architectures. I suspect…

I suppose hardware support would be very helpful, new instructions for bitpacked operations?

Re: Microsoft BitNet: inference framework for 1-bit LLMs

#14

Post by alkh »

Sorry for a stupid question but to clarify, even though it is a 1-bit model, it is supposed to be working with any types of embeddings, even taken from larger LLMs(in their example, they use HF1BitLLM/Llama3-8B-1.58-100B-tokens). I.e. it doesn't have an embedding layer built-in and relies on embedding provided separately?

Re: Microsoft BitNet: inference framework for 1-bit LLMs

#15

Post by not_a_bot_4sho »

faragon wrote: post #12

I'm glad Microsoft uses Bash in the example, instead of their own Windows shells. As a user I would like having something like "Git Bash" for Windows built in the system, as default shell.

WSL is where it's at today. It's not quite what you're asking for, as it is a separate virtual OS, but the integration is so tight that it feels like you're using your favorite shell natively in Windows.

Re: Microsoft BitNet: inference framework for 1-bit LLMs

#16

Post by diggan »

not_a_bot_4sho wrote: post #15
faragon wrote: post #12

I'm glad Microsoft uses Bash in the example, instead of their own Windows shells. As a user I would like having something like "Git Bash" for Windows built in the system, as default shell.

WSL is where it's at today. It's not quite what you're asking for, as it is a separate virtual OS, but the integration is so tight that it feels like you're using your favorite shell natively in Windows.

> integration is so tight that it feels like you're using your favorite shell natively in Windows

WSL1 certainly felt that way, WSL2 just feels like any other virtualization manager and basically works the same. Not sure why people sings the praise of WSL2, I gave it a serious try for months but there is a seemingly endless list of compatibility issues which I never had with VMWare or VirtualBox, so I just went back to those instead and the experience is the same more or less.

Re: Microsoft BitNet: inference framework for 1-bit LLMs

#18

Post by throwaway314155 »

diggan wrote: post #16
not_a_bot_4sho wrote: post #15

Earlier quoted context omitted.

WSL is where it's at today. It's not quite what you're asking for, as it is a separate virtual OS, but the integration is so tight that it feels like you're using your favorite shell natively in Windows.

> integration is so tight that it feels like you're using your favorite shell natively in Windows WSL1 certainly felt that way, WSL2 just feels like any other virtualization manager and basically works the same. Not sure why people sings the praise of WSL2, I gave it a serious try for months but there is a seemingly endless list of compatibility issues which I never had with VMWare or VirtualBox, so I just went back…

Probably because it has relatively painless GPU sharing with pass through. As far as I know that sort of feature requires a hypervisor-level VM, which is not something you get with VirtualBox.

Re: Microsoft BitNet: inference framework for 1-bit LLMs

#19

Post by varelse »

newfocogi wrote: post #3

I'm enthusiastic about BitNet and the potential of low-bit LLMs - the papers show impressive perplexity scores matching full-precision models while drastically reducing compute and memory requirements. What's puzzling is we're not seeing any major providers announce plans to leverage this for their flagship models, despite the clear efficiency gains that could theoretically enable much larger architectures. I suspect…

[dead]

Re: Microsoft BitNet: inference framework for 1-bit LLMs

#20

Post by diggan »

throwaway314155 wrote: post #18
diggan wrote: post #16

Earlier quoted context omitted.

> integration is so tight that it feels like you're using your favorite shell natively in Windows WSL1 certainly felt that way, WSL2 just feels like any other virtualization manager and basically works the same. Not sure why people sings the praise of WSL2, I gave it a serious try for months but there is a seemingly endless list of compatibility issues which I never had with VMWare or VirtualBox, so I just went back…

Probably because it has relatively painless GPU sharing with pass through. As far as I know that sort of feature requires a hypervisor-level VM, which is not something you get with VirtualBox.

Someone correct me if I'm wrong, but I think you can use a KVM or QEMU backend for VirtualBox and that way get GPU pass-through. Probably not out of the box though.
Post reply on HN