Microsoft BitNet: inference framework for 1-bit LLMs • Hacker News Board

Re: Microsoft BitNet: inference framework for 1-bit LLMs

#31

Post by hhejrkrn » Fri, Oct 18, 2024, 11:46 PM UTC

faragon wrote: post #30

layer8 wrote: post #26

Earlier quoted context omitted.

Just install Cygwin. Not sure what you mean by “default shell”. The default shell on Windows is this: https://en.wikipedia.org/wiki/Windows_shell . I don’t suppose you mean booting into Bash. Windows doesn’t have any other notion of a default shell.

I used Cygwin for more than a decade. I prefer Git Bash (msys-based).

I thought msysgit was also cygein based

Re: Microsoft BitNet: inference framework for 1-bit LLMs

#32

Post by EgoIncarnate » Sat, Oct 19, 2024, 1:23 AM UTC

diggan wrote: post #20

throwaway314155 wrote: post #18

Earlier quoted context omitted.

Probably because it has relatively painless GPU sharing with pass through. As far as I know that sort of feature requires a hypervisor-level VM, which is not something you get with VirtualBox.

Someone correct me if I'm wrong, but I think you can use a KVM or QEMU backend for VirtualBox and that way get GPU pass-through. Probably not out of the box though.

The WSL2 GPU passthrough is more like a virtual GPU than KVM style device passthrough. I believe it's effectively a device specific linux userland driver to device specific windows kernel driver with a linux kernel shim bridging the too. If I recall correctly, the linux userland drivers are actually provided by the windows driver.

Re: Microsoft BitNet: inference framework for 1-bit LLMs

#33

Post by ein0p » Sat, Oct 19, 2024, 1:28 AM UTC

1.58bpw is not “1 bit”.

Re: Microsoft BitNet: inference framework for 1-bit LLMs

#34

Post by aithrowawaycomm » Sat, Oct 19, 2024, 7:13 AM UTC

faragon wrote: post #12

I'm glad Microsoft uses Bash in the example, instead of their own Windows shells. As a user I would like having something like "Git Bash" for Windows built in the system, as default shell.

They are using Windows shells, just not on Macs. This is what the caption for the video says:

> A demo of bitnet.cpp running a BitNet b1.58 3B model on Apple M2:

And this is in their Windows build instructions:

> Important! If you are using Windows, please remember to always use a Developer Command Prompt / PowerShell for VS2022 for the following commands

Re: Microsoft BitNet: inference framework for 1-bit LLMs

#35

Post by sheerun » Mon, Oct 21, 2024, 1:16 PM UTC

When will AIs learn to no bla bla bla by default

Re: Microsoft BitNet: inference framework for 1-bit LLMs

#36

Post by sinuhe69 » Wed, Oct 23, 2024, 6:22 AM UTC

ein0p wrote: post #33

1.58bpw is not “1 bit”.

-1,1,0 => log2(3) = 1.58