Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: 1-Bit Bonsai, the First Commercially Viable 1-Bit LLMs

prismml.com

Post reply on HN
Print view View original

71–80 of 181 posts

Re: Show HN: 1-Bit Bonsai, the First Commercially Viable 1-Bit LLMs

#71

Post by jjcm »

abrookewood wrote: post #65
jjcm wrote: post #16

1 bit with a FP16 scale factor every 128 bits. Fascinating that this works so well. I tried a few things with it. Got it driving Cursor, which in itself was impressive - it handled some tool usage. Via cursor I had it generate a few web page tests. On a monte carlo simulation of pi, it got the logic correct but failed to build an interface to start the test. Requesting changes mostly worked, but left over some symbol…

man, that is really really quick. What is your desktop setup??? GPU?

It is fast, but I do have good hardware. A few people have asked for my local inference build, so I have an existing guide that mirrors my setup: https://non.io/Local-inference-build

Re: Show HN: 1-Bit Bonsai, the First Commercially Viable 1-Bit LLMs

#72

Post by cubefox »

drob518 wrote: post #56

I’m really curious how this scales up. Bonsai delivers an 8B model in 1.15 GB. How large would a 27B or 35B model be? Would it still retain the accuracy of those large models? If the scaling holds, we could see 100+B models in 64 GB of RAM.

Also depends on how expensive training these models is. It's probably at least as expensive as full precision models, otherwise they would have mentioned it.

Re: Show HN: 1-Bit Bonsai, the First Commercially Viable 1-Bit LLMs

#73

Post by cubefox »

alyxya wrote: post #11

I expect the trend of large machine learning models to go towards bits rather than operating on floats. There's a lot of inefficiency in floats because typically they're something like normally distributed, which makes the storage and computation with weights inefficient when most values are clustered in a small range. The foundation of neural networks may be rooted in real valued functions, which are simulated with…

> and standard ML theory works over real numbers.

This paper uses binary numbers only, even for training, with a solid theoretical foundation: https://proceedings.neurips.cc/paper_files/paper/2024/file/7...

TL;DR: They invent a concept called "Boolean variation" which is the binary analog to the Newton/Leibniz derivative. They are then able to do backpropagation directly in binary.

Re: Show HN: 1-Bit Bonsai, the First Commercially Viable 1-Bit LLMs

#74

Post by ddtaylor »

cubefox wrote: post #64
wild_egg wrote: post #43

Don't have a GPU so tried the CPU option and got 0.6t/s on my old 2018 laptop using their llama.cpp fork. Then found out they didn't implement AVX2 for their Q1_0_g128 CPU kernel. Added that and getting ~12t/s which isn't shabby for this old machine. Cool model.

"Not shabby" is a big understatement.

Why so?

Re: Show HN: 1-Bit Bonsai, the First Commercially Viable 1-Bit LLMs

#76

Post by andai »

jjcm wrote: post #61
andai wrote: post #48

Earlier quoted context omitted.

Thanks. Did you need to use Prism's llama.cpp fork to run this?

Yep.

Could you elaborate on what you did to get it working? I built it from source, but couldn't get it (the 4B model) to produce coherent English.

Sample output below (the model's response to "hi" in the forked llama-cli):

X ( Altern as the from (.. Each. ( the or,./, and, can the Altern for few the as ( (. . ( the You theb,’s, Switch, You entire as other, You can the similar is the, can the You other on, and. Altern. . That, on, and similar, and, similar,, and, or in

Re: Show HN: 1-Bit Bonsai, the First Commercially Viable 1-Bit LLMs

#77

Post by andai »

The site says 14x less memory usage. I'm a bit confused about that situation. The model file is indeed very small, but on my machine it used roughly the same RAM as 4 bit quants (on CPU).

Though I couldn't get actual English output from it, so maybe something went wrong while running it.

Re: Show HN: 1-Bit Bonsai, the First Commercially Viable 1-Bit LLMs

#78

Post by jjcm »

andai wrote: post #76
jjcm wrote: post #61

Earlier quoted context omitted.

Yep.

Could you elaborate on what you did to get it working? I built it from source, but couldn't get it (the 4B model) to produce coherent English. Sample output below (the model's response to "hi" in the forked llama-cli): X ( Altern as the from (.. Each. ( the or,./, and, can the Altern for few the as ( (. . ( the You theb,’s, Switch, You entire as other, You can the similar is the, can the You other on, and. Altern. .…

I did this: https://image.non.io/2093de83-97f6-43e1-a95e-3667b6d89b3f.we...

Literally just downloaded the model into a folder, opened cursor in that folder, and told it to get it running.

Prompt: The gguf for bonsai 8b are in this local project. Get it up and running so I can chat with it. I don't care through what interface. Just get things going quickly. Run it locally - I have plenty of vram. https://huggingface.co/prism-ml/Bonsai-8B-gguf/tree/main

I had to ask it to increase the context window size to 64k, but other than that it got it running just fine. After that I just told ngrok the port I was serving it on and voila.

Re: Show HN: 1-Bit Bonsai, the First Commercially Viable 1-Bit LLMs

#80

Post by selcuka »

pdyc wrote: post #62
jjcm wrote: post #16

1 bit with a FP16 scale factor every 128 bits. Fascinating that this works so well. I tried a few things with it. Got it driving Cursor, which in itself was impressive - it handled some tool usage. Via cursor I had it generate a few web page tests. On a monte carlo simulation of pi, it got the logic correct but failed to build an interface to start the test. Requesting changes mostly worked, but left over some symbol…

thanks, i tested it, failed in strawberry test. qwen 3.5 0.8B with similar size passes it and is far more usable.

Interesting. Qwen 3.5 0.8B failed the test for me.
Post reply on HN