Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: 1-Bit Bonsai, the First Commercially Viable 1-Bit LLMs

prismml.com

Post reply on HN
Print view View original

81–90 of 181 posts

Re: Show HN: 1-Bit Bonsai, the First Commercially Viable 1-Bit LLMs

#81

Post by freakynit »

andai wrote: post #76
jjcm wrote: post #61

Earlier quoted context omitted.

Yep.

Could you elaborate on what you did to get it working? I built it from source, but couldn't get it (the 4B model) to produce coherent English. Sample output below (the model's response to "hi" in the forked llama-cli): X ( Altern as the from (.. Each. ( the or,./, and, can the Altern for few the as ( (. . ( the You theb,’s, Switch, You entire as other, You can the similar is the, can the You other on, and. Altern. .…

I have older M1 air with 8GB, but still getting ober 23 t/s on 4B model.. and the quality of outputs is on par with top models of similar size.

1. Clone their forked repo: `git clone https://github.com/PrismML-Eng/llama.cpp.git`

2. Then (assuming you already have xcode build tools installed):

  cd llama.cpp
  cmake -B build -DGGML_METAL=ON
  cmake --build build --config Release -j$(sysctl -n hw.logicalcpu)
3. Finally, run it with (you can adjust arguments):

  ./build/bin/llama-server -m ~/Downloads/Bonsai-8B.gguf --port 80 --host 0.0.0.0 --ctx-size 0 --parallel 4 --flash-attn on --no-perf --log-colors on --api-key some_api_key_string
Model was first downloaded from: https://huggingface.co/prism-ml/Bonsai-8B-gguf/tree/main

Re: Show HN: 1-Bit Bonsai, the First Commercially Viable 1-Bit LLMs

#82

Post by simonw »

You can run this model on an iPhone via the latest update to this Locally AI app: https://apps.apple.com/us/app/locally-ai-local-ai-chat/id674...

For its size (1.2GB download) it's very impressive.

Here's a pelican it drew me running on my phone - the SVG comments are good, the image not so much: https://tools.simonwillison.net/svg-render#%3Csvg%20width%3D...

Re: Show HN: 1-Bit Bonsai, the First Commercially Viable 1-Bit LLMs

#84

Post by syntaxpr »

MarsIronPI wrote: post #57
bilsbie wrote: post #39

I can’t see how this is possible. You’re losing so much information.

It's because they're natively trained with 1 bit, so it's not losing anything. Now, the question might be how they manage to get decent predictive performance with such little precision. That I don't know.

Not training. Transposing rows/columns of matrices to group 128 parameters with similar (shared) scale factor. Qwen-3 model.

Re: Show HN: 1-Bit Bonsai, the First Commercially Viable 1-Bit LLMs

#86

Post by dodos »

kent8192 wrote: post #53

Oh, boy. This good tool hates my LM Studio... The following message appears when I run Bonsai in my LM Studio. I think my settings have done something wrong. ``` Failed to load the model Error loading model. (Exit code: null). Please check the settings and try loading the model again. ```

Same issue here, wanted to give it a shot but ran into that error trying to load the model in lm studio.

Re: Show HN: 1-Bit Bonsai, the First Commercially Viable 1-Bit LLMs

#87

Post by freakynit »

freakynit wrote: post #81
andai wrote: post #76

Earlier quoted context omitted.

Could you elaborate on what you did to get it working? I built it from source, but couldn't get it (the 4B model) to produce coherent English. Sample output below (the model's response to "hi" in the forked llama-cli): X ( Altern as the from (.. Each. ( the or,./, and, can the Altern for few the as ( (. . ( the You theb,’s, Switch, You entire as other, You can the similar is the, can the You other on, and. Altern. .…

I have older M1 air with 8GB, but still getting ober 23 t/s on 4B model.. and the quality of outputs is on par with top models of similar size. 1. Clone their forked repo: `git clone https://github.com/PrismML-Eng/llama.cpp.git ` 2. Then (assuming you already have xcode build tools installed): cd llama.cpp cmake -B build -DGGML_METAL=ON cmake --build build --config Release -j$(sysctl -n hw.logicalcpu) 3. Finally, run…

To the author: why is this taking 4.56GB ? I was expecting this to be under 1GB for 4B model. https://ibb.co/CprTGZ1c

And this is when Im serving zero prompts.. just loaded the model (using llama-server).

Re: Show HN: 1-Bit Bonsai, the First Commercially Viable 1-Bit LLMs

#88

Post by voxelghost »

simonw wrote: post #82

You can run this model on an iPhone via the latest update to this Locally AI app: https://apps.apple.com/us/app/locally-ai-local-ai-chat/id674... For its size (1.2GB download) it's very impressive. Here's a pelican it drew me running on my phone - the SVG comments are good, the image not so much: https://tools.simonwillison.net/svg-render#%3Csvg%20width%3D...
Did you ask for a pelican with a bicycle, or was that just an added bonus?

Re: Show HN: 1-Bit Bonsai, the First Commercially Viable 1-Bit LLMs

#90

Post by kennywinker »

txrx0000 wrote: post #69
bilsbie wrote: post #39

I can’t see how this is possible. You’re losing so much information.

I always remind myself and everyone else that human DNA is "only" 1.6 GB of data, and yet it encodes all of the complex systems of the human body including the brain, and can replicate itself. Our intuitive feel of how much stuff can be packed into how many bits are probably way off from the true limits of physics.

And anybody who’s ever met a baby can tell you, they score very poorly on most llm benchmarks.
Post reply on HN