[dead]
Show HN: 1-Bit Bonsai, the First Commercially Viable 1-Bit LLMs
41–50 of 181 posts
Re: Show HN: 1-Bit Bonsai, the First Commercially Viable 1-Bit LLMs
#421 bit with a FP16 scale factor every 128 bits. Fascinating that this works so well. I tried a few things with it. Got it driving Cursor, which in itself was impressive - it handled some tool usage. Via cursor I had it generate a few web page tests. On a monte carlo simulation of pi, it got the logic correct but failed to build an interface to start the test. Requesting changes mostly worked, but left over some symbol…
I reminds me of very early ChatGPT with mostly correct answers but some nonsense. Given its speed, it might be interesting to run it through a 'thinking' phase where it double checks its answers and/or use search grounding which would make it significantly more useful.
Re: Show HN: 1-Bit Bonsai, the First Commercially Viable 1-Bit LLMs
#43Don't have a GPU so tried the CPU option and got 0.6t/s on my old 2018 laptop using their llama.cpp fork.
Then found out they didn't implement AVX2 for their Q1_0_g128 CPU kernel. Added that and getting ~12t/s which isn't shabby for this old machine.
Cool model.
Re: Show HN: 1-Bit Bonsai, the First Commercially Viable 1-Bit LLMs
#44[flagged]
"Don't post generated comments or AI-edited comments. HN is for conversation between humans."
Re: Show HN: 1-Bit Bonsai, the First Commercially Viable 1-Bit LLMs
#45Don't have a GPU so tried the CPU option and got 0.6t/s on my old 2018 laptop using their llama.cpp fork. Then found out they didn't implement AVX2 for their Q1_0_g128 CPU kernel. Added that and getting ~12t/s which isn't shabby for this old machine. Cool model.
[deleted]
Re: Show HN: 1-Bit Bonsai, the First Commercially Viable 1-Bit LLMs
#46[dead]
Re: Show HN: 1-Bit Bonsai, the First Commercially Viable 1-Bit LLMs
#47Don't have a GPU so tried the CPU option and got 0.6t/s on my old 2018 laptop using their llama.cpp fork. Then found out they didn't implement AVX2 for their Q1_0_g128 CPU kernel. Added that and getting ~12t/s which isn't shabby for this old machine. Cool model.
[deleted]
[deleted]
Re: Show HN: 1-Bit Bonsai, the First Commercially Viable 1-Bit LLMs
#481 bit with a FP16 scale factor every 128 bits. Fascinating that this works so well. I tried a few things with it. Got it driving Cursor, which in itself was impressive - it handled some tool usage. Via cursor I had it generate a few web page tests. On a monte carlo simulation of pi, it got the logic correct but failed to build an interface to start the test. Requesting changes mostly worked, but left over some symbol…
Thanks. Did you need to use Prism's llama.cpp fork to run this?
Re: Show HN: 1-Bit Bonsai, the First Commercially Viable 1-Bit LLMs
#49[dead]
Re: Show HN: 1-Bit Bonsai, the First Commercially Viable 1-Bit LLMs
#50Does anyone know how to run this on CPU?
Do I need to build their llama.cpp fork from source?
Looks like they only offer CUDA options in the release page, which I think might support CPU mode but refuses to even run without CUDA installed. Seems a bit odd to me, I thought the whole point was supporting low end devices!
Edit: 30 minutes of C++ compile time later, I got it running. Although it uses 7GB of RAM then hangs at Loading model. I thought this thing was less memory hungry than 4 bit quants?
Edit 2: Got the 4B version running, but at 0.1 tok/s and the output seemed to be nonsensical. For comparison I can run, on the same machine, qwen 3.5 4B model (at 4 bit quant) correctly and about 50x faster.