I expect the trend of large machine learning models to go towards bits rather than operating on floats. There's a lot of inefficiency in floats because typically they're something like normally distributed, which makes the storage and computation with weights inefficient when most values are clustered in a small range. The foundation of neural networks may be rooted in real valued functions, which are simulated with floats, but float operations are just bitwise operations underneath. The only issue is that GPUs operate on floats and standard ML theory works over real numbers.
Show HN: 1-Bit Bonsai, the First Commercially Viable 1-Bit LLMs
11–20 of 181 posts
Re: Show HN: 1-Bit Bonsai, the First Commercially Viable 1-Bit LLMs
#12How do I run this on Android?
Re: Show HN: 1-Bit Bonsai, the First Commercially Viable 1-Bit LLMs
#13Doesn't Jevons paradox dictate larger 1-bit models?
Re: Show HN: 1-Bit Bonsai, the First Commercially Viable 1-Bit LLMs
#14What’s the trade-off? If it’s smaller, faster and more efficient - is it worse performance? A layman here, curious to know.
Re: Show HN: 1-Bit Bonsai, the First Commercially Viable 1-Bit LLMs
#15What’s the trade-off? If it’s smaller, faster and more efficient - is it worse performance? A layman here, curious to know.
Their own (presumably cherry picked) benchmarks put their models near the 'middle of the market' models (llama3 3b, qwen3 1.7b), not competing with claude, chatgtp, or gemini. These are not models you'd want to directly interact with. but these models can be very useful for things like classification or simple summarization or translation tasks.
These models quite impressive for their size: even an older raspberry pi would be able to handle these.
There's still a lots of use for this kind of model
Re: Show HN: 1-Bit Bonsai, the First Commercially Viable 1-Bit LLMs
#161 bit with a FP16 scale factor every 128 bits. Fascinating that this works so well.
I tried a few things with it. Got it driving Cursor, which in itself was impressive - it handled some tool usage. Via cursor I had it generate a few web page tests.
On a monte carlo simulation of pi, it got the logic correct but failed to build an interface to start the test. Requesting changes mostly worked, but left over some symbols which caused things to fail. Required a bit of manual editing.
Tried a Simon Wilson pelican as well - very abstract, not recognizable at all as a bird or a bicycle.
Pictures of the results here: https://x.com/pwnies/status/2039122871604441213
There doesn't seem to be a demo link on their webpage, so here's a llama.cpp running on my local desktop if people want to try it out. I'll keep this running for a couple hours past this post: https://unfarmable-overaffirmatively-euclid.ngrok-free.dev
Re: Show HN: 1-Bit Bonsai, the First Commercially Viable 1-Bit LLMs
#17Re: Show HN: 1-Bit Bonsai, the First Commercially Viable 1-Bit LLMs
#181 bit with a FP16 scale factor every 128 bits. Fascinating that this works so well. I tried a few things with it. Got it driving Cursor, which in itself was impressive - it handled some tool usage. Via cursor I had it generate a few web page tests. On a monte carlo simulation of pi, it got the logic correct but failed to build an interface to start the test. Requesting changes mostly worked, but left over some symbol…
here's the google colab link, https://colab.research.google.com/drive/1EzyAaQ2nwDv_1X0jaC5... since the ngrok like likely got ddosed by the number of individuals coming along
Re: Show HN: 1-Bit Bonsai, the First Commercially Viable 1-Bit LLMs
#19I feel like it's a little disingenuous to compare against full-precision models. Anyone concerned about model size and memory usage is surely already using at least an 8 bit quantization.
Their main contribution seems to be hyperparameter tuning, and they don't compare against other quantization techniques of any sort.
Re: Show HN: 1-Bit Bonsai, the First Commercially Viable 1-Bit LLMs
#201 bit with a FP16 scale factor every 128 bits. Fascinating that this works so well. I tried a few things with it. Got it driving Cursor, which in itself was impressive - it handled some tool usage. Via cursor I had it generate a few web page tests. On a monte carlo simulation of pi, it got the logic correct but failed to build an interface to start the test. Requesting changes mostly worked, but left over some symbol…
here's the google colab link, https://colab.research.google.com/drive/1EzyAaQ2nwDv_1X0jaC5... since the ngrok like likely got ddosed by the number of individuals coming along
Good call. Right now though traffic is low (1 req per min). With the speed of completion I should be able to handle ~100x that, but if the ngrok link doesn't work defo use the google colab link.