Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ternary Bonsai: Top Intelligence at 1.58 Bits

prismml.com

Post reply on HN
Print view View original

31–40 of 60 posts

Re: Ternary Bonsai: Top Intelligence at 1.58 Bits

#31

Post by sigmoid10 »

freakynit wrote: post #15

Open access for next 5 hours (Ternary-Bonsai-8B-Q2_0.gguf, running on RTX 3090) or until server crashes or the this spot instance gets taken away :) => https://uklkyvetsjf7qt-80.proxy.runpod.net ./build/bin/llama-server \ -m ../Ternary-Bonsai-8B-Q2_0.gguf \ -ngl 999 \ --flash-attn on \ --host 0.0.0.0 \ --port 80 \ --ctx-size 65500 \ --batch-size 512 \ --ubatch-size 512 \ --parallel 5 \ --cont-batching \ --threads 8 \…

Thanks a lot, I was about to clone their llama.cpp branch and do the same.

Some more interesting tidbits from my go-to tests:

* Fails the car wash test (basic logic seems to be weak in general)

* Fails simple watch face generation in html/css.

* Fails the "how many Rs in raspberry test" (not enough cross-token training data), but will funnily assume you may be talking about Indian Rupees and tell you a lot about raspberry prices in India without being asked. Possible Indian training data unbalance?

* Flat out refuses to talk about Tiananmen square when pushed directly - despite being from a US company. Again, perhaps they are exposed to some censored training data? Anyways, when slowly built up along the conversation by asking about locations and histories, it will eventually tell you about the massacre, so the censorship bias seems weak in general. Also has no problem immediately talking about anything Gaza/Israel/US or other sensitive topics.

* Happily tells you how to synthesize RDX with list of ingredients and chemical process step by step. At least it warns you that it is highly dangerous and legally controlled in the US.

Re: Ternary Bonsai: Top Intelligence at 1.58 Bits

#32

Post by londons_explore »

freakynit wrote: post #18
usernametaken29 wrote: post #17

I think it’s exciting to live in this quirky universe where we have simply accepted our hardware does weird and nonlinear stuff and that powers some math and that’s why your transform function works. Many people thought quantisation is not viable to the extent we see, but we clearly underestimated the effect of hardware on the actual non linearity of the models. Cool to see this pushed to the limits.

Nature has already set an absurdly high bar. The human brain runs on roughly 20 watts, yet delivers a level of intelligence we still can't clearly define, let alone replicate. Nothing we've built comes close... either in capability or efficiency. We're still very early in understanding what "intelligence" even means, much less engineering it. so, we have a long way to go, and push.

A 1980's desk calculator can multiply two 8 digit numbers with much less energy than your brain takes to do the same.

Modern LLM's similarly beat the human brain in lots of tasks for energy efficiency - mostly by the fact the LLM can produce the answer in 1 second and the brain has to spend half an hour researching and drafting something.

Re: Ternary Bonsai: Top Intelligence at 1.58 Bits

#33

Post by londons_explore »

Animats wrote: post #7
yodon wrote: post #4

So excited to see this - the big advantage of 1.58 bits is there are no multiplications at inference time, so you can run them on radically simpler and cheaper hardware.

At 4 bits, you could just have a hard-wired table lookup. Two 4 bit values in, 256 entry table. You can have saturating arithmetic and a post-processing function for free. Somebody must be building hardware like that.

Low level software engineers use look up tables.

Hardware engineers realise that a compiler will almost always find some combination of gates which is smaller/faster than the contents of any table.

Re: Ternary Bonsai: Top Intelligence at 1.58 Bits

#34

Post by opem »

freakynit wrote: post #15

Open access for next 5 hours (Ternary-Bonsai-8B-Q2_0.gguf, running on RTX 3090) or until server crashes or the this spot instance gets taken away :) => https://uklkyvetsjf7qt-80.proxy.runpod.net ./build/bin/llama-server \ -m ../Ternary-Bonsai-8B-Q2_0.gguf \ -ngl 999 \ --flash-attn on \ --host 0.0.0.0 \ --port 80 \ --ctx-size 65500 \ --batch-size 512 \ --ubatch-size 512 \ --parallel 5 \ --cont-batching \ --threads 8 \…

played around with it a little, works great for - basic conversational tasks - basic reasoning - creating little UI snippets and not good for one shot coding tasks. once it starts hallucinating about something you can't make it rectify that. during a coding task it wrote some madeup library function, despite of pointing it out, giving docs, it kept producing the same code with the madeup function all while acknowledging the problem. btw, thanks for putting it together!

Re: Ternary Bonsai: Top Intelligence at 1.58 Bits

#36

Post by [deleted] »

sigmoid10 wrote: post #31
freakynit wrote: post #15

Open access for next 5 hours (Ternary-Bonsai-8B-Q2_0.gguf, running on RTX 3090) or until server crashes or the this spot instance gets taken away :) => https://uklkyvetsjf7qt-80.proxy.runpod.net ./build/bin/llama-server \ -m ../Ternary-Bonsai-8B-Q2_0.gguf \ -ngl 999 \ --flash-attn on \ --host 0.0.0.0 \ --port 80 \ --ctx-size 65500 \ --batch-size 512 \ --ubatch-size 512 \ --parallel 5 \ --cont-batching \ --threads 8 \…

Thanks a lot, I was about to clone their llama.cpp branch and do the same. Some more interesting tidbits from my go-to tests: * Fails the car wash test (basic logic seems to be weak in general) * Fails simple watch face generation in html/css. * Fails the "how many Rs in raspberry test" (not enough cross-token training data), but will funnily assume you may be talking about Indian Rupees and tell you a lot about rasp…

[deleted]

Re: Ternary Bonsai: Top Intelligence at 1.58 Bits

#38

Post by yorwba »

sigmoid10 wrote: post #31
freakynit wrote: post #15

Open access for next 5 hours (Ternary-Bonsai-8B-Q2_0.gguf, running on RTX 3090) or until server crashes or the this spot instance gets taken away :) => https://uklkyvetsjf7qt-80.proxy.runpod.net ./build/bin/llama-server \ -m ../Ternary-Bonsai-8B-Q2_0.gguf \ -ngl 999 \ --flash-attn on \ --host 0.0.0.0 \ --port 80 \ --ctx-size 65500 \ --batch-size 512 \ --ubatch-size 512 \ --parallel 5 \ --cont-batching \ --threads 8 \…

Thanks a lot, I was about to clone their llama.cpp branch and do the same. Some more interesting tidbits from my go-to tests: * Fails the car wash test (basic logic seems to be weak in general) * Fails simple watch face generation in html/css. * Fails the "how many Rs in raspberry test" (not enough cross-token training data), but will funnily assume you may be talking about Indian Rupees and tell you a lot about rasp…

The 1-bit Bonsai and Ternary Bonsai models are all based on the corresponding Qwen3 model: https://raw.githubusercontent.com/PrismML-Eng/Bonsai-demo/re... (page 4)

Re: Ternary Bonsai: Top Intelligence at 1.58 Bits

#39

Post by sigmoid10 »

londons_explore wrote: post #35

How is the research on training these models directly in their quantized state going? That'll be the real game changer.

The original BitNet was natively trained on 1.58 bits. PrismML has not released any actual info on how they trained, but since they are based on Qwen, there was certainly some downstream quantization involved.

Re: Ternary Bonsai: Top Intelligence at 1.58 Bits

#40

Post by sigmoid10 »

yorwba wrote: post #38
sigmoid10 wrote: post #31

Earlier quoted context omitted.

Thanks a lot, I was about to clone their llama.cpp branch and do the same. Some more interesting tidbits from my go-to tests: * Fails the car wash test (basic logic seems to be weak in general) * Fails simple watch face generation in html/css. * Fails the "how many Rs in raspberry test" (not enough cross-token training data), but will funnily assume you may be talking about Indian Rupees and tell you a lot about rasp…

The 1-bit Bonsai and Ternary Bonsai models are all based on the corresponding Qwen3 model: https://raw.githubusercontent.com/PrismML-Eng/Bonsai-demo/re... (page 4)

Thanks, already suspected as much. Also gives context to the other comment here that says it is basically equivalent in accuracy to Qwen3.5-4B. Essentially seems to be a very good quantization of that model, not a new BitNet.
Post reply on HN