Open access for next 5 hours (Ternary-Bonsai-8B-Q2_0.gguf, running on RTX 3090) or until server crashes or the this spot instance gets taken away :) => https://uklkyvetsjf7qt-80.proxy.runpod.net ./build/bin/llama-server \ -m ../Ternary-Bonsai-8B-Q2_0.gguf \ -ngl 999 \ --flash-attn on \ --host 0.0.0.0 \ --port 80 \ --ctx-size 65500 \ --batch-size 512 \ --ubatch-size 512 \ --parallel 5 \ --cont-batching \ --threads 8 \…
Ternary Bonsai: Top Intelligence at 1.58 Bits
41–50 of 60 posts
Re: Ternary Bonsai: Top Intelligence at 1.58 Bits
#42On my single NVidia Spark I get 173.3 tokens/s on baseline config, 372.4 tokens/s with added tuning/parallel options. Most notably time to first token is incredibly low, similar models take ~6000ms. Bonsai was 70ms (almost 100x reduction) with flash attention
Having said all that, gemma4-e4b-q4km did much better and I can achieve 70% of the tokens/s on the same machine, specifically in context of tool use and for running agents.
Re: Ternary Bonsai: Top Intelligence at 1.58 Bits
#43Earlier quoted context omitted.
The 1-bit Bonsai and Ternary Bonsai models are all based on the corresponding Qwen3 model: https://raw.githubusercontent.com/PrismML-Eng/Bonsai-demo/re... (page 4)
Thanks, already suspected as much. Also gives context to the other comment here that says it is basically equivalent in accuracy to Qwen3.5-4B. Essentially seems to be a very good quantization of that model, not a new BitNet.
Re: Ternary Bonsai: Top Intelligence at 1.58 Bits
#44Would be interesting to see a comparison to quantized versions of the other models. If this model beats the others also in a fair comparison it gives more credibility to it.
Re: Ternary Bonsai: Top Intelligence at 1.58 Bits
#45Open access for next 5 hours (Ternary-Bonsai-8B-Q2_0.gguf, running on RTX 3090) or until server crashes or the this spot instance gets taken away :) => https://uklkyvetsjf7qt-80.proxy.runpod.net ./build/bin/llama-server \ -m ../Ternary-Bonsai-8B-Q2_0.gguf \ -ngl 999 \ --flash-attn on \ --host 0.0.0.0 \ --port 80 \ --ctx-size 65500 \ --batch-size 512 \ --ubatch-size 512 \ --parallel 5 \ --cont-batching \ --threads 8 \…
Fyi, I believe `--flash-attn on` doesn't do anything, you should instead use `--flash-attn 1`. I'm getting ~150t/s on a RTX 3080 10GB as well with f16 cache type.
Re: Ternary Bonsai: Top Intelligence at 1.58 Bits
#46Open access for next 5 hours (Ternary-Bonsai-8B-Q2_0.gguf, running on RTX 3090) or until server crashes or the this spot instance gets taken away :) => https://uklkyvetsjf7qt-80.proxy.runpod.net ./build/bin/llama-server \ -m ../Ternary-Bonsai-8B-Q2_0.gguf \ -ngl 999 \ --flash-attn on \ --host 0.0.0.0 \ --port 80 \ --ctx-size 65500 \ --batch-size 512 \ --ubatch-size 512 \ --parallel 5 \ --cont-batching \ --threads 8 \…
Re: Ternary Bonsai: Top Intelligence at 1.58 Bits
#47I don't see any mWh/token figures in that chart.
Re: Ternary Bonsai: Top Intelligence at 1.58 Bits
#48How is the research on training these models directly in their quantized state going? That'll be the real game changer.
https://proceedings.neurips.cc/paper_files/paper/2024/hash/7...
They train directly in the 1 bit domain, without any floating point weights. They don't use the classical Newton-Leibniz derivative (which operates on approximations of real numbers) for gradient descent / backpropagation. Instead they invented a binary version called "Boolean variation".
I don't know why this paper didn't get more attention.
Re: Ternary Bonsai: Top Intelligence at 1.58 Bits
#49How is the research on training these models directly in their quantized state going? That'll be the real game changer.
The original BitNet was natively trained on 1.58 bits. PrismML has not released any actual info on how they trained, but since they are based on Qwen, there was certainly some downstream quantization involved.
Unfortunately my mental model doesn't contain anything to even guess if that's possible or not, my AI times were at the falling flank of symbolic. Funny how one bit models feel a bit like approaching an approximation of symbolic again (until you read about the grouped scale factors and then the illusion is gone)
One thought that suggests rearranging is not involved,a thought that does not require any knowledge at all: if it did involve rearranging, someone would certainly have added some order by scale factor tricks with linear interpolation by address offset to lose even less precision.
Re: Ternary Bonsai: Top Intelligence at 1.58 Bits
#50Open access for next 5 hours (Ternary-Bonsai-8B-Q2_0.gguf, running on RTX 3090) or until server crashes or the this spot instance gets taken away :) => https://uklkyvetsjf7qt-80.proxy.runpod.net ./build/bin/llama-server \ -m ../Ternary-Bonsai-8B-Q2_0.gguf \ -ngl 999 \ --flash-attn on \ --host 0.0.0.0 \ --port 80 \ --ctx-size 65500 \ --batch-size 512 \ --ubatch-size 512 \ --parallel 5 \ --cont-batching \ --threads 8 \…
Thanks a lot, I was about to clone their llama.cpp branch and do the same. Some more interesting tidbits from my go-to tests: * Fails the car wash test (basic logic seems to be weak in general) * Fails simple watch face generation in html/css. * Fails the "how many Rs in raspberry test" (not enough cross-token training data), but will funnily assume you may be talking about Indian Rupees and tell you a lot about rasp…
Interestingly if you ask it "who is the dalai lama" it outputs an answer that certainly wouldn't be acceptable to post online in mainland china. It says:
The Dalai Lama is the spiritual and political leader of Tibet, and the title is held by the incarnate lama who is believed to be a reincarnation of the previous Dalai Lama. The Dalai Lama is a central figure in Tibetan Buddhism and holds a unique position in Tibetan society, both as a religious leader and as a symbol of Tibetan identity and independence. Key Points About the Dalai Lama: 1. Current Dalai Lama: Tenzin Gyatso (14th Dalai Lama)
Born: 1935 in Amdo, Tibet (now part of China).
Education: He studied at Ganden Monastery and later at Dunhuang University in China.
Title: He is known as the 14th Dalai Lama, a title that has been passed down through generations of lamas who are believed to be reincarnations.
Role: He is the spiritual leader of Tibetan Buddhism and the de facto leader of the Tibetan diaspora, particularly in India.
2. Historical Context The title of Dalai Lama has been held by Tibetan Buddhist lamas for centuries. It means "Ocean of Wisdom" in Tibetan.
The 13th Dalai Lama (1876–1959) was a key figure in Tibetan history, known for his efforts to maintain Tibetan independence and promote modernization.
He fled to India in 1950 after the Chinese Communist Party (CCP) took control of Tibet in 1951.
3. 14th Dalai Lama: Exile and Leadership The 14th Dalai Lama, Lama Tenzin Gyatso, was exiled to India in 1959 after the Tibetan Uprising.
He has led the Tibetan government-in-exile in India since then, advocating for Tibetan independence and cultural preservation.
He has also been a prominent voice for human rights and peace.
4. Political Status The 14th Dalai Lama is not recognized by the Chinese government as the legitimate ruler of Tibet, which is part of the People's Republic of China (PRC).
The Central Tibetan Administration (CTA), based in India, is the self-proclaimed government of Tibet.
The PRC considers Tibet to be an inseparable part of China and has implemented policies to integrate Tibet into the Chinese state.
5. International Recognition The 14th Dalai Lama is widely recognized internationally as the spiritual leader of Tibet and as the de facto leader of the Tibetan people.
Many countries and international organizations support the Tibetan cause, but they are not recognized as a sovereign state.
Summary:The Dalai Lama is the spiritual and political leader of Tibet, currently Lama Tenzin Gyatso (14th Dalai Lama), who has been in exile in India since 1959. He is a central figure in Tibetan identity and has played a key role in the global Tibetan movement for independence and cultural preservation.
If you'd like to know more about his background, policies, or role in the Tibetan independence movement, feel free to ask!