Open access for next 5 hours (Ternary-Bonsai-8B-Q2_0.gguf, running on RTX 3090) or until server crashes or the this spot instance gets taken away :) => https://uklkyvetsjf7qt-80.proxy.runpod.net ./build/bin/llama-server \ -m ../Ternary-Bonsai-8B-Q2_0.gguf \ -ngl 999 \ --flash-attn on \ --host 0.0.0.0 \ --port 80 \ --ctx-size 65500 \ --batch-size 512 \ --ubatch-size 512 \ --parallel 5 \ --cont-batching \ --threads 8 \…
update: Well, spot survived... and since a lot of the folks are still using it, I'm keeping it alive for 2 hours more.
I think it’s exciting to live in this quirky universe where we have simply accepted our hardware does weird and nonlinear stuff and that powers some math and that’s why your transform function works. Many people thought quantisation is not viable to the extent we see, but we clearly underestimated the effect of hardware on the actual non linearity of the models. Cool to see this pushed to the limits.
Nature has already set an absurdly high bar. The human brain runs on roughly 20 watts, yet delivers a level of intelligence we still can't clearly define, let alone replicate. Nothing we've built comes close... either in capability or efficiency. We're still very early in understanding what "intelligence" even means, much less engineering it. so, we have a long way to go, and push.
Not sure what you mean by efficiency as this was part of the article and I understand things differently—can you clarify? For the energy of 20 W in an hour on a laptop’s M4 pro, this model produces about 200k tokens (a book or two) at a typical electricity cost of less than a third of a US cent. Although clearly the intelligence of this particular model is unrelated to human intelligence, I always thought that there is no comparison between LLMs and humans in terms of efficiency: these models are way less energy expensive than humans. If you were to use data center scale optimizations, then serving LLMs is many additional orders of magnitude more efficient than serving LLMs at home. (The energy cost of inference on the M4 pro and iphone are listed in the article.)
Open access for next 5 hours (Ternary-Bonsai-8B-Q2_0.gguf, running on RTX 3090) or until server crashes or the this spot instance gets taken away :) => https://uklkyvetsjf7qt-80.proxy.runpod.net ./build/bin/llama-server \ -m ../Ternary-Bonsai-8B-Q2_0.gguf \ -ngl 999 \ --flash-attn on \ --host 0.0.0.0 \ --port 80 \ --ctx-size 65500 \ --batch-size 512 \ --ubatch-size 512 \ --parallel 5 \ --cont-batching \ --threads 8 \…
Thanks a lot, I was about to clone their llama.cpp branch and do the same. Some more interesting tidbits from my go-to tests: * Fails the car wash test (basic logic seems to be weak in general) * Fails simple watch face generation in html/css. * Fails the "how many Rs in raspberry test" (not enough cross-token training data), but will funnily assume you may be talking about Indian Rupees and tell you a lot about rasp…
I think the future of economically useful AI is to build efficient reasoners. A goal of model as oracle of truth or encylopedia for our world is orthagonal to a goal of a model that can reason about novel input. I think the focus on meme litmus tests is somewhat misguided. AI is not suited to be a spell checker, a news source, a history book, or an idependent developer. but if it can reasom about a prompt and augment human effort then that is useful. the idea of an ai without human like world model matching or exceeding our own world model or store of facts is misguided in my opinion.
Open access for next 5 hours (Ternary-Bonsai-8B-Q2_0.gguf, running on RTX 3090) or until server crashes or the this spot instance gets taken away :) => https://uklkyvetsjf7qt-80.proxy.runpod.net ./build/bin/llama-server \ -m ../Ternary-Bonsai-8B-Q2_0.gguf \ -ngl 999 \ --flash-attn on \ --host 0.0.0.0 \ --port 80 \ --ctx-size 65500 \ --batch-size 512 \ --ubatch-size 512 \ --parallel 5 \ --cont-batching \ --threads 8 \…
Thanks a lot, I was about to clone their llama.cpp branch and do the same. Some more interesting tidbits from my go-to tests: * Fails the car wash test (basic logic seems to be weak in general) * Fails simple watch face generation in html/css. * Fails the "how many Rs in raspberry test" (not enough cross-token training data), but will funnily assume you may be talking about Indian Rupees and tell you a lot about rasp…
I think the future of economically useful AI is to build efficient reasoners. A goal of model as oracle of truth or encylopedia for our world is orthagonal to a goal of a model that can reason about novel input. I think the focus on meme litmus tests is somewhat misguided. AI is not suited to be a spell checker, a news source, a history book, or an idependent developer. but if it can reason about a prompt and augment human effort then that is useful. the idea of an ai without human like world model matching or exceeding our own world model or store of facts is misguided in my opinion.
Thanks a lot, I was about to clone their llama.cpp branch and do the same. Some more interesting tidbits from my go-to tests: * Fails the car wash test (basic logic seems to be weak in general) * Fails simple watch face generation in html/css. * Fails the "how many Rs in raspberry test" (not enough cross-token training data), but will funnily assume you may be talking about Indian Rupees and tell you a lot about rasp…
I think the future of economically useful AI is to build efficient reasoners. A goal of model as oracle of truth or encylopedia for our world is orthagonal to a goal of a model that can reason about novel input. I think the focus on meme litmus tests is somewhat misguided. AI is not suited to be a spell checker, a news source, a history book, or an idependent developer. but if it can reasom about a prompt and augment…
The future of AI I would imagine is that AI acts purely as logic within constraints
The original BitNet was natively trained on 1.58 bits. PrismML has not released any actual info on how they trained, but since they are based on Qwen, there was certainly some downstream quantization involved.
Is it just quantization or is it also rearranging the weights to get clusters with (almost) the same factors? If it's the latter it would very much be training in full precision (but also hardly any precision lost by the compression). Unfortunately my mental model doesn't contain anything to even guess if that's possible or not, my AI times were at the falling flank of symbolic. Funny how one bit models feel a bit li…
It's not just quantization. I verified that if you naïvely quantize to 1 bit from the original Qwen model (and set grouped scale factors based on what the original model's weights were like), it just spits out gibberish.
> One thought that suggests rearranging is not involved,a thought that does not require any knowledge at all: if it did involve rearranging, someone would certainly have added some order by scale factor tricks with linear interpolation by address offset to lose even less precision.
Nature has already set an absurdly high bar. The human brain runs on roughly 20 watts, yet delivers a level of intelligence we still can't clearly define, let alone replicate. Nothing we've built comes close... either in capability or efficiency. We're still very early in understanding what "intelligence" even means, much less engineering it. so, we have a long way to go, and push.
Depending on how you convert synapse count to parameters, the brain also has something like a thousand trillion parameters. In that light it's pretty darn surprising that an artificial neural network can produce anything like coherent text.
Biological brains need to run a self reproducing machine. Still alien tech we are hoping to one day understand.
Nature has already set an absurdly high bar. The human brain runs on roughly 20 watts, yet delivers a level of intelligence we still can't clearly define, let alone replicate. Nothing we've built comes close... either in capability or efficiency. We're still very early in understanding what "intelligence" even means, much less engineering it. so, we have a long way to go, and push.
A 1980's desk calculator can multiply two 8 digit numbers with much less energy than your brain takes to do the same. Modern LLM's similarly beat the human brain in lots of tasks for energy efficiency - mostly by the fact the LLM can produce the answer in 1 second and the brain has to spend half an hour researching and drafting something.
A little clarification on my original comment: "intelligence/watt", not "work/watt", is the metric I'm talking about.