Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: 1-Bit Bonsai, the First Commercially Viable 1-Bit LLMs

prismml.com

Post reply on HN
Print view View original

101–110 of 181 posts

Re: Show HN: 1-Bit Bonsai, the First Commercially Viable 1-Bit LLMs

#101

Post by freakynit »

ggerganov wrote: post #98
freakynit wrote: post #93

Open access for next 5 hours (8GiB model, running on RTX 3090) or until server crashes or the this spot instance gets taken away :) => https://ofo1j9j6qh20a8-80.proxy.runpod.net ./build/bin/llama-server \ -m ../Bonsai-8B.gguf \ -ngl 999 \ --flash-attn on \ --host 0.0.0.0 \ --port 80 \ --ctx-size 65500 \ --batch-size 512 \ --ubatch-size 512 \ --parallel 5 \ --cont-batching \ --threads 8 \ --threads-batch 8 \ --cache-t…

Better keep the KV cache in full precision

Wow.. the GOAT himself.. thank you sooo much for creating llama.cpp ... will re-deploy with full kv cache once requests stop coming.

Re: Show HN: 1-Bit Bonsai, the First Commercially Viable 1-Bit LLMs

#103

Post by londons_explore »

cubefox wrote: post #72
drob518 wrote: post #56

I’m really curious how this scales up. Bonsai delivers an 8B model in 1.15 GB. How large would a 27B or 35B model be? Would it still retain the accuracy of those large models? If the scaling holds, we could see 100+B models in 64 GB of RAM.

Also depends on how expensive training these models is. It's probably at least as expensive as full precision models, otherwise they would have mentioned it.

My guess is the training process is their secret sauce...

Re: Show HN: 1-Bit Bonsai, the First Commercially Viable 1-Bit LLMs

#104

Post by kgeist »

freakynit wrote: post #93

Open access for next 5 hours (8GiB model, running on RTX 3090) or until server crashes or the this spot instance gets taken away :) => https://ofo1j9j6qh20a8-80.proxy.runpod.net ./build/bin/llama-server \ -m ../Bonsai-8B.gguf \ -ngl 999 \ --flash-attn on \ --host 0.0.0.0 \ --port 80 \ --ctx-size 65500 \ --batch-size 512 \ --ubatch-size 512 \ --parallel 5 \ --cont-batching \ --threads 8 \ --threads-batch 8 \ --cache-t…

[dead]

Re: Show HN: 1-Bit Bonsai, the First Commercially Viable 1-Bit LLMs

#105

Post by cubefox »

londons_explore wrote: post #103
cubefox wrote: post #72

Earlier quoted context omitted.

Also depends on how expensive training these models is. It's probably at least as expensive as full precision models, otherwise they would have mentioned it.

My guess is the training process is their secret sauce...

Yes, but their training speed is not secret. If their process were fast, they would have said so.

Re: Show HN: 1-Bit Bonsai, the First Commercially Viable 1-Bit LLMs

#106

Post by londons_explore »

Xx_crazy420_xX wrote: post #99
simian1983 wrote: post #5

Ran this prompt: Share an interesting scientific discovery or historical fact that most people don't know about. Please explain why it's significant and its impact on our understanding. Got this nonsense answer: Interesting Scientific Discovery: The Discovery of the "Boson" in 1924 by Charles K. Alton ("Chuck") Alton What it is: In 1924, American physicist Charles K. Alton discovered the "boson", a particle that woul…

Either we will be expecting the models to compress whole wikipedia and stale on the size reduction, or focus on the reasoning capabilities. My intuition is that by forcing models to remember everything we are wasting parameter space which can be allocated for more abstract thinking.

Integrating tool use into the training process should fix this.

Rather than learn about President Lincoln, the model can learn to look that info up with a search tool and use it to get better answers.

Just like a human does. I don't learn what 76x35 is... I learn that a calculator can give me that answer so I don't need to memorize it.

Re: Show HN: 1-Bit Bonsai, the First Commercially Viable 1-Bit LLMs

#107

Post by tim-projects »

onlyrealcuzzo wrote: post #35
ariwilson wrote: post #32

Very cool and works pretty well!

I'm fascinated by these smaller models. The amount of progress they've been making is incredible. Is anyone following this space more closely? Is anyone predicting performance at certain parameter sizes will plateau soon? Unlike the frontier models, these don't seem to be showing much progress of slowing down.

On the harness side there's a huge amount of optimisation room to go as well.

I strongly think smaller models will end up being able to do most coding tasks in the future, once they are reigned in properly

Re: Show HN: 1-Bit Bonsai, the First Commercially Viable 1-Bit LLMs

#109

Post by freakynit »

txrx0000 wrote: post #69
bilsbie wrote: post #39

I can’t see how this is possible. You’re losing so much information.

I always remind myself and everyone else that human DNA is "only" 1.6 GB of data, and yet it encodes all of the complex systems of the human body including the brain, and can replicate itself. Our intuitive feel of how much stuff can be packed into how many bits are probably way off from the true limits of physics.

Human DNA has 3.2 billion base pairs, and with 2x the information density compared to binary systems (due to 4-letters as opposed 2), that's roughly 800MB of informational data.

Second, what's even more crazy is that roughly 98% of that DNA is actually non-coding.. just junk.

So, we are talking about encoding entirety of the logic to construct a human body in just around 16MB of data!!!

That's some crazy levels of recursive compression.. maybe it's embedding "varying" parsing logic, mixed with data, along the chain.

Post reply on HN