Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: 1-Bit Bonsai, the First Commercially Viable 1-Bit LLMs

prismml.com

Post reply on HN
Print view View original

91–100 of 181 posts

Re: Show HN: 1-Bit Bonsai, the First Commercially Viable 1-Bit LLMs

#91

Post by qingcharles »

adityashankar wrote: post #18
jjcm wrote: post #16

1 bit with a FP16 scale factor every 128 bits. Fascinating that this works so well. I tried a few things with it. Got it driving Cursor, which in itself was impressive - it handled some tool usage. Via cursor I had it generate a few web page tests. On a monte carlo simulation of pi, it got the logic correct but failed to build an interface to start the test. Requesting changes mostly worked, but left over some symbol…

here's the google colab link, https://colab.research.google.com/drive/1EzyAaQ2nwDv_1X0jaC5... since the ngrok like likely got ddosed by the number of individuals coming along

Thanks, that works. I only tested the 1.7B. It has that original GPT3 feel to it. Hallucinates like crazy when it doesn't know something. For something that will fit on a GTX1080, though, it's solid.

We're only a couple of years into optimization tech for LLMs. How many other optimizations are we yet to find? Just how small can you make a working LLM that doesn't emit nonsense? With the right math could we have been running LLMs in the 1990s?

Re: Show HN: 1-Bit Bonsai, the First Commercially Viable 1-Bit LLMs

#92

Post by Dwedit »

noman-land wrote: post #75
tacotime wrote: post #44

Earlier quoted context omitted.

"Don't post generated comments or AI-edited comments. HN is for conversation between humans." https://news.ycombinator.com/newsguidelines.html#generated

How can you tell?

Presumably because a new account, and an offtopic post about AI. Then you look at the post history.

Re: Show HN: 1-Bit Bonsai, the First Commercially Viable 1-Bit LLMs

#93

Post by freakynit »

Open access for next 5 hours (8GiB model, running on RTX 3090) or until server crashes or the this spot instance gets taken away :) =>

https://ofo1j9j6qh20a8-80.proxy.runpod.net

  ./build/bin/llama-server \
   -m ../Bonsai-8B.gguf \
   -ngl 999 \
   --flash-attn on \
   --host 0.0.0.0 \
   --port 80 \
   --ctx-size 65500 \
   --batch-size 512 \
   --ubatch-size 512 \
   --parallel 5 \
   --cont-batching \
   --threads 8 \
   --threads-batch 8 \
   --cache-type-k q4_0 \
   --cache-type-v q4_0 \
   --log-colors on
The server can serve 5 parallel request, with each request capped at around `13K` tokens...

A bit of of benchmarks I did:

1. Input: 700 tokens, ttfs: ~0 second, outputs: 1822 tokens ~190t/s

1. Input: 6400+ tokens, ttfs: ~2 second, outputs: 2012 tokens at ~135t/s

Vram usage was consistently at ~4GiB.

Re: Show HN: 1-Bit Bonsai, the First Commercially Viable 1-Bit LLMs

#94

Post by logicallee »

freakynit wrote: post #93

Open access for next 5 hours (8GiB model, running on RTX 3090) or until server crashes or the this spot instance gets taken away :) => https://ofo1j9j6qh20a8-80.proxy.runpod.net ./build/bin/llama-server \ -m ../Bonsai-8B.gguf \ -ngl 999 \ --flash-attn on \ --host 0.0.0.0 \ --port 80 \ --ctx-size 65500 \ --batch-size 512 \ --ubatch-size 512 \ --parallel 5 \ --cont-batching \ --threads 8 \ --threads-batch 8 \ --cache-t…

That was really impressive. https://pastebin.com/PmJmTLJN pretty much instantly. (Very weak models can't do this.)

Re: Show HN: 1-Bit Bonsai, the First Commercially Viable 1-Bit LLMs

#97

Post by TRCat »

freakynit wrote: post #93

Open access for next 5 hours (8GiB model, running on RTX 3090) or until server crashes or the this spot instance gets taken away :) => https://ofo1j9j6qh20a8-80.proxy.runpod.net ./build/bin/llama-server \ -m ../Bonsai-8B.gguf \ -ngl 999 \ --flash-attn on \ --host 0.0.0.0 \ --port 80 \ --ctx-size 65500 \ --batch-size 512 \ --ubatch-size 512 \ --parallel 5 \ --cont-batching \ --threads 8 \ --threads-batch 8 \ --cache-t…

Thank you! I am impressed by the speed of it.

Re: Show HN: 1-Bit Bonsai, the First Commercially Viable 1-Bit LLMs

#98

Post by ggerganov »

freakynit wrote: post #93

Open access for next 5 hours (8GiB model, running on RTX 3090) or until server crashes or the this spot instance gets taken away :) => https://ofo1j9j6qh20a8-80.proxy.runpod.net ./build/bin/llama-server \ -m ../Bonsai-8B.gguf \ -ngl 999 \ --flash-attn on \ --host 0.0.0.0 \ --port 80 \ --ctx-size 65500 \ --batch-size 512 \ --ubatch-size 512 \ --parallel 5 \ --cont-batching \ --threads 8 \ --threads-batch 8 \ --cache-t…

Better keep the KV cache in full precision

Re: Show HN: 1-Bit Bonsai, the First Commercially Viable 1-Bit LLMs

#99

Post by Xx_crazy420_xX »

simian1983 wrote: post #5

Ran this prompt: Share an interesting scientific discovery or historical fact that most people don't know about. Please explain why it's significant and its impact on our understanding. Got this nonsense answer: Interesting Scientific Discovery: The Discovery of the "Boson" in 1924 by Charles K. Alton ("Chuck") Alton What it is: In 1924, American physicist Charles K. Alton discovered the "boson", a particle that woul…

Either we will be expecting the models to compress whole wikipedia and stale on the size reduction, or focus on the reasoning capabilities. My intuition is that by forcing models to remember everything we are wasting parameter space which can be allocated for more abstract thinking.

Re: Show HN: 1-Bit Bonsai, the First Commercially Viable 1-Bit LLMs

#100

Post by humanjir »

txrx0000 wrote: post #69
bilsbie wrote: post #39

I can’t see how this is possible. You’re losing so much information.

I always remind myself and everyone else that human DNA is "only" 1.6 GB of data, and yet it encodes all of the complex systems of the human body including the brain, and can replicate itself. Our intuitive feel of how much stuff can be packed into how many bits are probably way off from the true limits of physics.

That's not strictly true - DNA doesnt replicate itself, a cell with DNA replicates itself.

You need to count the information contained in the non-DNA part of the cell too.

Just in case it's not obvious, you can't take human DNA and put it in a cat cell, it won't work, that cell won't replicate.

Post reply on HN