Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: 1-Bit Bonsai, the First Commercially Viable 1-Bit LLMs

prismml.com

Post reply on HN
Print view View original

171–180 of 181 posts

Re: Show HN: 1-Bit Bonsai, the First Commercially Viable 1-Bit LLMs

#171

Post by UncleOxidant »

UncleOxidant wrote: post #67
wild_egg wrote: post #43

Don't have a GPU so tried the CPU option and got 0.6t/s on my old 2018 laptop using their llama.cpp fork. Then found out they didn't implement AVX2 for their Q1_0_g128 CPU kernel. Added that and getting ~12t/s which isn't shabby for this old machine. Cool model.

Are you getting anything besides gibberish out of it? I tried their recommended commandline and it's dog slow even though I built their llama.cpp fork with AVX2 enabled. This is what I get: $ ./build/bin/llama-cli -hf prism-ml/Bonsai-8B-gguf -p "Explain quantum computing in simple terms." -n 256 --temp 0.5 --top-p 0.85 --top-k 20 -ngl 99 > Explain quantum computing in simple terms. \( , None ( no for the. (,./. all.2…

UPDATE: I was using the llama.cpp CPU backend and was still getting gibberish. On Google colab they're running with CUDA. I turned Claude loose on the problem and it discovered a problem in the llama.cpp CPU backend code where a float was being converted to an int and basically going to 0. Now it runs fine locally with the CPU backend.

Re: Show HN: 1-Bit Bonsai, the First Commercially Viable 1-Bit LLMs

#172

Post by dragonwriter »

riidom wrote: post #154
liuliu wrote: post #59

Earlier quoted context omitted.

It needs a mlx fork because the lowest bit in mlx is 2 currently (for affine quantization).

That mlx is for apple hardware only, though? Or did I misunderstand something.

It needs a llama.cpp fork, too; so the stock runtime (based on stock llama.cpp) used by LM Studio presumably won't work for it.

Re: Show HN: 1-Bit Bonsai, the First Commercially Viable 1-Bit LLMs

#173

Post by in-silico »

bilsbie wrote: post #39

I can’t see how this is possible. You’re losing so much information.

In theory you do lose information compared to parameters with more bits.

In practice, neural networks aren't able to store much more than 2-4 bits of useful information per parameter (regardless of the precision), so models like this are mostly getting rid of redundancy.

Re: Show HN: 1-Bit Bonsai, the First Commercially Viable 1-Bit LLMs

#175

Post by gorgonical »

UncleOxidant wrote: post #171
UncleOxidant wrote: post #67

Earlier quoted context omitted.

Are you getting anything besides gibberish out of it? I tried their recommended commandline and it's dog slow even though I built their llama.cpp fork with AVX2 enabled. This is what I get: $ ./build/bin/llama-cli -hf prism-ml/Bonsai-8B-gguf -p "Explain quantum computing in simple terms." -n 256 --temp 0.5 --top-p 0.85 --top-k 20 -ngl 99 > Explain quantum computing in simple terms. \( , None ( no for the. (,./. all.2…

UPDATE: I was using the llama.cpp CPU backend and was still getting gibberish. On Google colab they're running with CUDA. I turned Claude loose on the problem and it discovered a problem in the llama.cpp CPU backend code where a float was being converted to an int and basically going to 0. Now it runs fine locally with the CPU backend.

Mind sharing the fix as a patch? I would like to run it this way, too.

Re: Show HN: 1-Bit Bonsai, the First Commercially Viable 1-Bit LLMs

#177

Post by UncleOxidant »

gorgonical wrote: post #175
UncleOxidant wrote: post #171

Earlier quoted context omitted.

UPDATE: I was using the llama.cpp CPU backend and was still getting gibberish. On Google colab they're running with CUDA. I turned Claude loose on the problem and it discovered a problem in the llama.cpp CPU backend code where a float was being converted to an int and basically going to 0. Now it runs fine locally with the CPU backend.

Mind sharing the fix as a patch? I would like to run it this way, too.

https://github.com/philtomson/llama.cpp

Re: Show HN: 1-Bit Bonsai, the First Commercially Viable 1-Bit LLMs

#179

Post by nnevod »

qingcharles wrote: post #91
adityashankar wrote: post #18

Earlier quoted context omitted.

here's the google colab link, https://colab.research.google.com/drive/1EzyAaQ2nwDv_1X0jaC5... since the ngrok like likely got ddosed by the number of individuals coming along

Thanks, that works. I only tested the 1.7B. It has that original GPT3 feel to it. Hallucinates like crazy when it doesn't know something. For something that will fit on a GTX1080, though, it's solid. We're only a couple of years into optimization tech for LLMs. How many other optimizations are we yet to find? Just how small can you make a working LLM that doesn't emit nonsense? With the right math could we have been…

I think that not just could, we should have had them.

As far as I understand, neural networks were very hyped in 60s and 70s and when hype bust, they've fallen out of focus. Hardware was not there yet.

Then they were neglected for many years and really pioneer science was apparently only done by Google. Theoretical breakthroughs came in 2010s, after GPT-2 masses attention caught up and we (over)focused on neural networks again. GPT-2 was way below the capabilities of current hardware, we quickly caught up and now we're optimising.

Had it not be the burst of previous hype bubble, the NN wouldn't be essentially forgotten, and we'd have steady stream of optimisations and improvements while using the maximum of currently availible hardware.

Something like voice translation model running locally should have been possible by the end of 1990s. That way we'd have steady increase of LLM capabilities, no hype, and time to adapt and understand how to properly use them with no disruption.

Re: Show HN: 1-Bit Bonsai, the First Commercially Viable 1-Bit LLMs

#180

Post by gorgonical »

UncleOxidant wrote: post #177
gorgonical wrote: post #175

Earlier quoted context omitted.

Mind sharing the fix as a patch? I would like to run it this way, too.

https://github.com/philtomson/llama.cpp

Works like a charm for me, just like for you. Getting ~8 tok/s on my middle-of-the-road workstation laptop. Thanks for this!
Post reply on HN