Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: 1-Bit Bonsai, the First Commercially Viable 1-Bit LLMs

prismml.com

Post reply on HN
Print view View original

61–70 of 181 posts

Re: Show HN: 1-Bit Bonsai, the First Commercially Viable 1-Bit LLMs

#61

Post by jjcm »

andai wrote: post #48
jjcm wrote: post #16

1 bit with a FP16 scale factor every 128 bits. Fascinating that this works so well. I tried a few things with it. Got it driving Cursor, which in itself was impressive - it handled some tool usage. Via cursor I had it generate a few web page tests. On a monte carlo simulation of pi, it got the logic correct but failed to build an interface to start the test. Requesting changes mostly worked, but left over some symbol…

Thanks. Did you need to use Prism's llama.cpp fork to run this?

Yep.

Re: Show HN: 1-Bit Bonsai, the First Commercially Viable 1-Bit LLMs

#62

Post by pdyc »

jjcm wrote: post #16

1 bit with a FP16 scale factor every 128 bits. Fascinating that this works so well. I tried a few things with it. Got it driving Cursor, which in itself was impressive - it handled some tool usage. Via cursor I had it generate a few web page tests. On a monte carlo simulation of pi, it got the logic correct but failed to build an interface to start the test. Requesting changes mostly worked, but left over some symbol…

thanks, i tested it, failed in strawberry test. qwen 3.5 0.8B with similar size passes it and is far more usable.

Re: Show HN: 1-Bit Bonsai, the First Commercially Viable 1-Bit LLMs

#64

Post by cubefox »

wild_egg wrote: post #43

Don't have a GPU so tried the CPU option and got 0.6t/s on my old 2018 laptop using their llama.cpp fork. Then found out they didn't implement AVX2 for their Q1_0_g128 CPU kernel. Added that and getting ~12t/s which isn't shabby for this old machine. Cool model.

"Not shabby" is a big understatement.

Re: Show HN: 1-Bit Bonsai, the First Commercially Viable 1-Bit LLMs

#65

Post by abrookewood »

jjcm wrote: post #16

1 bit with a FP16 scale factor every 128 bits. Fascinating that this works so well. I tried a few things with it. Got it driving Cursor, which in itself was impressive - it handled some tool usage. Via cursor I had it generate a few web page tests. On a monte carlo simulation of pi, it got the logic correct but failed to build an interface to start the test. Requesting changes mostly worked, but left over some symbol…

man, that is really really quick. What is your desktop setup??? GPU?

Re: Show HN: 1-Bit Bonsai, the First Commercially Viable 1-Bit LLMs

#66

Post by abrookewood »

jjcm wrote: post #27
najarvg wrote: post #24

Earlier quoted context omitted.

Thanks for sharing the link to your instance. Was blazing fast in responding. Tried throwing a few things at it with the following results: 1. Generating an R script to take a city and country name and finding it's lat/long and mapping it using ggmaps. Generated a pretty decent script (could be more optimal but impressive for the model size) with warnings about using geojson if possible 2. Generate a latex script to…

> Was blazing fast in responding. I should note this is running on an RTX 6000 pro, so it's probably at the max speed you'll get for "consumer" hardware.

Holy hell ... that's a monster of a card

Re: Show HN: 1-Bit Bonsai, the First Commercially Viable 1-Bit LLMs

#67

Post by UncleOxidant »

wild_egg wrote: post #43

Don't have a GPU so tried the CPU option and got 0.6t/s on my old 2018 laptop using their llama.cpp fork. Then found out they didn't implement AVX2 for their Q1_0_g128 CPU kernel. Added that and getting ~12t/s which isn't shabby for this old machine. Cool model.

Are you getting anything besides gibberish out of it? I tried their recommended commandline and it's dog slow even though I built their llama.cpp fork with AVX2 enabled. This is what I get:

    $ ./build/bin/llama-cli     -hf prism-ml/Bonsai-8B-gguf -p "Explain quantum computing in simple terms." -n 256 --temp 0.5 --top-p 0.85 --top-k 20 -ngl 99
    > Explain quantum computing in simple terms.

     \( ,

      None ( no for the. (,./. all.2... the                                                                                                                                ..... by/
EDIT: It runs fine in their collab notebook. Looking at that you have to do: git checkout prism (in the llama.cpp repo) before you build. That's a missing instruction if you're going straight to their fork of llama.cpp. Works fine now.

Re: Show HN: 1-Bit Bonsai, the First Commercially Viable 1-Bit LLMs

#69

Post by txrx0000 »

bilsbie wrote: post #39

I can’t see how this is possible. You’re losing so much information.

I always remind myself and everyone else that human DNA is "only" 1.6 GB of data, and yet it encodes all of the complex systems of the human body including the brain, and can replicate itself. Our intuitive feel of how much stuff can be packed into how many bits are probably way off from the true limits of physics.
Post reply on HN