Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: 1-Bit Bonsai, the First Commercially Viable 1-Bit LLMs

prismml.com

Post reply on HN
Print view View original

161–170 of 181 posts

Re: Show HN: 1-Bit Bonsai, the First Commercially Viable 1-Bit LLMs

#161

Post by CamperBob2 »

monarchwadia wrote: post #144
jjcm wrote: post #29

Earlier quoted context omitted.

I ran it through a rudimentary thinking harness, and it still failed, fwiw: The question is about the best mode of transportation to a car wash located 100 meters away. Since the user is asking for a recommendation, it's important to consider practical factors like distance, time, and convenience. Walking is the most convenient and eco-friendly option, especially if the car wash is within a short distance. It avoids…

Honestly, the fact that we have models that can coherently reason about this problem at all is a technological miracle. And to have it runnable in a 1.15GB memory footprint? Is insanity.

Exactly. It's not that the pig dances poorly, or that the dog's stock tips never seem to pan out. It's the fact that it's happening at all.

Re: Show HN: 1-Bit Bonsai, the First Commercially Viable 1-Bit LLMs

#162

Post by guerrilla »

hrmtst93837 wrote: post #121
alyxya wrote: post #11

I expect the trend of large machine learning models to go towards bits rather than operating on floats. There's a lot of inefficiency in floats because typically they're something like normally distributed, which makes the storage and computation with weights inefficient when most values are clustered in a small range. The foundation of neural networks may be rooted in real valued functions, which are simulated with…

[flagged]

Well this is perfect then. We just post-process models like this after training.

Re: Show HN: 1-Bit Bonsai, the First Commercially Viable 1-Bit LLMs

#163

Post by iJohnDoe »

Tried running the models with the latest LM Studio, llama.cpp, and Ollama. All failed.

https://huggingface.co/prism-ml/Bonsai-8B-gguf

tensor 'token_embd.weight' has invalid ggml type 41. should be in [0, 41) loader knows tensor types 0..40, but the model contains type 41

Re: Show HN: 1-Bit Bonsai, the First Commercially Viable 1-Bit LLMs

#164

Post by monarchwadia »

CamperBob2 wrote: post #161
monarchwadia wrote: post #144

Earlier quoted context omitted.

Honestly, the fact that we have models that can coherently reason about this problem at all is a technological miracle. And to have it runnable in a 1.15GB memory footprint? Is insanity.

Exactly. It's not that the pig dances poorly, or that the dog's stock tips never seem to pan out. It's the fact that it's happening at all.

But the fact that we have convinced a pig to dance, and trained a dog to provide stock tips? That can be improved upon over time. We've gotten here, haven't we? It really is a miracle, and I'll stick to that opinion.

Re: Show HN: 1-Bit Bonsai, the First Commercially Viable 1-Bit LLMs

#167

Post by Dylan16807 »

ineedasername wrote: post #58
jjcm wrote: post #27

Earlier quoted context omitted.

> Was blazing fast in responding. I should note this is running on an RTX 6000 pro, so it's probably at the max speed you'll get for "consumer" hardware.

consumer hardware ? That... pft. Nevermind, I'm just jealous

Well in this context it's a 5090 with extra unused memory.

Re: Show HN: 1-Bit Bonsai, the First Commercially Viable 1-Bit LLMs

#168

Post by m0do1 »

iJohnDoe wrote: post #163

Tried running the models with the latest LM Studio, llama.cpp, and Ollama. All failed. https://huggingface.co/prism-ml/Bonsai-8B-gguf tensor 'token_embd.weight' has invalid ggml type 41. should be in [0, 41) loader knows tensor types 0..40, but the model contains type 41

prismML provides a llama.cpp fork which is compatible with the 1 bit models:

https://github.com/PrismML-Eng/llama.cpp

After fails with Ollama and main llama.cpp the fork worked on my M5 MBA.

Edit: Typos

Re: Show HN: 1-Bit Bonsai, the First Commercially Viable 1-Bit LLMs

#169

Post by cztomsik »

pdyc wrote: post #62
jjcm wrote: post #16

1 bit with a FP16 scale factor every 128 bits. Fascinating that this works so well. I tried a few things with it. Got it driving Cursor, which in itself was impressive - it handled some tool usage. Via cursor I had it generate a few web page tests. On a monte carlo simulation of pi, it got the logic correct but failed to build an interface to start the test. Requesting changes mostly worked, but left over some symbol…

thanks, i tested it, failed in strawberry test. qwen 3.5 0.8B with similar size passes it and is far more usable.

I hope you are kidding, how is that a test of any capabilities? it's a miracle that any model can learn strawberry because it cannot see the actual characters and ALSO, it's likely misspelled a lot in the corpus. I've been playing with this model and I'm pleasantly surprised, it certainly knows a lot, quite a lot for 1.1G

Re: Show HN: 1-Bit Bonsai, the First Commercially Viable 1-Bit LLMs

#170

Post by w10-1 »

anecdotal experience report:

They link the (free) locally.ai iPhone app, but the bonsai model doesn't present in the list. You have to get it via settings.

On my ancient SE-2, Siri integration falls down, but the chat in their app runs about half the speed I can read. So far, more than 50% correct, and usable (and seems to speed up as you use it).

I'll try it just to clean up input in a pipeline to another model. I gave it a paragraph from the NYTimes and it did a great job, so it should be good at correcting voice input and keyboard typos.

Post reply on HN