Cool demo, but even the smallest model does 2.5 tps on my 7900xtx which is not great.
18.8 tps on an 11th gen mobile i7 here.
1-Bit LLM in the Browser
51–58 of 58 posts
Re: 1-Bit LLM in the Browser
#52I'm currently working on an open-source engine [1] exactly for this purpose. If anyone wants to try or has any suggestions, I'm happy to listen :) 1. https://github.com/stfurkan/bitgpu 2. https://aidekin.com --> this is one of my projects that's currently using bitgpu engine
Cool stuff! I see you're targeting two verticals here (runtime on top of ONNX + TS library), I'm curious did you consider simply creating a AI SDK Provider ( https://ai-sdk.dev/providers/community-providers/custom-prov... ) instead of crafting a separate library? Besides it, how are you dealing with the harness the model can use on web? I'm currently building https://github.com/bolojs/bolo to extend the base harness…
Thank you :) I wanted to create it from zero myself so I can control it better. e.g. I started with ONNX support but now it support gguf models too. It might be a good idea to add some popular connectors in the future. Good luck with your project :)
Re: 1-Bit LLM in the Browser
#53Impressive, I asked it how many r's in strawberry and it was immediately wrong with 2. any further questions I asked it proceeded to re-explain why there were 2 r's in strawberry, despite asking the silly "walk or drive to get my car washed" question. I struggle to see what these sorts of models could possibly be good at.
Re: 1-Bit LLM in the Browser
#54Earlier quoted context omitted.
IMHO from guy working with heavily quantized models and trying to get results. I do think quantization at less than 4-bit requires more advanced mathematics than is usually used. See Google Scholar Babak Hassibi (founder of Prism ML). Generally am amazed about the resiliency of these new data structures to approximation.
There was also a NeurIPS paper which successfully trained 1-bit models directly without floating point weights or any quantization: https://arxiv.org/abs/2405.16339 I didn't see any further work in this direction however. The paper seems to have been mostly overlooked.
While what can be done with less bits per parameter may be interesting, this is not really the important number for an LLM.
Much more important is the total amount of memory needed by an LLM, which can be decreased either by reducing the number of bits per parameter or the number of parameters. Arithmetic circuits are already cheap so for the inference cost the amount of memory transfers is more important.
It is not clear yet which number of bits per parameter allows the minimization of the total memory requirement, at a given LLM quality.
Re: 1-Bit LLM in the Browser
#55Earlier quoted context omitted.
There was also a NeurIPS paper which successfully trained 1-bit models directly without floating point weights or any quantization: https://arxiv.org/abs/2405.16339 I didn't see any further work in this direction however. The paper seems to have been mostly overlooked.
While what can be done with less bits per parameter may be interesting, this is not really the important number for an LLM. Much more important is the total amount of memory needed by an LLM, which can be decreased either by reducing the number of bits per parameter or the number of parameters. Arithmetic circuits are already cheap so for the inference cost the amount of memory transfers is more important. It is not…
I thought the multiplications during training (backpropagation) are quite computationally expensive. The method from the paper above doesn't need any multiplications.
Re: 1-Bit LLM in the Browser
#56Earlier quoted context omitted.
Interesting. But trying the demo with 27G model and simple 8+5 did not work. Guessing there is some work to do still ``` calculate({"expression":"g"}) → error: only numbers, + - * / and parentheses are allowed error: No user query found in messages. What is 8+5? Use the calculator get_time({}) → Mon Jul 20 2026 10:36:28 GMT+0200 (Central European Summer Time) error: No user query found in messages. hi Hello! 2 tokens…
Thank you for trying it :) I just added the 27B support 1 hour ago. Let me check and try to fix it. All other models should work as expected but if you have issues with them too, I can take a look.
Hi again :) I updated the engine and 27B model should work in the demo[1]. We can still consider it as experimental but I am trying to make it better and efficient.
Re: 1-Bit LLM in the Browser
#57[dead]
Re: 1-Bit LLM in the Browser
#58Earlier quoted context omitted.
Thats absolutly not the point of these models. They are here to route things, do tool calls, ask experts.
That is completely false. It is a reasonable type of question and a reasonable expectation. You can't route correctly if you can't do reasoning. Imagine a prompt: ``` INSTRUCTION: You're a customer service agent that routes customer service requests. CUSTOMER MESSAGE: I haven't received my order and want a refund. CONTEXT: A signature is required at delivery. No one was available to sign. You can route to one of: * B…
IT is not black and white and just because you are able to bring up a random example doesn't mean you are right.
These models are build to be small, we have a lot of diverse tasks which do not need fable or whatever because if you are a company and you have to route 100.000 requests per day, costs matter.
And these small models, if they are not good enough for my specific use case, i can easily finetune (which i have btw.).
You can also easily solve your example by having a second stage. If the customer is unhappy about the first thing or an employee flags it as 'wrong', you can either send it automatically to a more expensive model or the employee fixes it and you got automatic training data.