41–50 of 58 posts
I tried the smallest ome, the only available, and got an error: ``` Could not load. Error: failed to call OrtRun(). ERROR_CODE: 1, ERROR_MESSAGE: Non-zero status code returned while running GroupQueryAttention node. Name:'/model/layers.0/attn/GroupQueryAttention' Status Message: Failed to create a WebGPU compute pipeline: A valid external Instance reference no longer exists. ```
1-bit is not much though. Here's what I got: Me: > Describe the process of pasteurisation Response: > pasteurization is a, which a, the which which is, the and process, the the past, the and the, and and and past, and past, and and and and the, the and past, process is the is is is is, and the, past, and and future, process, and and and and and and and the, past, process, process, the the, the the, is and the, the th…
But yes, this will happen. 1.7B is already a tiny model, running that at 1-bit is going to lead to some behavioral issues and will need tuning of temperature/topP/repeatPenalty/etc, and probably a software fallback that detects pathological cases like this and simply retries. Just like we all had to do in the times of GPT 2 and GPT 3
1-bit is not much though. Here's what I got: Me: > Describe the process of pasteurisation Response: > pasteurization is a, which a, the which which is, the and process, the the past, the and the, and and and past, and past, and and and and the, the and past, process is the is is is is, and the, past, and and future, process, and and and and and and and the, past, process, process, the the, the the, is and the, the th…
I got a decent response with the same prompt But yes, this will happen. 1.7B is already a tiny model, running that at 1-bit is going to lead to some behavioral issues and will need tuning of temperature/topP/repeatPenalty/etc, and probably a software fallback that detects pathological cases like this and simply retries. Just like we all had to do in the times of GPT 2 and GPT 3
i found this space to work better/at all: https://huggingface.co/spaces/webml-community/bonsai-webgpu-...
Fails "car wash" test miserably. Ask it: > I want to wash my car. The car wash is 50 meters away. Should I walk or drive? Ref: https://opper.ai/blog/car-wash-test
Thats absolutly not the point of these models. They are here to route things, do tool calls, ask experts.
Imagine a prompt:
```
INSTRUCTION: You're a customer service agent that routes customer service requests.
CUSTOMER MESSAGE: I haven't received my order and want a refund.
CONTEXT: A signature is required at delivery. No one was available to sign.
You can route to one of:
* Billing & refunds
* Shipping & tracking
* Order cancelation
* Level II customer service
```
The dumb model, e.g. 1.7B, wrongly routes to Billing. The smart model, e.g. GPT-5.6-Medium, correctly routes to Shipping. It matters. In the real world, the CONTEXT will even be 10-100x larger and noisier.
1-bit is not much though. Here's what I got: Me: > Describe the process of pasteurisation Response: > pasteurization is a, which a, the which which is, the and process, the the past, the and the, and and and past, and past, and and and and the, the and past, process is the is is is is, and the, past, and and future, process, and and and and and and and the, past, process, process, the the, the the, is and the, the th…
1-bit is not much though. Here's what I got: Me: > Describe the process of pasteurisation Response: > pasteurization is a, which a, the which which is, the and process, the the past, the and the, and and and past, and past, and and and and the, the and past, process is the is is is is, and the, past, and and future, process, and and and and and and and the, past, process, process, the the, the the, is and the, the th…
IMHO from guy working with heavily quantized models and trying to get results. I do think quantization at less than 4-bit requires more advanced mathematics than is usually used. See Google Scholar Babak Hassibi (founder of Prism ML). Generally am amazed about the resiliency of these new data structures to approximation.
I didn't see any further work in this direction however. The paper seems to have been mostly overlooked.