Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

1-Bit LLM in the Browser

huggingface.co

Post reply on HN
Print view View original

31–40 of 58 posts

Re: 1-Bit LLM in the Browser

#31

Post by stfurkan »

luca-3 wrote: post #23
stfurkan wrote: post #2

I'm currently working on an open-source engine [1] exactly for this purpose. If anyone wants to try or has any suggestions, I'm happy to listen :) 1. https://github.com/stfurkan/bitgpu 2. https://aidekin.com --> this is one of my projects that's currently using bitgpu engine

This is great for high-privacy use cases. If you could add an agentic loop that can call tools within the website using the user’s session, that would be great. Then the user can chat with an AI about sensitive information in the most secure way possible. Also, do you have data on the technical requirements for this? If someone uses an old phone, e.g., does it still work?

Hi, the engine itself already has tool calling capabilities but if you're asking for the aidekin I didn't add tool calling there on purpose.

It needs webgpu support but I haven't had a chance to test with different devices.

Re: 1-Bit LLM in the Browser

#32

Post by estebarb »

I tried the smallest ome, the only available, and got an error:

``` Could not load.

Error: failed to call OrtRun(). ERROR_CODE: 1, ERROR_MESSAGE: Non-zero status code returned while running GroupQueryAttention node. Name:'/model/layers.0/attn/GroupQueryAttention' Status Message: Failed to create a WebGPU compute pipeline: A valid external Instance reference no longer exists. ```

Re: 1-Bit LLM in the Browser

#33

Post by stfurkan »

EGreg wrote: post #30
stfurkan wrote: post #2

I'm currently working on an open-source engine [1] exactly for this purpose. If anyone wants to try or has any suggestions, I'm happy to listen :) 1. https://github.com/stfurkan/bitgpu 2. https://aidekin.com --> this is one of my projects that's currently using bitgpu engine

Yes, I’m very interested in exploring this stuff. Want to connect and discuss it? https://calendly.com/safebotsai

Hello, I sent you a LinkedIn connection request. We can chat first and schedule a meeting. Thanks

Re: 1-Bit LLM in the Browser

#34

Post by [deleted] »

christkv wrote: post #28
kees99 wrote: post #26

Earlier quoted context omitted.

Qwen is quite slow in general. I'm getting ~0.5 tps from [qwen], and ~10 tps from [gemma] - roughly the same size, same quant, same hardware (8-core CPU) same software (llama.cpp). [qwen] Qwen3.6-27B-Q4_K_M.gguf [gemma] gemma-4-26B-A4B-it-qat-UD-Q4_K_XL.gguf

Quite different models though Qwen 3.6 model has 27B active parameters. gemma-4 has 26B parameters with 4B active at any given time. You should compare with Qwen 3.6 35B A3B

[deleted]

Re: 1-Bit LLM in the Browser

#35

Post by espetro »

stfurkan wrote: post #2

I'm currently working on an open-source engine [1] exactly for this purpose. If anyone wants to try or has any suggestions, I'm happy to listen :) 1. https://github.com/stfurkan/bitgpu 2. https://aidekin.com --> this is one of my projects that's currently using bitgpu engine

Cool stuff! I see you're targeting two verticals here (runtime on top of ONNX + TS library), I'm curious did you consider simply creating a AI SDK Provider (https://ai-sdk.dev/providers/community-providers/custom-prov...) instead of crafting a separate library?

Besides it, how are you dealing with the harness the model can use on web? I'm currently building https://github.com/bolojs/bolo to extend the base harness agents can use on web and I'm interested in hearing from your experience on crafting tools for bitgpu

Re: 1-Bit LLM in the Browser

#38

Post by lelanthran »

1-bit is not much though. Here's what I got:

Me:

> Describe the process of pasteurisation

Response:

> pasteurization is a, which a, the which which is, the and process, the the past, the and the, and and and past, and past, and and and and the, the and past, process is the is is is is, and the, past, and and future, process, and and and and and and and the, past, process, process, the the, the the, is and the, the the, and and and, and process, and and the, past, and and and past, and the, the, and the past, the the, process, process, and past, the past, past, the and, the past, and and and and and and and and and the, and and and and and and and, the the, the the, the or and and, the the, the and the, which the, and the, the the, past, and n the process, and and and and, past, and, and and, the past, and, the the, past,, and the, the the, the is the, past, and and and and and, and and and past, the and the, the the, the the, are and past, and which the, the and n, n the, the n past, past, n the, and n, the the process, which past, the the, the n, the the, the is past, the the, is, past, the the, past, and past, process, the the, the the, the and and the, and which past, the

(and that basically just goes on and on like that)

Post reply on HN