Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

1-Bit LLM in the Browser

huggingface.co

Post reply on HN
Print view View original

21–30 of 58 posts

Re: 1-Bit LLM in the Browser

#21

Post by Lwerewolf »

Pretty sure this might be a duplicate. Regardless, tried the 1bit bonsai 27b gguf three different ways - their llama.cpp fork (prism, was it) on 2 machines (1255u/16g, m5 max/128g) and the web (on the 1255u). With llama.cpp it works. on the web it emits the same thing over and over. Prompt was "do you see anything wrong with this code: ", result was literally "Yes, let'ssomestruct_struct_tsomestructstructstructstruct.

Locally, it's definitely not the full 3.6 27b, but for ~6G with the context, it's quite impressive, and I like what this implies for larger models. Speeds on the 1255u are abysmal, though (~1TPS or less) - granted, CPU-only.

Re: 1-Bit LLM in the Browser

#23

Post by luca-3 »

stfurkan wrote: post #2

I'm currently working on an open-source engine [1] exactly for this purpose. If anyone wants to try or has any suggestions, I'm happy to listen :) 1. https://github.com/stfurkan/bitgpu 2. https://aidekin.com --> this is one of my projects that's currently using bitgpu engine

This is great for high-privacy use cases. If you could add an agentic loop that can call tools within the website using the user’s session, that would be great. Then the user can chat with an AI about sensitive information in the most secure way possible.

Also, do you have data on the technical requirements for this? If someone uses an old phone, e.g., does it still work?

Re: 1-Bit LLM in the Browser

#25

Post by [deleted] »

indianmouse wrote: post #14

Could not get any usable output from it for any real usecase. Can't see any real value in using 1 bit llms except and apart for the sake of running it for demo or feel good factor of running a higher param model locally and in limited hardware! Since use as it is very limited, could someone throw some light on any actual use case?

[deleted]

Re: 1-Bit LLM in the Browser

#26

Post by kees99 »

Lwerewolf wrote: post #21

Pretty sure this might be a duplicate. Regardless, tried the 1bit bonsai 27b gguf three different ways - their llama.cpp fork (prism, was it) on 2 machines (1255u/16g, m5 max/128g) and the web (on the 1255u). With llama.cpp it works. on the web it emits the same thing over and over. Prompt was "do you see anything wrong with this code: ", result was literally "Yes, let'ssomestruct_struct_t somestruct structstructstru…

Qwen is quite slow in general.

I'm getting ~0.5 tps from [qwen], and ~10 tps from [gemma] - roughly the same size, same quant, same hardware (8-core CPU) same software (llama.cpp).

[qwen] Qwen3.6-27B-Q4_K_M.gguf

[gemma] gemma-4-26B-A4B-it-qat-UD-Q4_K_XL.gguf

Re: 1-Bit LLM in the Browser

#27

Post by whstl »

geon wrote: post #16
woadwarrior01 wrote: post #15

Earlier quoted context omitted.

Works with the Bonsai-27B-1bit-mlx model on my mac. Edit: this is running with MLX, and not with WebGPU on the browser, also it is a fail. """ You should *walk*. Here’s why: - *Distance*: 50 meters is very short — about 0.3 miles or 0.5 kilometers. - *Time*: Walking takes roughly 2–3 minutes. Driving would take longer due to parking, starting the engine, and maneuvering. - *Effort*: Walking is light exercise and avoi…

How is that not a fail?

I think GP also failed the car wash test.

Re: 1-Bit LLM in the Browser

#28

Post by christkv »

kees99 wrote: post #26
Lwerewolf wrote: post #21

Pretty sure this might be a duplicate. Regardless, tried the 1bit bonsai 27b gguf three different ways - their llama.cpp fork (prism, was it) on 2 machines (1255u/16g, m5 max/128g) and the web (on the 1255u). With llama.cpp it works. on the web it emits the same thing over and over. Prompt was "do you see anything wrong with this code: ", result was literally "Yes, let'ssomestruct_struct_t somestruct structstructstru…

Qwen is quite slow in general. I'm getting ~0.5 tps from [qwen], and ~10 tps from [gemma] - roughly the same size, same quant, same hardware (8-core CPU) same software (llama.cpp). [qwen] Qwen3.6-27B-Q4_K_M.gguf [gemma] gemma-4-26B-A4B-it-qat-UD-Q4_K_XL.gguf

Quite different models though Qwen 3.6 model has 27B active parameters. gemma-4 has 26B parameters with 4B active at any given time. You should compare with Qwen 3.6 35B A3B

Re: 1-Bit LLM in the Browser

#30

Post by EGreg »

stfurkan wrote: post #2

I'm currently working on an open-source engine [1] exactly for this purpose. If anyone wants to try or has any suggestions, I'm happy to listen :) 1. https://github.com/stfurkan/bitgpu 2. https://aidekin.com --> this is one of my projects that's currently using bitgpu engine

Yes, I’m very interested in exploring this stuff.

Want to connect and discuss it? https://calendly.com/safebotsai

Post reply on HN