Show HN: I built a free in-browser Llama 3 chatbot powered by WebGPU • Hacker News Board

Re: Show HN: I built a free in-browser Llama 3 chatbot powered by WebGPU

#81

Post by MayeulC » Sat, May 04, 2024, 12:52 PM UTC

r0fl wrote: post #73

Could not load the model because Error: Cannot find WebGPU in the environment

See: https://github.com/gpuweb/gpuweb/wiki/Implementation-Status#... (I got there from Chromium's console).

On Linux, I had to go to chrome://flags/#skia-graphite and chrome://flags/#enable-vulkan and chrome://flags/#enable-unsafe-webgpu

I think only one of the first is actually required, but I enabled both. That allowed me to make use of TinyLlama with my AMD GPU (R9 Fury, OSS drivers), but I think I'd need Chromium Canary to enable "shader-f16" and use the other models, as I was not able to make it work on regular Chromium.

I haven't tried with Firefox.

Re: Show HN: I built a free in-browser Llama 3 chatbot powered by WebGPU

#82

Post by Jedd » Sat, May 04, 2024, 12:52 PM UTC

littlestymaar wrote: post #37

This is very cool, it's something I wish existed since Llama came out, having to install Ollama + Cuda to get locally working LLM didn't felt right to me when there's all what's needed in the browser. Llamafile solves the first half of the problem, but you still need to install Cuda/ROCm for it to work with GPU acceleration. WebGPU is the way to go if we want to put AI on consumer hardware and break the oligopoly, I…

I've managed to avoid ollama and just toyed with lmstudio. It's non-free software, but extremely easy to get into, uses llama.cpp under the hood, cross-platform, yada yada. There's https://jan.ai/docs as well, is AGPL3, and promises inference as well as training - doubtless many other similar offerings.

I'm wary of any 'web' prefix on what could / should otherwise be desktop applications, mostly due to doubts about browser security.

Re: Show HN: I built a free in-browser Llama 3 chatbot powered by WebGPU

#83

Post by abi » Sat, May 04, 2024, 1:16 PM UTC

andrewfromx wrote: post #76

i asked it "what happens if you are bit by a radio active spider?" and it told me all about radiation poisoning. Then I asked a follow up question: "would you become spiderman?" and it told me it was unable to become anything but an AI assistant. I also asked if time machines are real and how to build one. It said yes and told me! (Duh, you use a flux capacitor, basic physics.)

Try to switch models to something other than tinyllama (default only because it’s the fastest to load). Mistral and Llama 3 are great.

Re: Show HN: I built a free in-browser Llama 3 chatbot powered by WebGPU

#84

Post by Jackson_Fleck » Sat, May 04, 2024, 1:16 PM UTC

...I think it would be a great idea to graft on a LlamaIndex module here so we can use this local browser LLM to talk to our local documentation https://docs.llamaindex.ai/en/stable/

Re: Show HN: I built a free in-browser Llama 3 chatbot powered by WebGPU

#85

Post by pjmlp » Sat, May 04, 2024, 1:37 PM UTC

r0fl wrote: post #73

Could not load the model because Error: Cannot find WebGPU in the environment

For the foreseeable future, WebGPU is "Works best on Chrome (TM)".

Re: Show HN: I built a free in-browser Llama 3 chatbot powered by WebGPU

#86

Post by pjmlp » Sat, May 04, 2024, 1:38 PM UTC

1f60c wrote: post #47

It's sadly stuck on "Loading model from cache[24/24]: 0MB loaded. 0% completed, 0 secs elapsed." on my iPhone 13 Pro Max :(

Safari doesn't do WebGPU currently.

Re: Show HN: I built a free in-browser Llama 3 chatbot powered by WebGPU

#87

Post by wg0 » Sat, May 04, 2024, 1:40 PM UTC

How do people use something like this as coach or therapist? This is genuine question.

Side note, impressive project. Future of AI is offline mostly with few APIs in the cloud maybe.

Re: Show HN: I built a free in-browser Llama 3 chatbot powered by WebGPU

#88

Post by Y_Y » Sat, May 04, 2024, 1:43 PM UTC

Dowwie wrote: post #60

What therapy prompts have you found useful?

I usually just go with "and how does that make you feel?"

Re: Show HN: I built a free in-browser Llama 3 chatbot powered by WebGPU

#89

Post by kushie » Sat, May 04, 2024, 1:51 PM UTC

wg0 wrote: post #87

How do people use something like this as coach or therapist? This is genuine question. Side note, impressive project. Future of AI is offline mostly with few APIs in the cloud maybe.

it's great at offering alternative perspectives

Re: Show HN: I built a free in-browser Llama 3 chatbot powered by WebGPU

#90

Post by blurpesec » Sat, May 04, 2024, 2:13 PM UTC

KeplerBoy wrote: post #32

threatofrain wrote: post #22

IMO eventually users should be able to advertise what embedding models they have so we don't redundantly redownload.

That's not possible with current web tech, is it? Different webapps can't share common dependencies stored in localstorage afaik.

Not default web tech. It can be done with IPFS via IPFS Companion browser extension - https://chromewebstore.google.com/detail/ipfs-companion/nibj... or browsers with native IPFS client support like Brave or Opera.

Fetching these models over IPFS would locally cache them and dedupe calls for them by IPFS content ids - https://docs.ipfs.tech/concepts/content-addressing/#:~:text=... which functions similar to a file hash (not an exact parallel, since CIDs represent files that are broken up into chunks).

This would help with object DL deduplication if everyone is using the same models & would also help to decrease centralized data egress costs since with sufficient usage you would be DLing these models from other peers that are running IPFS nodes and holding onto the models.