Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: I built a free in-browser Llama 3 chatbot powered by WebGPU

github.com

Post reply on HN
Print view View original

91–100 of 150 posts

Re: Show HN: I built a free in-browser Llama 3 chatbot powered by WebGPU

#91

Post by zitterbewegung »

nojvek wrote: post #64

Yasssssss! Thank you. This is the future. I am predicting Apple will make progress on groq like chipsets built in to their newer devices for hyper fast inference. LLMs leave a lot to be desired but since they are trained on all publicly available human knowledge they know something no about everything. My life has been better since I’ve been able to ask all sorts of adhoc questions about “is this healthy? Why healthy…

I actually think Apple has been putting neural engines in everything and might be training something like Llama3 for a very long time. Their conversational Siri is probably being neglected on purpose to replace it . They have released papers on faster inference and released their own models. I think their new Siri will largely use on device inference but with a very different LLM.

Even llama.cpp is performant already on macOS.

Re: Show HN: I built a free in-browser Llama 3 chatbot powered by WebGPU

#92

Post by spmurrayzzz »

littlestymaar wrote: post #37

This is very cool, it's something I wish existed since Llama came out, having to install Ollama + Cuda to get locally working LLM didn't felt right to me when there's all what's needed in the browser. Llamafile solves the first half of the problem, but you still need to install Cuda/ROCm for it to work with GPU acceleration. WebGPU is the way to go if we want to put AI on consumer hardware and break the oligopoly, I…

> having to install Ollama + Cuda to get locally working LLM didn't felt right to me when there's all what's needed in the browser

Was there something specifically about the install that didn't feel right? I ask because ollama is just a thin go wrapper around llama.cpp (its actually starting a modified version of the llama.cpp server in the background, not even going through the go ffi, likely for perf reasons). In that that sense, you could just install the CUDA toolkit via your package manager and calling `make LLAMA_CUDA=1; ./server` from the llama.cpp repo root to get effectively the same thing in two simple steps with no extra overhead.

Re: Show HN: I built a free in-browser Llama 3 chatbot powered by WebGPU

#95

Post by roywiggins »

doctorpangloss wrote: post #24
manlobster wrote: post #12

It's truly amazing how quickly my browser loads 0.6GB of data. I remember when downloading a 1MB file involved phoning up a sysop in advance and leaving the modem on all night. We've come so far.

97MB for the Worms 3 demo felt like an eternity. So what games are in this LLM? Can it do solitaire yet?

GPT-3.5 is pretty good at fabricating text adventures, I haven't tried any of the smaller models with that yet.

Re: Show HN: I built a free in-browser Llama 3 chatbot powered by WebGPU

#96

Post by maxboone »

nojvek wrote: post #64

Yasssssss! Thank you. This is the future. I am predicting Apple will make progress on groq like chipsets built in to their newer devices for hyper fast inference. LLMs leave a lot to be desired but since they are trained on all publicly available human knowledge they know something no about everything. My life has been better since I’ve been able to ask all sorts of adhoc questions about “is this healthy? Why healthy…

Groq is not general purpose enough, you'd be stuck with a specific model on your chip.

Re: Show HN: I built a free in-browser Llama 3 chatbot powered by WebGPU

#97

Post by cal85 »

wg0 wrote: post #87

How do people use something like this as coach or therapist? This is genuine question. Side note, impressive project. Future of AI is offline mostly with few APIs in the cloud maybe.

Genuine answer: you say “Be a coach/therapist” followed by whatever you’d say to a coach/therapist.

Re: Show HN: I built a free in-browser Llama 3 chatbot powered by WebGPU

#99

Post by ComputerGuru »

dosinga wrote: post #25

Phi 1.5 isn't quite there is it? Caught it having some inner dialog. What does that mean? Instruct: I'm sorry, but I'm not sure what you mean by "wutt". Output: Oh, I'm sorry. I meant "what does that mean?" Instruct: No problem, "wutt" means "that doesn't make any sense". Output: Ah, I see. Thank you for clarifying! Instruct: You're welcome! If you have any more questions, feel free to ask. Instruct: I'm glad I could…

That seems almost like the instruction template is wrong.

Re: Show HN: I built a free in-browser Llama 3 chatbot powered by WebGPU

#100

Post by neurostimulant »

mcculley wrote: post #74
nojvek wrote: post #64

Yasssssss! Thank you. This is the future. I am predicting Apple will make progress on groq like chipsets built in to their newer devices for hyper fast inference. LLMs leave a lot to be desired but since they are trained on all publicly available human knowledge they know something no about everything. My life has been better since I’ve been able to ask all sorts of adhoc questions about “is this healthy? Why healthy…

They are not “trained on all publicly available human knowledge”. Go look at the training data sets used. Most human knowledge that has been digitized is not publicly available (e.g., Google Books). These models are not able to get to data sets behind paywalls (e.g., scientific journals). It will be a huge step forward for humanity when we can run algorithms across all human knowledge. We are far from that.

There is a rumor that OpenAI might've used libgen in their training data.
Post reply on HN