[dead]
Show HN: I built a free in-browser Llama 3 chatbot powered by WebGPU
121–130 of 150 posts
Re: Show HN: I built a free in-browser Llama 3 chatbot powered by WebGPU
#122Earlier quoted context omitted.
I commented too soon. The TinyLlama model didn't seem to be able to summarize web pages but Llama 3 worked perfectly! Very cool.
Are you sure it is not hallucinating? Most likely these models don't have an access to the Internet. edit: typo
Yes, I got way too excited and comment trigger happy. It does not appear to browse the web and was just hallucinating. The hallucinations were surprisingly convincing for a couple of the pages I tested. But on examining the network requests, no fetches were made to the pages. Llama 3 was just a lot better at hallucinating convincing results than Tiny Llama.
Re: Show HN: I built a free in-browser Llama 3 chatbot powered by WebGPU
#123Earlier quoted context omitted.
Nice personal hosted image service!
I’m just using CleanShotX [0] which is an awesome image annotation tool for macOS. It’s way better than the built-in tool that macOS comes with. You can also record as a gif for video which is nice, I use it often to make guides for my day job and my business. [0] https://cleanshot.com
I'm using flameshot ( https://flameshot.org/ ), which sounds pretty similar, but FOSS and cross platform.
Re: Show HN: I built a free in-browser Llama 3 chatbot powered by WebGPU
#124Amazing work, feels like a step forward for LLM usability. Would be interesting if there was a web browser that managed the download/install of models so you could go to a site like this, or any other LLM site/app and it detects whether or not you have models, similar to detecting if you have a webcam or mic for a video call. The user can click "Allow" to allow use of GPU and allow running of models in the background…
Window AI (https://windowai.io/) is an attempt to do something like this with a browser extension.
Re: Show HN: I built a free in-browser Llama 3 chatbot powered by WebGPU
#125This is absolutely wonderful, I am a HUGE fan of local first apps. Running models locally is such a powerful thing I wish more companies could leverage it to build smarter apps which can run offline. I tried this on my M1 and ran LLama3, I think it's the quantized 7B version. It ran with around 4-5 tokens per second which was way faster than I expected on my browser.
Appreciate the kind words :)
Re: Show HN: I built a free in-browser Llama 3 chatbot powered by WebGPU
#126It's a wrapper of https://github.com/mlc-ai/web-llm
Yes. Web-llm is a wrapper of tvmjs: https://github.com/apache/tvm
Just wrappers all the way down
Re: Show HN: I built a free in-browser Llama 3 chatbot powered by WebGPU
#127Very cool! I wish there was chat history. Also if you click the "New Chat" button while an answer is generating I think some of the output gets fed back into the model, it causes some weird output [0] but was kind of cool/fun. Here is a video of it as well [1], I almost think this should be some kind of special mode you can run. I'd be interested to know what the bug causes, is it just the existing output sent as inp…
Thanks for the bug report. Yeah, it’s a bug with not resetting the state properly when new chat is clicked. Will fix tomorrow.
Chat history shouldn’t be hard to add with local storage and Indexed DB.
Re: Show HN: I built a free in-browser Llama 3 chatbot powered by WebGPU
#128Earlier quoted context omitted.
I tried using Claude and ChatGPT like this: I would just write a free form journal entry. The feedback it gave was typically very useful and made journaling more rewarding.
Given user data is folded back into the models, there is a snowball's chance in hell that I would input stuff I'd talk to a therapist about. When are people going to realize that their interactions with AIs are likely being analyzed/characterized, and that at some point, that analysis will be monetized?
Use secret llama in a incognito window. Turn off the Internet and close the window when done.
Re: Show HN: I built a free in-browser Llama 3 chatbot powered by WebGPU
#129This is very cool, it's something I wish existed since Llama came out, having to install Ollama + Cuda to get locally working LLM didn't felt right to me when there's all what's needed in the browser. Llamafile solves the first half of the problem, but you still need to install Cuda/ROCm for it to work with GPU acceleration. WebGPU is the way to go if we want to put AI on consumer hardware and break the oligopoly, I…
Tested on Ubuntu 22.04 with Chrome, sure enough, "Could not load the model because Error: Cannot find adapter that matches the request". It really is too bad WebGPU isn't supported on Linux, I mean, that's a no-brainer right there.
Likewise (same error) with Chrome on Windows.
Currently running Ollama / Open WebUI and finding lama3:8B quite useful for writing snippets of powershell, javascript, golang etc.
Re: Show HN: I built a free in-browser Llama 3 chatbot powered by WebGPU
#130Earlier quoted context omitted.
Mozilla won't even allow WebSerial to be implemented because it was deemed "too dangerous" - with all sorts of absurd whinging about the devastation that could be unleashed by unsuspecting users allowing a malicious site to access USB serial devices. When someone pointed out that Chrome has had this functionality for years and the world has not imploded...and has enabled many open source projects and web-based microc…
> (If Chrome is such a user privacy dumpsterfire, why not implement WebSerial so that people don't have to run Chrome in order to communicate with and program microcontrollers?) This doesn't seem like a logical comparison. Is there no other way to program microcontrollers outside of Chrome?
Well sure, downloadable executables. Which I feel like isn't much better and in a lot of ways worse.