Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: WebGPU enables local LLM in the browser – demo site with AI chat

andreinwald.github.io

Post reply on HN
Print view View original

31–40 of 60 posts

Re: Show HN: WebGPU enables local LLM in the browser – demo site with AI chat

#31

Post by wongarsu »

andreinwald wrote: post #23
echoangle wrote: post #20

Fun demo but the model that's used seems to be pretty stupid: > What's the best way to get to space? >> Unfortunately, it's not currently possible for humans to travel to space in the same way that astronauts do. While there have been several manned missions to space, such as those to the International Space Station, the technology and resources required to make interstellar travel feasible are still in the early sta…

It works on small Llama-3.2-1B model, specially for less powerfull GPU devices

The answer is still terrible for the model size. Maybe it's the 4 bit quantization, smaller models tend to react worse to that

For reference, [1] is what stock quen3-0.6B would answer. Not a perfect answer, but much better at nearly half the number of parameters

1: https://markdownpastebin.com/?id=7ad4ad9f325d4354a858480abdc...

Re: Show HN: WebGPU enables local LLM in the browser – demo site with AI chat

#32

Post by fooker »

echoangle wrote: post #20

Fun demo but the model that's used seems to be pretty stupid: > What's the best way to get to space? >> Unfortunately, it's not currently possible for humans to travel to space in the same way that astronauts do. While there have been several manned missions to space, such as those to the International Space Station, the technology and resources required to make interstellar travel feasible are still in the early sta…

It’s broadly correct, if you interpret ‘humans’ as ordinary humans who are not astronauts.

Re: Show HN: WebGPU enables local LLM in the browser – demo site with AI chat

#33

Post by fooker »

apitman wrote: post #28

Does anyone know why all of these WebGPU LLM demos have you download the models to browser storage rather than letting you open a gguf already on your local drive? I have several models downloaded already that I would be interested in trying.

Browsers are sandboxed away from user storage.

You can change this by changing settings, command line arguments, build flags, etc. But can’t really expect people to do this just to use your website.

Re: Show HN: WebGPU enables local LLM in the browser – demo site with AI chat

#34

Post by KTibow »

wongarsu wrote: post #31
andreinwald wrote: post #23

Earlier quoted context omitted.

It works on small Llama-3.2-1B model, specially for less powerfull GPU devices

The answer is still terrible for the model size. Maybe it's the 4 bit quantization, smaller models tend to react worse to that For reference, [1] is what stock quen3-0.6B would answer. Not a perfect answer, but much better at nearly half the number of parameters 1: https://markdownpastebin.com/?id=7ad4ad9f325d4354a858480abdc...

It's likely the quantization on "Llama-3.2-1B-Instruct-q4f16_1-MLC". inference.net generated this more coherent answer: https://hst.sh/ovilewofox.md

Re: Show HN: WebGPU enables local LLM in the browser – demo site with AI chat

#35

Post by dragonwriter »

fooker wrote: post #32
echoangle wrote: post #20

Fun demo but the model that's used seems to be pretty stupid: > What's the best way to get to space? >> Unfortunately, it's not currently possible for humans to travel to space in the same way that astronauts do. While there have been several manned missions to space, such as those to the International Space Station, the technology and resources required to make interstellar travel feasible are still in the early sta…

It’s broadly correct, if you interpret ‘humans’ as ordinary humans who are not astronauts.

That generous redefinition.of “humans” deals with only the error in the first sentence of the response; the rest is still not “broadly correct” even with it in place.

Re: Show HN: WebGPU enables local LLM in the browser – demo site with AI chat

#36

Post by wongarsu »

fooker wrote: post #32
echoangle wrote: post #20

Fun demo but the model that's used seems to be pretty stupid: > What's the best way to get to space? >> Unfortunately, it's not currently possible for humans to travel to space in the same way that astronauts do. While there have been several manned missions to space, such as those to the International Space Station, the technology and resources required to make interstellar travel feasible are still in the early sta…

It’s broadly correct, if you interpret ‘humans’ as ordinary humans who are not astronauts.

Blue Origin is built on taking ordinary humans to space. So is/was Virgin Galactic, though they are in a bit of a transitionary phase right now. SpaceX is also willing, they might even take you on a flyby of the moon if you bring money and patience (iirc the last customer lost patience). Basically just ring up your favorite multi-billionaire's space program. And while the estimated price would be correct for SpaceX, rumored prices for Blue Origin's New Shepard are only in the hundreds of thousands per seat

Edit: also the "but if you do that you are an astronaut, so it's still true that only astronauts can do that" loophole was closed when the FAA redefined the word astronaut in 2021. At least if you follow their definition of the word

Re: Show HN: WebGPU enables local LLM in the browser – demo site with AI chat

#38

Post by AndrewDucker »

I asked "Why is the sky blue?" and got back a response of

"coppia RR TalentDevExpressincer+'.//////////////////////////////////////////////////////////////////////// cha ولا.AutoSizesaving proleงคicate Like"/>

infos эти za cornerback economical (%]\ enumligne.execRELEASEPropagation_station Bucks проHEME seas GASPOST[Unit(suffix Gloves"

(and so on, for a few more paragraphs).

Am I missing something?

Post reply on HN