Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

WebGPU GPT Model Demo

kmeans.org

Post reply on HN
Print view View original

11–20 of 32 posts

Re: WebGPU GPT Model Demo

#11

Post by zamadatix »

MuffinFlavored wrote: post #9

> At the moment, WebGPT averages ~300ms per token on GPT-2 124M running on a 2020 M1 Mac with Chrome Canary. How do ChatGPT on GPT-3.5 / GPT-4 compare?

I don't have exact numbers but trying all 3 out: on a local 4090 this seems mildly slower than GPT-4 and ridiculously slower than GPT 3.5 (both via cloud GPUs of course). That said all are within the realm of usability in terms of speed though GPT 3.5 is really in a whole different class of being able to be used nearly interactively without delay.

Re: WebGPU GPT Model Demo

#13

Post by DustinBrett »

fredrickd wrote: post #8
DustinBrett wrote: post #4

It indeed works and loads quick. I am more interested currently in the Vicuna 7B example from https://mlc.ai/web-llm/ Also instead of just "Update Chrome to v113" the domain owner could sign up for an origin trial https://developer.chrome.com/origintrials/#/view_trial/11821...

Thanks for the tip!

Np! Although there may still be some special WebGPU features in v113 that make v94-112 less ideal.

Re: WebGPU GPT Model Demo

#14

Post by lxe »

MuffinFlavored wrote: post #9

> At the moment, WebGPT averages ~300ms per token on GPT-2 124M running on a 2020 M1 Mac with Chrome Canary. How do ChatGPT on GPT-3.5 / GPT-4 compare?

Interesting, I'm getting 100ms/token on plain old wasm with 4 threads via ggml, using a 1.7B quantized cerebras model.

Re: WebGPU GPT Model Demo

#15

Post by brucethemoose2 »

MuffinFlavored wrote: post #9

> At the moment, WebGPT averages ~300ms per token on GPT-2 124M running on a 2020 M1 Mac with Chrome Canary. How do ChatGPT on GPT-3.5 / GPT-4 compare?

I can do 60 tokens in 24s on my 2020 ASUS G14, Thorium 111 + Windows 10. Nvidia-smi says my RTX 2060 is 24% loaded and has ~1.1G eaten up.

Thats slower than Vicuna 7B (aka LLaMa 7000M, GPT 3ish? model) on linux on the same machine, where I get about 3.5 tokens/sec and 97% usage. So... yeah, performance is not so great yet.

Re: WebGPU GPT Model Demo

#16

Post by brucethemoose2 »

doodlesdev wrote: post #3

> WebGPU is supported in your browser! > Uncaught (in promise) DOMException: WebGPU is not yet available in Release or Beta builds. Anyone using Chromium care to chime in? If no one chimes in I might set up a Chromium browser up just to take a look at this, seems pretty cool.

Thorium 111 is working with the WebGPU flags enabled.

Re: WebGPU GPT Model Demo

#17

Post by [deleted] »

doodlesdev wrote: post #7
EMM_386 wrote: post #5

Earlier quoted context omitted.

I'm on the latest Brave (Chromium 112.0.5615.165) and it tells me it is not supported.

It needs to be Chromium 113+, otherwise you'll have to enable webGPU in the browser flags like I have for Firefox.

[deleted]

Re: WebGPU GPT Model Demo

#19

Post by lxe »

FL33TW00D wrote: post #18

My 250M parameter model runs in 50ms/token ;) Releasing April 26th when Chrome 113 hits stable. Open source NPM library you can add to any project. Preview here: https://twitter.com/fleetwood___/status/1646608499126816799?...

That's pretty impressive. What model are you using?

Re: WebGPU GPT Model Demo

#20

Post by FL33TW00D »

lxe wrote: post #19
FL33TW00D wrote: post #18

My 250M parameter model runs in 50ms/token ;) Releasing April 26th when Chrome 113 hits stable. Open source NPM library you can add to any project. Preview here: https://twitter.com/fleetwood___/status/1646608499126816799?...

That's pretty impressive. What model are you using?

FLAN-T5 Base currently, 780M parameter variant coming shortly!
Post reply on HN