Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Transformers.js – Run Transformers directly in the browser

github.com

Post reply on HN
Print view View original

51–54 of 54 posts

Re: Transformers.js – Run Transformers directly in the browser

#51

Post by xenova »

jph00 wrote: post #42
xenova wrote: post #19

Earlier quoted context omitted.

We’ve put out a ton of demos that use much smaller models (10-60 MB), including: - (44MB) In-browser background removal: https://huggingface.co/spaces/Xenova/remove-background-web . (We also put out a WebGPU version: https://huggingface.co/spaces/Xenova/remove-background-webgp... ). - (51MB) Whisper Web for automatic speech recognition: https://huggingface.co/spaces/Xenova/whisper-web (just select the quantized versi…

Why is only one of them on WebGPU? Is it because there additional tricky steps required to make a model work on WebGPU, or is there a limitation on what ops are supported there? I'm keen to do more stuff with WebGPU, so very interested to learn about challenges and limitations here.

We have some other WebGPU demos, including:

- WebGPU embedding benchmark: https://huggingface.co/spaces/Xenova/webgpu-embedding-benchm...

- Real-time object detection: https://huggingface.co/spaces/Xenova/webgpu-video-object-det...

- Real-time background removal: https://huggingface.co/spaces/Xenova/webgpu-video-background...

- WebGPU depth estimation: https://huggingface.co/spaces/Xenova/webgpu-depth-anything

- Image background removal: https://huggingface.co/spaces/Xenova/remove-background-webgp...

You can follow the progress for full WebGPU support in the v3 development branch (https://github.com/xenova/transformers.js/pull/545).

To answer your question, while there are certain ops missing, the main limitation at the moment is for models with decoders... which are not very fast (yet) due to inefficient buffer reuse and many redundant copies between CPU and GPU. We're working closely with the ORT team to fix these issues though!

Re: Transformers.js – Run Transformers directly in the browser

#52

Post by pests »

bilekas wrote: post #50
janalsncm wrote: post #44

Earlier quoted context omitted.

Usually the inference time is small compared with download time so even if this were technically feasible you wouldn’t save much time. For reference I have a 31mb vision transformer I run in my browser. Building the inputs, running inference, and parsing the response takes less than half a second.

> Usually the inference time is small compared with download time so even if this were technically feasible you wouldn’t save much time. I can understand that but where time is not a factor and solely a question of data, can a model be streamed?

LLMs like ChatGPT only generate one token at a time. To generate more you run inference repeatedly until you reach a stop token or some other predetermined limit.

I don't see streaming helping anything besides maybe Time-To-First-Inference, but regardless, you're still not getting any output until the entire weights are downloaded.

Re: Transformers.js – Run Transformers directly in the browser

#54

Post by goldcaddy77 »

ctrlaltdylan wrote: post #38

Also this opens the possibility of running these models on Node.js serverless functions no? That certainly also has to open up possibilities for on-demand predictions?

This is interesting. Would love to see some examples of this.
Post reply on HN