Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Transformers.js – Run Transformers directly in the browser

github.com

Post reply on HN
Print view View original

41–50 of 54 posts

Re: Transformers.js – Run Transformers directly in the browser

#41

Post by Vetch »

fauigerzigerk wrote: post #10
jfoster wrote: post #2

This is super cool, but unfortunately it also seems super impractical. Models tend to be quite large, so even if a browser can run them, getting them to the browser involves either: 1. Large downloads on every visit to a website. 2. Large downloads and high storage consumption for each website using large models. (150 websites x 800 MB models => 120 GB of storage used) Both of those options seem terrible. I think it…

It's an inherent problem with on-device AI processing, not just in the browser. I think this will only get better when operating systems start to preinstall models and provide an API that browser vendors can use as well. Even then I think cloud hosted models will probably always be far better for most tasks.

This specific problem is certainly not one for all on-devices AI processing. As someone else mentioned, there are unique UX and browser constraints that come from serving large compute intensive binary blobs through the browser (that are almost identically shared by games).

Separately, having to rely on preinstallation very likely means stagnating on overly sanitized poorly done official instruction-tunes. With the exception of mixtral7x8, the trend has been the community overtime arrives at finetunes which far eclipse official ones.

Re: Transformers.js – Run Transformers directly in the browser

#42

Post by jph00 »

xenova wrote: post #19
jfoster wrote: post #2

This is super cool, but unfortunately it also seems super impractical. Models tend to be quite large, so even if a browser can run them, getting them to the browser involves either: 1. Large downloads on every visit to a website. 2. Large downloads and high storage consumption for each website using large models. (150 websites x 800 MB models => 120 GB of storage used) Both of those options seem terrible. I think it…

We’ve put out a ton of demos that use much smaller models (10-60 MB), including: - (44MB) In-browser background removal: https://huggingface.co/spaces/Xenova/remove-background-web . (We also put out a WebGPU version: https://huggingface.co/spaces/Xenova/remove-background-webgp... ). - (51MB) Whisper Web for automatic speech recognition: https://huggingface.co/spaces/Xenova/whisper-web (just select the quantized versi…

Why is only one of them on WebGPU? Is it because there additional tricky steps required to make a model work on WebGPU, or is there a limitation on what ops are supported there?

I'm keen to do more stuff with WebGPU, so very interested to learn about challenges and limitations here.

Re: Transformers.js – Run Transformers directly in the browser

#43

Post by bilekas »

jfoster wrote: post #2

This is super cool, but unfortunately it also seems super impractical. Models tend to be quite large, so even if a browser can run them, getting them to the browser involves either: 1. Large downloads on every visit to a website. 2. Large downloads and high storage consumption for each website using large models. (150 websites x 800 MB models => 120 GB of storage used) Both of those options seem terrible. I think it…

This is probably a really stupid question, but can the models be streamed as they're being ran? So that you as the browser wouldn't need to wait for the entire download first? Or there is even the concept of ordered model transformers?

As I ask it seems wrong to me but just to confirm?

Re: Transformers.js – Run Transformers directly in the browser

#44

Post by janalsncm »

bilekas wrote: post #43
jfoster wrote: post #2

This is super cool, but unfortunately it also seems super impractical. Models tend to be quite large, so even if a browser can run them, getting them to the browser involves either: 1. Large downloads on every visit to a website. 2. Large downloads and high storage consumption for each website using large models. (150 websites x 800 MB models => 120 GB of storage used) Both of those options seem terrible. I think it…

This is probably a really stupid question, but can the models be streamed as they're being ran? So that you as the browser wouldn't need to wait for the entire download first? Or there is even the concept of ordered model transformers? As I ask it seems wrong to me but just to confirm?

Usually the inference time is small compared with download time so even if this were technically feasible you wouldn’t save much time.

For reference I have a 31mb vision transformer I run in my browser. Building the inputs, running inference, and parsing the response takes less than half a second.

Re: Transformers.js – Run Transformers directly in the browser

#47

Post by spmurrayzzz »

Tenoke wrote: post #46

Is training not possible? I did some stuff years ago where I create and train small NNs in the browser and I'm curious if that type of thing would work better today with a small custom transformer.

In theory its definitely possible, but I suspect that maybe performance concerns are probably the reason its not implemented (yet). They have a webgpu embeddings benchmark in an HF space to give you a sense of the forward pass dynamics: https://huggingface.co/spaces/Xenova/webgpu-embedding-benchm...

Its impressive for what it is, but training would be painful at those latencies (fp16, batch 32, sequence length 512 generates a ~500ms forward pass with a 22M param model)

Re: Transformers.js – Run Transformers directly in the browser

#48

Post by dheera »

spmurrayzzz wrote: post #47
Tenoke wrote: post #46

Is training not possible? I did some stuff years ago where I create and train small NNs in the browser and I'm curious if that type of thing would work better today with a small custom transformer.

In theory its definitely possible, but I suspect that maybe performance concerns are probably the reason its not implemented (yet). They have a webgpu embeddings benchmark in an HF space to give you a sense of the forward pass dynamics: https://huggingface.co/spaces/Xenova/webgpu-embedding-benchm... Its impressive for what it is, but training would be painful at those latencies (fp16, batch 32, sequence length 512 ge…

There might be applications for much smaller transformers in UI design.

Like for example

- did the user tap the wrong location on the screen because their device was physically jolted, and can you correct for that, considering you have access to accelerometer in HTML5

- does the user keep repeating an action (checking every box in a list of e-mails) and can you extrapolate the rest of what the user wants to do

- did the user bounce because you popped up a stupid intercom box or newsletter popup, and did you learn anything about what you need to do if you want to retain this particular user in the future

these kinds of things could be done with hundreds or thousands of parameters or less

Re: Transformers.js – Run Transformers directly in the browser

#49

Post by spmurrayzzz »

dheera wrote: post #48
spmurrayzzz wrote: post #47

Earlier quoted context omitted.

In theory its definitely possible, but I suspect that maybe performance concerns are probably the reason its not implemented (yet). They have a webgpu embeddings benchmark in an HF space to give you a sense of the forward pass dynamics: https://huggingface.co/spaces/Xenova/webgpu-embedding-benchm... Its impressive for what it is, but training would be painful at those latencies (fp16, batch 32, sequence length 512 ge…

There might be applications for much smaller transformers in UI design. Like for example - did the user tap the wrong location on the screen because their device was physically jolted, and can you correct for that, considering you have access to accelerometer in HTML5 - does the user keep repeating an action (checking every box in a list of e-mails) and can you extrapolate the rest of what the user wants to do - did…

Yea definitely. But in that case, you could train _much_ faster in pytorch, then convert to ONNX, and load in the browser for inference (as the transformers.js docs recommend)

EDIT: (I responded before your full edit with the bullet list). This next comment is orthogonal to the slow training performance topic I think, but the use cases you reference there don't seem to be well-suited at all for an autoregressive decoder-only model architecture.

Re: Transformers.js – Run Transformers directly in the browser

#50

Post by bilekas »

janalsncm wrote: post #44
bilekas wrote: post #43

Earlier quoted context omitted.

This is probably a really stupid question, but can the models be streamed as they're being ran? So that you as the browser wouldn't need to wait for the entire download first? Or there is even the concept of ordered model transformers? As I ask it seems wrong to me but just to confirm?

Usually the inference time is small compared with download time so even if this were technically feasible you wouldn’t save much time. For reference I have a 31mb vision transformer I run in my browser. Building the inputs, running inference, and parsing the response takes less than half a second.

> Usually the inference time is small compared with download time so even if this were technically feasible you wouldn’t save much time.

I can understand that but where time is not a factor and solely a question of data, can a model be streamed?

Post reply on HN