Transformers.js – Run Transformers directly in the browser
1–10 of 54 posts
Re: Transformers.js – Run Transformers directly in the browser
#2This is super cool, but unfortunately it also seems super impractical. Models tend to be quite large, so even if a browser can run them, getting them to the browser involves either:
1. Large downloads on every visit to a website.
2. Large downloads and high storage consumption for each website using large models. (150 websites x 800 MB models => 120 GB of storage used)
Both of those options seem terrible.
I think it might make sense for browsers to ship with some models built in and be exposed via standardized web APIs in the future, but I haven't heard of any efforts to make that happen yet.
Re: Transformers.js – Run Transformers directly in the browser
#3This is super cool, but unfortunately it also seems super impractical. Models tend to be quite large, so even if a browser can run them, getting them to the browser involves either: 1. Large downloads on every visit to a website. 2. Large downloads and high storage consumption for each website using large models. (150 websites x 800 MB models => 120 GB of storage used) Both of those options seem terrible. I think it…
Basically the same problem that's plagued games on the web ever since the first Unreal/Unity asmjs demos a decade ago, and pretty much no progress has been made towards a solution in that time. You just can't practically make a web app which needs gigs of data on the client because there's no reliable way to make sure it stays cached for as long as the user wants it to, and as you say, even if you could reliably cache it the download and storage would still be duplicated per site using the same model due to browsers cache partitioning policies.
Re: Transformers.js – Run Transformers directly in the browser
#4Ok so now we can make a browser plugin which will pick out all bicycles or bridges in a Google captcha, right?
Re: Transformers.js – Run Transformers directly in the browser
#5This is super cool, but unfortunately it also seems super impractical. Models tend to be quite large, so even if a browser can run them, getting them to the browser involves either: 1. Large downloads on every visit to a website. 2. Large downloads and high storage consumption for each website using large models. (150 websites x 800 MB models => 120 GB of storage used) Both of those options seem terrible. I think it…
Some of the models are quite small and worth doing on-device vs the opposite of sending all the data to the server to process. The other huge benefit here is that transformers run in node.js and getting things running is way easier than trying to get some odd combination of python snd its dependencies to work
Re: Transformers.js – Run Transformers directly in the browser
#6The Syntax podcast recently did an episode on Transformers.js and the developer…
https://syntax.fm/show/740/local-ai-models-in-javascript-mac...
Re: Transformers.js – Run Transformers directly in the browser
#7This is super cool, but unfortunately it also seems super impractical. Models tend to be quite large, so even if a browser can run them, getting them to the browser involves either: 1. Large downloads on every visit to a website. 2. Large downloads and high storage consumption for each website using large models. (150 websites x 800 MB models => 120 GB of storage used) Both of those options seem terrible. I think it…
If they are single files or directories they could be drag dropped on use. Not very convenient though.
Maybe just some sort of api to give the website fine grained access to the filesystem might be enough. You'd specify a directory or single file the website can read from at any time.
However at some point you will have to download large files. I feel when done implicitly it's bad user experience.
On top of that the developer should implement a robust downloading system that can resume downloads, check for validity, etc. Developers rarerly bother with this, so the user experience is that it sucks.
Re: Transformers.js – Run Transformers directly in the browser
#8This is super cool, but unfortunately it also seems super impractical. Models tend to be quite large, so even if a browser can run them, getting them to the browser involves either: 1. Large downloads on every visit to a website. 2. Large downloads and high storage consumption for each website using large models. (150 websites x 800 MB models => 120 GB of storage used) Both of those options seem terrible. I think it…
This is pure free-association: models are below 80 MB, the rest are LLMs and aren't in scope. Whisper is 40 MB, embeddings are 23 MB. (n.b. parts of original comment that actively disclaim understanding: "seems super impractical. Models tend to be quite large...150 websites x 800 MB models")
Re: Transformers.js – Run Transformers directly in the browser
#9This is super cool, but unfortunately it also seems super impractical. Models tend to be quite large, so even if a browser can run them, getting them to the browser involves either: 1. Large downloads on every visit to a website. 2. Large downloads and high storage consumption for each website using large models. (150 websites x 800 MB models => 120 GB of storage used) Both of those options seem terrible. I think it…
Browsers can store stuff that's downloaded. Using e.g. the Filesystem API. These files can be accessed from multiple websites. Browser applications can run offline with service workers.
Js/browser based solutions seem to be very often knee-jerk dismissed based on decade old understanding of browser capabilities.
Re: Transformers.js – Run Transformers directly in the browser
#10This is super cool, but unfortunately it also seems super impractical. Models tend to be quite large, so even if a browser can run them, getting them to the browser involves either: 1. Large downloads on every visit to a website. 2. Large downloads and high storage consumption for each website using large models. (150 websites x 800 MB models => 120 GB of storage used) Both of those options seem terrible. I think it…
It's an inherent problem with on-device AI processing, not just in the browser. I think this will only get better when operating systems start to preinstall models and provide an API that browser vendors can use as well.
Even then I think cloud hosted models will probably always be far better for most tasks.